WO2017213307A1

WO2017213307A1 - Apparatus and method for processing digital image on basis of nui/nux

Info

Publication number: WO2017213307A1
Application number: PCT/KR2016/012235
Authority: WO
Inventors: 조경은; 치옥용; 백승대
Original assignee: 동국대학교 산학협력단
Priority date: 2016-06-07
Filing date: 2016-10-28
Publication date: 2017-12-14
Also published as: KR101775080B1

Abstract

The present invention relates to an apparatus and a method for processing a digital image on the basis of an NUI/NUX, which learn a drawing pattern of a user in advance by receiving the actual drawing image of the user and the pressure of the hand at the time of drawing; recognize the movement of a drawing means held by the user by receiving a drawing motion of the user; read stroke information by matching the recognized movement of the drawing means with the previously learned drawing pattern; and output, in real-time, a drawing image corresponding to the stroke information which corresponds to the drawing motion of the user.

Description

Apparatus and method for processing digital video based on NUI / NUX

DETAILED DESCRIPTION The present invention relates to a technique for implementing digital drawing or digital sculpting, particularly an intuitive drawing image or sculpture from a drawing action or a hand action through a natural user interface (NUI) that is most intuitive to humans. A digital image processing apparatus, a method for generating an image, and a recording medium recording the method.

With the popularity of Internet cartoon services such as web-toons, more people use tablets to draw pictures, and the general public is increasingly using digital digitizers such as drawing calipers or hobbies. .

Conventional digital drawing technology uses an electronic pen to implement the effect of drawing on a digital canvas through special software. In addition, there is a method of drawing a picture by tracking a trace of a finger or a stylus using touch technology. These methods require the use of special pens or sensing means, and when drawing, their senses are different from those of actual pens and brushes, and they cannot provide the operability that can be felt in actual drawings. In addition, these sensations were different from actual drawings, which did not help in drawing practice.

In addition, it is a very unfamiliar concept for artists to sculpt in a digital environment, and it was only possible to imagine that digital sculpture could be realized using gestures to cut a sculpture object using a real sculpture.

In the following prior art document, a technical means for detecting a user input using a pen having a magnetic field generating means is introduced, which is also mentioned above in that a dedicated electronic pen with a sensor and a special device is used. The same problem is expected.

<Prior art document> Korean Patent Publication 10-2015-0080480, 2015.07.09 publication

Therefore, for ordinary people, it is not necessary to use ordinary stick objects that replace the sculpture or provide a feeling similar to drawing with a real pencil, pen or brush, without using special equipment such as a dedicated electronic pen or tablet. There is a need for simple digital image processing means that can provide a immersive experience similar to carving.

The technical problem to be solved by the present invention is that the conventional digital drawing technology has to utilize expensive professional equipment, such as a dedicated electronic pen or tablet, solve the problem that is difficult for general users to use, and using the electronic pen Overcoming the limitation that drawing is different from the actual drawing and providing a sense of reality, in the case of sculpture art, it tries to go beyond the fixed point of view where no attempt was made to implement it through the virtual world.

In order to solve the above technical problem, a drawing image processing method according to an embodiment of the present invention, comprising the steps of pre-learning the user's drawing pattern by receiving the user's actual drawing image and the pressure of the hand during drawing; Receiving a drawing motion of the user and recognizing a movement of the drawing means held by the user from the input drawing motion; Reading stroke information by matching the recognized movement of the drawing means with the previously learned drawing pattern; And outputting a drawing image corresponding to the stroke information in real time in response to the drawing motion of the user.

In the drawing image processing method according to an embodiment, the step of pre-learning a drawing pattern of the user may include obtaining a 3D image of the actual drawing of the user by using a depth camera; Measuring pressure of a hand corresponding to the actual drawing of the user using an electromyogram (EMG) sensor attached to the user's hand and converting the pressure into a pen pressure of a drawing means; And matching state values, movements, speeds, and pressures of the drawing means according to the obtained 3D image, and storing the state values together with the stroke shapes of the corresponding viewpoints. In addition, the drawing pattern of the user is stored for each of a plurality of frames (frames) for the actual drawing, and may include a time series change of the state value and the pen pressure of the drawing means included in each frame. Further, the drawing pattern of the user may be differentiated and stored for each individual user as unique identification information for the user by combining at least two of a state value, a movement, a speed, and a pressure of the drawing means.

In the drawing image processing method according to an embodiment, the step of recognizing the movement of the drawing means may include a 3D image and a 2D image of the drawing means held by the user from a drawing motion of the user input using a depth camera. Extracting each; And calculating the inclination angle of the drawing means, whether the drawing means is in contact with a plane corresponding to the virtual canvas, and the distance by matching the extracted 3D image and the 2D image. Can be. In addition, the drawing means may have a form of a pen or a brush, and the method may further include identifying a type of the drawing means by using unique information included in a two-dimensional image of the drawing means. have.

In the drawing image processing method according to an exemplary embodiment, the reading of the stroke information may include calculating at least two motion information among a state value, a movement, and a speed of the drawing means from the recognized movement of the drawing means; And searching for a value matching the calculated motion information from the previously learned drawing pattern, and returning stroke information reflecting a pen pressure.

According to an embodiment, the drawing image processing method may further include outputting a virtual canvas space using a display means, and displaying the drawing image output in real time in the virtual canvas space.

In order to solve the above technical problem, the engraving image processing method according to another embodiment of the present invention, by receiving the actual sculpting image of the user and the pressure of the hand of the sculpture received by the type of the carving pattern of the user by the type of carving means Learning in advance; Receiving a sculpting motion of the user and recognizing the type of sculpting means held by the user and the movement of the sculpting means from the input sculpting motion; Reading piece carving information according to the type of carving means by matching the recognized movement of the carving means with the previously learned carving pattern; And outputting a fragment image corresponding to the fragment mark information in real time in response to the fragment motion of the user.

In the engraving image processing method according to another embodiment, the learning of the engraving pattern of the user in advance may include: obtaining a 3D image of the actual carving behavior of the user using a depth camera; Measuring the pressure of the hand corresponding to the actual engraving behavior of the user by using an electromyogram (EMG) sensor attached to the user's hand and converting the pressure into a pressure of the engraving means; And matching a state value, a movement, a speed, and a pressure of the engraving means according to the obtained 3D image, and storing the state values together with the shape of the engraving marks at the corresponding point of time according to the type of the engraving means. In addition, the engraving pattern of the user is stored for each of a plurality of frames (frame) for the actual engraving behavior, and may include a time-series change in the state value and pressure of the engraving means included in each frame.

In the engraving image processing method according to another embodiment, the step of recognizing the movement of the engraving means, the three-dimensional image and the two-dimensional image of the engraving means held by the user from the user's engraving motion input using the depth camera Extracting each; And matching the extracted 3D image and the 2D image to calculate an inclination angle of the engraving means, whether the engraving means is in contact with a reference position corresponding to the virtual engraving object, and a distance. have. In addition, the carving means may have a form of a carving knife (carving knife), it may further comprise the step of identifying the type of the carving means using the unique information contained in the two-dimensional image of the carving means.

In the engraving image processing method according to another exemplary embodiment, the reading of the engraving mark information may include: calculating at least two pieces of motion information of a state value, a movement, and a speed of the carving means from the recognized movement of the carving means; And searching for a value matched with the calculated motion information from the previously learned piece pattern, and returning piece carving information corresponding to the type of piece means and reflecting pressure. The method may further include adjusting the pressure or the size of the engraving mark in proportion to the speed of the engraving means.

In the fragment image processing method according to another embodiment, the method may further include outputting a virtual fragment object using a display means, and displaying the fragment image output in real time on the virtual fragment object.

Embodiments of the present invention, it is possible to provide a user with a more realistic drawing or sculpture experience by implementing digital art through NUI the drawing motion or sculpture motion that is most familiar to humans, not a special electronic pen or digitizer, NUI / Based on NUX, it enables fast and familiar adaptation and easy manipulation of digital images to non-professional people of all ages.

1 is a flowchart illustrating a method of processing a drawing image based on NUI / NUX according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a general environment in which embodiments of the present invention are implemented and accompanying apparatus.

3 is a diagram illustrating the flow of data according to the operation of embodiments of the present invention.

4 and 5 are diagrams for explaining a method of recognizing the movement of the drawing means.

6 is a view for explaining a method of measuring the user's pen pressure.

7 is a block diagram illustrating an apparatus for processing a drawing image based on NUI / NUX according to an embodiment of the present invention.

8 is a diagram illustrating an environment for implementing a fragment image and an apparatus associated with the fragment image according to another embodiment of the present invention.

9 is a flowchart illustrating a method of processing a fragment image based on NUI / NUX according to another embodiment of the present invention.

700: drawing image processing apparatus

10: depth camera

20: EMG sensor

30: Processor

40: memory

50: learning database

60, 60A, 60B: display means

According to an embodiment of the present invention, a drawing image processing method may include: learning a drawing pattern of a user in advance by receiving an actual drawing image of a user and a pressure of a hand during drawing; Receiving a drawing motion of the user and recognizing a movement of the drawing means held by the user from the input drawing motion; Reading stroke information by matching the recognized movement of the drawing means with the previously learned drawing pattern; And outputting a drawing image corresponding to the stroke information in real time in response to the drawing motion of the user.

In accordance with another aspect of the present invention, there is provided a method of processing a sculpture image, the method comprising: pre-learning a sculpture pattern of a user for each type of engraving means by receiving an actual sculpting image of a user and a pressure of a hand among pieces; Receiving a sculpting motion of the user and recognizing the type of sculpting means held by the user and the movement of the sculpting means from the input sculpting motion; Reading piece carving information according to the type of carving means by matching the recognized movement of the carving means with the previously learned carving pattern; And outputting a fragment image corresponding to the fragment mark information in real time in response to the fragment motion of the user.

Conventional technical means for implementing digital drawing requires the use of specially prepared electronic pens, which makes it difficult for non-professional users to use. The difference is that it is not very helpful for actual conversation practice. In other words, even if you repeat the drawing practice through the digitizer, you get a different feeling from drawing on a real canvas, which makes it difficult for ordinary users who are still inexperienced to draw on canvas or paper.

Embodiments of the present invention described below have been made to solve the above-mentioned problems, and try to implement a summoning reality-based digital drawing system using NUI technology that is most intuitive to humans. Such a digital drawing system can realize digital drawing using only ordinary drawing means such as a pencil, pen or brush, or even a flat bar without using a special electronic pen of a general tablet. Through the embodiments of the present invention, the user may draw a picture while freely taking a drawing motion on a screen or a display screen projected by the project as if the picture is drawn on a real canvas. In addition, similar to when using the actual drawing means, the picture effect can be naturally realized according to the inclination angle of the pen / brush, the size of the pressing force or the speed of moving the pen / brush. Users can use common drawing means, even bars, and they can be used on a variety of planes / surfaces, such as plain paper, canvas or even walls, regardless of where they are drawing (e.g. special equipment such as conventional tablets) or their location. You can draw with a sense similar to In addition, while measuring the force of the user grasping the pen / brush to convert to a pen pressure, realistic picture effects can be implemented according to the direction of the pen / brush and the size of the force.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, in the following description and the accompanying drawings, detailed descriptions of well-known functions or configurations that may obscure the subject matter of the present invention will be omitted. In addition, it should be noted that like elements throughout the drawings are represented by the same name and reference numerals as much as possible.

In operation S110, the drawing image processing apparatus receives the actual drawing image of the user and the pressure of the hand during drawing to learn the drawing pattern of the user in advance. Drawings or writings differ in habits from person to person, and in the embodiments of the present invention, a drawing pattern is learned based on the force of applying a drawing / drawing habit of an individual to a finger and a moving angle and direction information by holding a pen / brush. When drawing or writing, the force applied to the fingertips is constantly changing, which can be seen from individual EMG data patterns acquired through electromyogram (EMG) sensors attached to the user's hands.

That is, the S110 process of learning the user's drawing pattern in advance, obtains a 3D image of the actual drawing of the user by using a depth camera, and uses an electromyogram (EMG) sensor attached to the user's hand. The pressure of the hand corresponding to the actual drawing of the user is measured and converted into the pen pressure of the drawing means, and the state value, the movement, the speed and the pen pressure of the drawing means according to the obtained 3D image are matched, By storing together, it can be implemented. In particular, when the user generates an actual digital drawing image by using the EMG data accumulated in the process of learning the drawing pattern, the stroke degree suitable for the natural drawing motion only without the cumbersome work such as the attachment of the EMG sensor Can be obtained.

In operation S120, the drawing image processing apparatus receives a drawing motion of the user by using a depth camera, recognizes a movement of the drawing means held by the user from the input drawing motion, and separately in this process. EMG sensor is unnecessary.

In operation S130, the drawing image processing apparatus reads stroke information by matching the movement of the drawing means recognized in operation S120 with the drawing pattern previously learned in operation S110. Even if the same trajectory line is drawn, the direction or angle of holding the pen is different for each user.Each user can estimate the user's pen pressure from the learned EMG data, and after estimating the pen / brush area from the acquired image, The state of the pen can be estimated from the depth image for.

More specifically, in step S130 of reading the stroke information, at least two or more motion information of the state value, the movement and the speed of the drawing means is calculated from the recognized movement of the drawing means, and the calculation from the previously learned drawing pattern It is preferable to return the stroke information reflecting the pen pressure by searching for a value matching the motion information.

In operation S140, the drawing image processing apparatus outputs a drawing image corresponding to the stroke information in real time in response to the drawing motion of the user. That is, based on the computing power of the current digital processor, steps S120 to S140 may be processed in real time, and operate as if the drawing image is drawn at the end of the drawing means held by the user in response to the user's natural drawing motion. Done. To this end, the means for outputting the drawing image preferably recognizes the end of the drawing means held by the user using a depth camera or a sensor, and displays the previously generated drawing image corresponding to the position of the recognized end.

The projector 60A or the display device 60B is provided as display means for outputting an image on a normal plane. The virtual canvas space is output using the display means 60A or 60B, and then the NUX is provided to the user by displaying a drawing image output in real time in response to the user's drawing motion in the virtual canvas space.

In addition, a depth camera 10 for detecting a user's drawing motion is installed, and an EMG sensor 20 for detecting a pen pressure is attached to a part of a user's hand. The projector 60A and the depth camera 10 may be installed at a position that does not affect the user, for example, about 1.5 m. The installed depth camera 10 can recognize the movements of the user's hand and pen / brush, and the electrostatic pressure sensor 20 measures the pressure property by measuring how much force the user uses when drawing. You can judge.

As described above with reference to FIG. 2, in view of implementation, a processor 30 to a controller, for example, a computer, which drives software for implementing the drawing image processing method, is used, but the computer and the projector 60 and the depth camera ( 10) Connect it and place it face up on the desk. When the projection screen is projected on the desk through the connected projector 60 and the drawing program is executed, the depth camera 10 starts to operate.

The depth camera 10 recognizes a pen / brush held by the user and switches to a drawing mode in which a user can draw or write, and the user uses a pen / brush on a canvas or white paper to draw a drawing motion. Will draw through. If the user takes a drawing motion that draws or writes on paper, it recognizes this and creates a digital image based on the pen's moving position, which is recognized as a connected line in vector form, and projects it on the screen on the desk.

When the user draws with the pen / brush, the pen pressure is compared based on the depth data of the height of the pen, and if necessary, the data (50) pre-trained using the EMG sensor to determine the user's EMG. It can be estimated. In addition, not only the user's drawing function, but also the gesture recognition function, a simple gesture can be used to quickly change the color and switch to the eraser function.

As described above with reference to FIG. 1, embodiments of the present invention should be preceded by a process of learning a user's drawing pattern in advance. More specifically, this process is to obtain a three-dimensional image of the user's actual drawing using the depth camera 10, the user's actual using an electromyogram (EMG) sensor attached to the user's hand The pressure of the hand corresponding to the drawing is measured and converted into the pressure of the drawing means, and matching the state values, movements, speeds, and pressures of the drawing means according to the three-dimensional image obtained, and storing them together with the stroke shape of the corresponding time point. desirable.

For this pre-learning, the user performs drawing or writing with a pen while wearing an EMG sensor, and collects EMG data and depth data (including angles and postures of pens / brushes) of the user to a personalized drawing pattern. Build a learning database (50). For example, the pen / brush state data of one frame may be configured as a data set as shown in Equation 1 below.

Where Angle is the pen angle and Velocity is the pen's velocity vector. n represents the current frame and n-1 represents the previous frame.

The learning progresses by matching the EMG data with this state value of the pen / brush. Learning is performed by matching changes in EMG data values according to differences in pen / brush state values in two consecutive frames. After the learning is performed, the pen pressure intended by the user may be estimated without the EMG sensor based on the pen / brush state that changes as the user manipulates the pen / brush.

In summary, the drawing pattern of the user is stored in a plurality of frames for the actual drawing, and preferably includes a time series change of the state value and the pen pressure of the drawing means included in each frame. In addition, the drawing pattern of the user may be stored in a differentiated manner for each individual user as unique identification information for the user by combining at least two or more of a state value, a movement, a speed, and a pressure of the drawing means. Inquiry 50 helps to recognize an individual's unique stroke.

Referring to [A] of FIG. 4, first, a depth value of a canvas space (for example, may be a desk plane) is stored through depth data obtained through the depth camera 10. Using this, only the depth value higher than the reference depth value is later selected and the remaining information is discarded.

Referring now to [B] of FIG. 4, when the user's hand enters the recognition range of the depth camera 10, the depth data of the hand part is sensed, and the 2D image data of the corresponding position is analyzed to analyze the 2D image data. The color distribution can be determined and recognized as a hand region.

Referring to FIG. 5, the movement and type of a pen / brush may be recognized by utilizing 2D and 3D image data of an object other than a hand at a position where a user's hand is closest to the canvas. When the lower part of the hand is not visible, the area on the hand is checked to calculate the distribution of the 3D data, and if the object forms a cylindrical shape, the pen / brush is judged. Depending on the angle of the cylinder and the height of the hand, you can find the location of the lower part of the invisible pen. In addition, the angle of the pen can be measured in this manner, and the stroke effect of the pen / brush can be implemented according to the angle.

Meanwhile, 2D data corresponding to 3D data obtained from the upper and lower regions of the pen may be acquired. The 2D data thus obtained is used to compare the pens / brushes stored in the reference database to estimate the type and select the most similar pen / brush stroke effect. From this, it is possible to create a natural picture regardless of the user using any kind of pen or brush.

In summary, embodiments of the present invention extract the 3D image and the 2D image of the drawing means of the user from the drawing motion of the user input using a depth camera, respectively, and extract the extracted 3D image and the 3D image. The movement of the drawing means can be recognized by calculating a tilt angle of the drawing means, whether the drawing means is in contact with the plane corresponding to the virtual canvas, and a distance by matching a two-dimensional image. Such drawing means may be in the form of a pen or brush or even be a rod, and identifying the type of the drawing means using the unique information contained in the two-dimensional image of the drawing means. It is possible.

FIG. 6 is a diagram for describing a method of measuring a user's pen pressure, and illustrates a situation in which EMG data of a finger muscle is measured during a drawing of a user using an EMG sensor 20 formed in a ring shape.

First, EMG data of a finger at the time of writing and drawing is collected for a plurality of users. The EMG data may be distinguished according to the stationary state and the writing state of the muscle, and the data may be classified according to the gender and age of the user. The data collected from the user is classified through a classification algorithm such as a support vector machine (SVM).

Now you can distinguish between the less power-intensive and the less-powerful situations for a user, and create a trained database in this way. However, when used by an actual user, it is desirable to first perform a user's writing operation and correct the accuracy of EMG data. Since there is a difference in muscle mass for each person, the measured EMG data of the user may be compared with the EMG data in the pre-established database before correction of the drawing image.

FIG. 7 is a block diagram illustrating an apparatus 700 for processing a drawing image based on NUI / NUX according to an embodiment of the present invention, and corresponds to each step of the drawing image processing method described above with reference to FIG. 1. It includes a functional block. Therefore, in order to avoid duplication of description, the operation of each component will be outlined here based on the connection structure of hardware.

The depth camera 10 is a means for receiving an actual drawing image of the user or a drawing motion of the user, and an electromyogram (EMG) sensor 20 is attached to the user's hand to It is a means of measuring the pressure of the hand corresponding to the actual drawing. The values measured from these

devices

10, 20 are provided to the processor 30.

The memory 40 stores a program for generating a drawing image corresponding to a drawing motion of a user, and includes at least one processor 30 to drive a program for generating the drawing image.

At this time, the program stored in the memory 40, the input of the actual drawing image of the user and the pressure of the hand in the drawing to learn the drawing pattern of the user in advance, the input of the user's drawing motion of the drawing means of the user hold Recognizes a movement, reads stroke information by matching the recognized movement of the drawing means with the previously learned drawing pattern, and in real time a drawing image corresponding to the stroke information in response to the drawing motion of the user; Contains the commands to print.

The program stored in the memory 40 acquires a three-dimensional image of the actual drawing image of the user, converts the pressure of the hand into a pressure of the drawing means, and a state value of the drawing means according to the obtained three-dimensional image. It is preferable to learn the drawing pattern of the user in advance by matching the movement, the speed, and the pen pressure with the stroke shape of the corresponding viewpoint. In addition, the drawing pattern of the user is stored for each of a plurality of frames (frames) for the actual drawing, and may include a time series change of the state value and the pen pressure of the drawing means included in each frame. Further, the drawing pattern of the user may be differentiated and stored for each individual user as unique identification information for the user by combining at least two of a state value, a movement, a speed, and a pressure of the drawing means.

The program stored in the memory 40 extracts a 3D image and a 2D image of the drawing means of the user from the drawing motion of the user, and matches the extracted 3D image and the 2D image. It is preferable to recognize the movement of the drawing means by calculating the inclination angle of the drawing means, whether the drawing means is in contact with the plane corresponding to the virtual canvas and the distance. Here, the drawing means has a form of a pen or a brush, and the program stored in the memory identifies the type of the drawing means by using unique information contained in a two-dimensional image of the drawing means. can do.

The program stored in the memory 40 may calculate motion information of at least two of a state value, a movement, and a speed of the drawing means from the recognized movement of the drawing means, and calculate the motion information from the previously learned drawing pattern. It is preferable to read the stroke information by searching for a matching value and returning stroke information reflecting the pen pressure.

Furthermore, the drawing image processing apparatus 700 according to an embodiment of the present invention further includes a display means 60 for outputting a virtual canvas space, wherein the display means 60 is a drawing output in real time. An image may be displayed in the virtual canvas space. In addition, the drawing image displayed at this time is preferably displayed corresponding to the position of the end of the drawing means held by the user.

Hereinafter, a method of applying the above-described digital drawing image processing technique to a digital fragment image processing technique will be described.

FIG. 8 is a diagram illustrating an environment for implementing a fragment image and an accompanying device according to another embodiment of the present invention, and assumes a situation similar to that of FIG. 2.

The user performs an action of carving a virtual sculpture object using a carving means, for example, a carving knife, and photographs it using a depth camera 10 to recognize a motion. Unlike in the case of FIG. 2, in FIG. 8, the display means 60A (e.g., beam project) is removed and a head mounted display (HMD) or augmented reality (AR) is implemented according to implementation needs. ) It is possible to output the fragment image generated by replacing with a device such as glasses.

Referring to FIG. 8, the hand gestures when the user sculpts using the engraving means are more diverse than the hand gestures when drawing. However, since the carving means also have characteristics such as a pen or a brush, it is possible to easily recognize based on the type information of the previously learned carving means. However, it is necessary to determine the direction of the engraving means from various angles. The direction of the engraving means can be determined by the color of the position corresponding to the end of the knife (knife), considering the material of the end of the engraving means can be recognized according to the color of the metal material. The remainder of the engraving means may similarly adopt the method previously employed in drawing recognition. However, if the user recognizes in advance what kind of engraving means is selected through the camera, the accuracy of the recognition may be increased. Once the type of engraving means has been identified, the resulting blade shape can predict the engraving marks that will be formed on the engraving target.

The sensing of the hand / finger pressure through the EMG sensor 20 is slightly different from the drawing recognition. In the case of the fragment image processing, it is necessary to more precisely correlate the correlation between the moving speed of the user's hand and the pressure. For example, the faster the user moves his or her hand, the more pressure will have to be reflected in the engraving. On the contrary, when the engraving is performed in a small motion, it is necessary to process the pressure applied to the engraving object more precisely based on the pressure data measured from the finger. Particularly, in the case of engraving, when a very small operation is performed at a speed below a reference value, it is preferable to mainly use data measured by the EMG sensor 20.

In operation S910, the engraving image processing apparatus receives the actual sculpting image of the user and the pressure of the hand among the pieces, and learns the engraving pattern of the user in advance for each type of engraving means. In this process, a 3D image of the actual engraving behavior of the user is obtained by using a depth camera, and an electromyogram (EMG) sensor attached to the user's hand corresponds to the actual carving behavior of the user. The pressure of the hand is measured and converted into the pressure of the engraving means, and the state values, movements, speeds and pressures of the engraving means according to the three-dimensional image obtained are matched together with the shape of the engraving marks of the corresponding point of time according to the type of the engraving means. Save it. Here, the engraving pattern of the user is stored for each of a plurality of frames (frame) for the actual engraving behavior, it is preferable to include a time-series change in the state value and pressure of the engraving means included in each frame.

In operation S920, the engraving image processing apparatus receives a sculpting motion of the user, and recognizes the type of carving means and the movement of the carving means held by the user from the input carving motion. In this process, the 3D image and the 2D image of the engraving means held by the user are respectively extracted from the user's engraving motion input using a depth camera, and the extracted 3D image and the 2D image are matched. The inclination angle of the engraving means, whether or not the contact between the engraving means and the reference position corresponding to the virtual engraving object is calculated. Here, the carving means is in the form of a carving knife (carving knife), it is preferable to identify the type of the carving means using the unique information contained in the two-dimensional image of the carving means.

In operation S930, the engraving image processing apparatus reads carving information corresponding to the type of engraving means by matching the movement of the carving means recognized in operation S920 with the carving pattern previously learned in operation S910. do. In this process, the motion information of at least two of the state value, the movement and the speed of the engraving means is calculated from the recognized movement of the engraving means, and the value matching the calculated motion information is retrieved from the previously learned engraving pattern. Carving information corresponding to the type of engraving means and reflecting the pressure is returned. At this time, in the engraving image processing, by adjusting the pressure or the size of the engraving mark in proportion to the speed of the engraving means, it is possible to simulate the engraving marks imprinted on the object to be carved by the engraving means more similar to the actual engraving behavior. Do.

In operation S940, the fragment image processing apparatus outputs a fragment image corresponding to the fragment mark information in real time in response to the fragment motion of the user.

Furthermore, the engraving image processing apparatus outputs a virtual sculpture object by using display means such as a head mounted display device or an augmented reality glasses, and displays the sculpture image output in real time on the virtual sculpture object through step S940. You may.

Meanwhile, embodiments of the present invention can be implemented by computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored.

Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. And functional programs, codes and code segments for implementing the present invention can be easily inferred by programmers in the art to which the present invention belongs.

The present invention has been described above with reference to various embodiments thereof. Those skilled in the art will understand that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

As described above, when using a conventional tablet or a digitizer, a user needs a dedicated electronic pen or special sensing means, so the user may feel different from actually drawing. In contrast, the embodiments of the present invention described above, by recognizing the drawing motion or sculpture motion that is most familiar to humans through the NUI, it is possible to provide the user with the feeling of actually drawing or sculpting. In particular, by implementing digital drawing or digital sculpture based on NUI / NUX, it enables quick and familiar adaptation and easy operation to non-professional ordinary people including most ages.

Claims

Learning the drawing pattern of the user in advance by receiving an actual drawing image of the user and pressure of a hand during drawing;

Receiving a drawing motion of the user and recognizing a movement of the drawing means held by the user from the input drawing motion;

Reading stroke information by matching the recognized movement of the drawing means with the previously learned drawing pattern; And

And outputting a drawing image corresponding to the stroke information in real time in response to the drawing motion of the user.
The method of claim 1,

Pre-learning the drawing pattern of the user,

Acquiring a 3D image of the actual drawing of the user using a depth camera;

Measuring pressure of a hand corresponding to the actual drawing of the user using an electromyogram (EMG) sensor attached to the user's hand and converting the pressure into a pen pressure of a drawing means; And

And matching a state value, a movement, a speed and a pen pressure of the drawing means according to the obtained 3D image, and storing the state value together with the stroke shape of the corresponding view point.
The method of claim 2,

The drawing pattern of the user,

And a time series change of the pressure value and the state value of the drawing means included in each frame, the frame being stored for each actual frame.
The method of claim 2,

The drawing pattern of the user,

And at least two of a state value, a movement, a speed, and a pressure of the drawing means are differentiated and stored for each individual user as unique identification information for the user.
The method of claim 1,

Recognizing the movement of the drawing means,

Extracting a 3D image and a 2D image of the drawing means held by the user from the input drawing motion of the user using a depth camera; And

Calculating the inclination angle of the drawing means, whether the drawing means is in contact with a plane corresponding to the virtual canvas, and the distance by matching the extracted 3D image and the 2D image. Image processing method.
The method of claim 5,

The drawing means has the form of a pen or brush,

And identifying the type of the drawing means by using the unique information included in the two-dimensional image of the drawing means.
The method of claim 1,

The reading of the stroke information may include:

Calculating motion information of at least two of a state value, a movement, and a speed of the drawing means from the recognized movement of the drawing means; And

And searching for a value matching the calculated motion information from the previously learned drawing pattern, and returning stroke information reflecting a pen pressure.
The method of claim 1,

And outputting a virtual canvas space by using display means, and displaying the drawing image output in real time in the virtual canvas space.
Receiving a user's actual sculpting image and the pressure of a hand in the sculpture and learning the engraving pattern of the user in advance by the type of engraving means;

Receiving a sculpting motion of the user and recognizing the type of sculpting means held by the user and the movement of the sculpting means from the input sculpting motion;

Reading piece carving information according to the type of carving means by matching the recognized movement of the carving means with the previously learned carving pattern; And

And outputting a fragment image corresponding to the fragment mark information in real time in response to the fragment motion of the user.
The method of claim 9,

Pre-learning the engraving pattern of the user,

Acquiring a 3D image of the actual carving behavior of the user by using a depth camera;

Measuring the pressure of the hand corresponding to the actual engraving behavior of the user by using an electromyogram (EMG) sensor attached to the user's hand and converting the pressure into a pressure of the engraving means; And

And matching the state values, motions, speeds, and pressures of the engraving means according to the obtained three-dimensional image, and storing them together with the shape of the engraving marks of the corresponding point of time according to the type of the engraving means.
The method of claim 10,

The engraving pattern of the user,

And a time series change of pressure and state values of engraving means included in each frame.
The method of claim 9,

Recognizing the movement of the engraving means,

Extracting a three-dimensional image and a two-dimensional image of the engraving means held by the user from the engraving motion of the user input using the depth camera; And

Calculating the inclination angle of the engraving means, whether the engraving means is in contact with a reference position corresponding to the virtual engraving object, and matching the extracted three-dimensional image and the two-dimensional image. Treatment method.
The method of claim 12,

The engraving means is in the form of a carving knife,

And identifying the type of engraving means by using the unique information contained in the two-dimensional image of the engraving means.
The method of claim 9,

Reading the piece mark information,

Calculating at least two motion information of the state value, the movement and the speed of the carving means from the recognized movement of the carving means; And

And searching for a value matching the calculated motion information from the previously learned piece pattern, and returning piece carving information corresponding to a type of piece means and reflecting pressure.
The method of claim 14,

And adjusting the pressure or the size of the engraving mark in proportion to the speed of the engraving means.
The method of claim 9,

And outputting a virtual piece object using a display means, and displaying the piece piece image output in real time on the virtual piece object.