KR102664254B1 - An apparatus for recognizing hand signals based on vision artificial intelligence and a method for recognizing hand signals using the same - Google Patents

Abstract

One embodiment of the present invention provides a technology for implementing a hand signal recognition device with a simple configuration by minimizing the configuration in addition to an imaging device such as a vision sensor. A vision artificial intelligence-based hand signal recognition device according to an embodiment of the present invention includes an imaging unit that captures a motion of a user's hand and generates a captured image; a shape detection module that receives the captured image from the imaging unit and generates shape detection data, which is data generated by recognizing the shape and change of the user's hand; a trajectory detection module that receives the captured image from the imaging unit and generates trace detection data, which is data generated by recognizing the trace of the user's hand; and a learning module that receives the shape detection data from the shape detection module, receives the trajectory detection data from the trajectory detection module, and performs learning on the shape detection data or the trajectory detection data.

Description

Vision artificial intelligence-based hand signal recognition device and hand signal recognition method using the same {AN APPARATUS FOR RECOGNIZING HAND SIGNALS BASED ON VISION ARTIFICIAL INTELLIGENCE AND A METHOD FOR RECOGNIZING HAND SIGNALS USING THE SAME}

The present invention relates to a vision artificial intelligence-based hand signal recognition device and a hand signal recognition method using the same. More specifically, it relates to a technology for implementing a hand signal recognition device with a simple configuration by minimizing the configuration in addition to an imaging device such as a vision sensor.

As a recent example of research related to existing hand signal recognition, sign-to-speech technology based on ASL (American Sign Language) recognition technology developed by UCLA researchers was announced. The sign-to-speech research case demonstrated a high ASL recognition rate of over 95% by implementing technology to distinguish complex finger shapes by attaching a sensor to acquire a hand signal recognition signal, but it required attachment of a glove-type sensor that is difficult to carry at all times. Accordingly, the scope of use is limited. Additionally, Intel's Touchless technology implements the touch function without a separate sensor, but is limited to a technology that replaces touch and click at a location close to the monitor.

In addition, as a domestic commercialization example, the technology owned by VTOUCH has a remote click type that is used for vehicle interior control displays or kiosks, but it is limited to the range of functions such as mouse use and click, volume control gesture recognition, and complex characters. In order to input information, you must click exactly on the corresponding position on the virtual keyboard displayed on the screen using a mouse-type control, so there is a limitation that typos often occur due to position errors in tracking the mouse cursor.

In Republic of Korea Patent No. 10-2121654 (title of the invention: deep learning-based automatic gesture recognition method and system), the gesture recognition system includes the steps of extracting a number of outlines from an input image; A step of the gesture recognition system generating learning data by normalizing outline information constituting each of the outlines; A method is disclosed, including a step of the gesture recognition system training an artificial intelligence model for gesture recognition using the generated learning data, wherein the outlines are capable of overlapping.

Republic of Korea Patent No. 10-2121654

The purpose of the present invention to solve the above problems is to implement a hand signal recognition device with a simple configuration by minimizing the configuration other than an imaging device such as a vision sensor.

The technical problem to be achieved by the present invention is not limited to the technical problem mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

The configuration of the present invention for achieving the above object includes: an imaging unit that captures the motion of the user's hand and generates an image; a shape detection module that receives the captured image from the imaging unit and generates shape detection data, which is data generated by recognizing the shape and change of the user's hand; a trajectory detection module that receives the captured image from the imaging unit and generates trace detection data, which is data generated by recognizing the trace of the user's hand; and a learning module that receives the shape detection data from the shape detection module, receives the trajectory detection data from the trajectory detection module, and performs learning on the shape detection data or the trajectory detection data.

In an embodiment of the present invention, a plurality of imaging units are formed, and the captured image may include a visible light image that is an image by visible light imaging, an infrared image that is an image by infrared imaging, or a three-dimensional image.

In an embodiment of the present invention, the shape detection module includes a shape tracking unit that receives the captured image from the imaging unit and generates data by recognizing the shape and change of the user's hand; and a shape determination unit that receives data from the shape tracking unit, determines whether the user's hand movement corresponds to a hand signal, classifies it, and generates data.

In an embodiment of the present invention, the shape detection module may further include a shape detection unit that receives data from the shape determination unit and performs learning on the data to generate the shape detection data.

In an embodiment of the present invention, the shape tracking unit may perform convex tracking using a convex-hull algorithm.

In an embodiment of the present invention, the learning module includes: a tracking unit that performs real-time learning on images of the shape detection data or the trajectory detection data to infer tracking coordinates of a reaction map; and a data accumulation unit that accumulates data received from the tracking unit.

In an embodiment of the present invention, the tracking unit includes a feature extraction unit that extracts features from an image of the shape detection data or the trajectory detection data; and a learning unit that learns the data transmitted from the feature extraction unit using an artificial neural network and generates the response map for the image.

In an embodiment of the present invention, the tracking unit includes: a correction unit that performs correction on the response map transmitted to the learning unit; and a feature enhancement unit that performs feature enhancement processing on the image object using the response map delivered from the correction unit.

In an embodiment of the present invention, the trajectory detection module receives the captured image from the imaging unit and generates data by recognizing the movement trace of the user's hand formed on a virtual canvas, which is a predetermined three-dimensional area. a trajectory tracking unit; and a trajectory detection unit that receives data from the trajectory tracking unit and performs learning on the data to generate the trajectory detection data.

In an embodiment of the present invention, the hand signal display unit may further include a hand signal display unit that displays an image using the generated shape detection data or the trajectory detection data on the screen.

The configuration of the present invention for achieving the above object includes: a first step of capturing the user's hand movements in a plurality of imaging units and collecting the captured images; A second step in which the captured image is transmitted to the shape detection module and the trajectory detection module; A third step in which the shape detection data is formed in the shape detection module and the trajectory detection data is formed in the trajectory detection module; And a fourth step in which the image based on the shape detection data or the image based on the trajectory detection data is displayed on the hand signal display unit.

In an embodiment of the present invention, a fifth step of performing learning in the shape detection module and the trajectory detection module using learning data generated by the learning module may be further included.

The effect of the present invention according to the above configuration is that it is convenient and economical because hand signals are recognized through imaging by an imaging device such as a vision sensor without the need to attach a sensor-attached input device to the user's hand.

In addition, the effect of the present invention is that it can be used as an input device even at a distance from the imaging device and can be used as a control technology for a large display, increasing utilization.

Additionally, the effect of the present invention is that there is no limit to the number of types of input signals, such as simple gestures to complex text or signature information, so various information can be input quickly.

The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description or claims of the present invention.

Figure 1 is a schematic diagram of a shape detection module according to an embodiment of the present invention.
Figure 2 is a schematic diagram of a shape tracking unit and a shape determination unit according to an embodiment of the present invention.
Figure 3 is a schematic diagram of a shape detection unit according to an embodiment of the present invention.
Figure 4 is a schematic diagram of a trajectory detection module according to an embodiment of the present invention.
Figure 5 is a schematic diagram of a trajectory tracking unit according to an embodiment of the present invention.
Figure 6 is a schematic diagram of a hand signal display unit according to an embodiment of the present invention.
Figure 7 is a schematic diagram of a trajectory determination unit according to an embodiment of the present invention.
Figure 8 is a schematic diagram of the configuration of a hand signal recognition device according to an embodiment of the present invention.
Figure 9 is a schematic diagram of a tracking unit according to an embodiment of the present invention.
Figure 10 is a conceptual diagram of hand trace tracking according to an embodiment of the present invention.
Figures 11 and 12 are images of a hand signal recognition test using a hand signal recognition device according to an embodiment of the present invention.

Hereinafter, the present invention will be described with reference to the attached drawings. However, the present invention may be implemented in various different forms and, therefore, is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts unrelated to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected (connected, contacted, combined)" with another part, this means not only "directly connected" but also "indirectly connected" with another member in between. "Includes cases where it is. Additionally, when a part is said to “include” a certain component, this does not mean that other components are excluded, but that other components can be added, unless specifically stated to the contrary.

The terms used in this specification are merely used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Hereinafter, the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a schematic diagram of the shape detection module 100 according to an embodiment of the present invention, and Figure 2 is a schematic diagram of the shape tracking unit 110 and the shape judgment unit 120 according to an embodiment of the present invention. , Figure 3 is a schematic diagram of the shape detection unit 130 according to an embodiment of the present invention.

In addition, Figure 4 is a schematic diagram of the trajectory detection module 200 according to an embodiment of the present invention, Figure 5 is a schematic diagram of the trajectory tracking unit 210 according to an embodiment of the present invention, and Figure 6 is a schematic diagram of the trajectory detection module 200 according to an embodiment of the present invention. This is a schematic diagram of the hand signal display unit 420 according to an embodiment of the invention. And, Figure 7 is a schematic diagram of a trajectory determination unit according to an embodiment of the present invention.

As shown in Figures 1 to 7, the hand signal recognition device of the present invention includes an imaging unit 410 that captures the motion of the user's hand and generates an image; A shape detection module 100 that receives a captured image from the imaging unit 410 and generates shape detection data, which is data generated by recognizing the shape and change of the user's hand; a trajectory detection module 200 that receives an image from the imaging unit 410 and generates trajectory detection data, which is data generated by recognizing the trajectory of the user's hand; and a learning module that receives shape detection data from the shape detection module 100, receives trajectory detection data from the trajectory detection module 200, and performs learning on the shape detection data or trajectory detection data.

A plurality of imaging units 410 are formed, and the captured image may include a visible light image that is an image by visible light imaging, an infrared image that is an image by infrared imaging, or a three-dimensional image.

To capture a visible light image, the imaging unit 410 may be equipped with a digital camera or an RGB camera sensor. Additionally, the imaging unit 410 may be equipped with an infrared (IR) camera to capture an infrared image. Additionally, in order to capture a 3D image, the imaging unit 410 may be equipped with a 3D camera.

One imaging unit 410 can capture visible light images, infrared images, and three-dimensional images simultaneously or separately, and one imaging unit 410 among the plurality of imaging units 410 captures visible light images. And another imaging unit 410 can capture an infrared image, and another imaging unit 410 can capture a three-dimensional image.

In addition, a plurality of imaging units 410 as described above may be formed, and the shape detection data or the trace of the user's hand generated by recognizing the shape and change of the user's hand in the image captured by each imaging unit 410 Trajectory detection data generated through recognition can be transmitted to the learning module and learned.

Image data from a plurality of captured domains can be referred to as multi-domain data, and multi-domain data acquired by a plurality of imaging units 410 is generated by the shape detection module 100 and the trajectory detection module 200. ) can be transmitted.

The shape detection module 100 includes a shape tracking unit 110 that receives an image from the imaging unit 410 and generates data by recognizing the shape and change of the user's hand; And it may include a shape determination unit 120 that receives data from the shape tracking unit, determines whether the user's hand movement corresponds to a hand signal, classifies it, and generates data.

In addition, the shape detection module 100 may further include a shape detection unit 130 that receives data from the shape determination unit 120 and performs learning on the data to generate shape detection data.

The shape tracking unit 110 may perform convex tracking using a convex-hull algorithm. As shown in Figures 1 and 2, convex tracking uses the convex-hull algorithm to capture three-dimensional characteristics of information about the user's hand motion input from a plurality of imaging units 410 (multiple domains). This may be done by analyzing and calculating the position and speed of each point of the user's hand that generates the hand signal.

Specifically, each of the above-described two-dimensional visible light images, infrared images, or three-dimensional images may be included as an image sequence unit consisting of a combination of at least one or more frames, and a plurality of such imaging units ( When the captured image is transmitted from 410 to the shape tracking unit 110, the shape tracking unit 110 can obtain the hand area using each captured image.

In addition, the shape tracking unit 110 obtains a convex-hull including all of the hand areas obtained as above, extracts boundary points from the convex hull of the hand area, and calculates the three-dimensional position and velocity of each of the boundary points. can be calculated.

In addition, the shape tracking unit 110 can track and recognize the user's hand movements using the change and degree of change in the three-dimensional position and speed of each of the boundary points described above. Data on the hand movement recognized in this way may be transmitted to the shape determination unit 120.

The shape determination unit 120 performs machine learning using an artificial neural network using the data transmitted from the shape tracking unit 110, and can determine whether the user's hand movement corresponds to a hand signal including the user's intention. . In one embodiment, when a user performs an action such as taking a picture on the screen, it may be determined whether the hand action corresponds to a hand signal.

Here, the artificial neural network may include, but is not limited to, CNN (Convolution Neural Network), DNN (Deep Neural Network), RNN (Recurrent Neural Network), etc.

And, as described above, the data classified as corresponding to a hand signal in the shape determination unit 120 is transmitted to the shape detection unit 130, and the shape detection unit 130 uses an artificial intelligence multi-stage detection/classification model to determine the shape of the hand signal. This allows recognition, and data can be generated to form shape detection data.

Here, the shape detection unit 130 may use a two-stage detection model (2-Stage Detector). Specifically, the shape detection unit 130 may use ResNet, RCNN series (R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN...) etc. can be used. In Figure 3, details about the image of shape detection data generated by detection/classification by the shape detection unit 130 are disclosed.

As described above, the method used by the shape detection module 100 is based on the hand signal shape recognition mode (model), where the user inputs numbers/characters at an instantaneous speed by simply memorizing the form of the hand signal such as ASL (American Sign Language). This may be a possible method.

The hand signal recognition device of the present invention may further include a hand signal display unit 420 that displays an image using the generated shape detection data or trajectory detection data on the screen. As shown in Figure 1, when a user performs an ASL hand signal operation using a shape hand signal, which is a hand signal based on the shape of the hand, along with the user's hand signal operation, numbers, letters, etc. that the hand signal represents are displayed on the screen of the hand signal display unit 420. It can be displayed in .

The screen display based on trajectory detection data in the hand signal display unit 420 will be described in detail below.

The trajectory detection module 200 receives the captured image from the imaging unit 410 and generates data by recognizing the movement trace of the user's hand formed on the virtual canvas 201, which is a predetermined three-dimensional area. tracking unit 210; And it may include a trajectory detection unit 220 that receives data from the trajectory tracking unit 210, performs learning on the data, and generates trajectory detection data.

The trajectory tracking unit 210 can create a virtual canvas 201 space, which is a virtual three-dimensional space area, in front of the imaging unit 410 based on a point on the imaging unit 410 that is close to the user. A predetermined three-dimensional space can be set as a virtual canvas 201.

As shown in FIGS. 4 and 6, one of the plurality of imaging units 410 may be formed by combining one of the imaging units 410 and the hand signal display unit 420, and the front of the imaging unit 410 and the hand signal display unit A predetermined three-dimensional space formed away from the front of 420 may be set as the virtual canvas 201.

Additionally, each point of the virtual canvas 201 can be set to three-dimensional coordinates, and when the user's hand moves within the virtual canvas 201, the trajectory tracking unit 210 recognizes the user's hand and tracks the user's hand movement. It can be recognized by tracking the movement trajectory of the user's hand.

The trajectory tracking unit 210 is also equipped with a convex-hull algorithm and, like the shape tracking unit 110, can acquire the hand area using each captured image and includes all acquired hand areas. You can obtain a convex-hull, extract boundary points from the convex hull of the hand area, and calculate the 3D position and velocity of each boundary point.

At this time, the user can create a trajectory by moving the hand while fixing the shape of the hand, and the hand signal display unit 420 is the three-dimensional coordinate point closest to the hand signal display unit 420 among the three-dimensional coordinates of the above-described boundary points. can be set as the trajectory center point. Accordingly, the user's fingertip, which is most forward, is set as the trajectory center point, and the trajectory tracking unit 210 can recognize the trajectory of the trajectory center point and generate data.

In addition, the trajectory tracking unit 210 recognizes the trajectory of the trajectory center point using the change and degree of change in the three-dimensional position and speed of the trajectory center point, and in this way, tracks the hand signal trajectory, which is the trace of the user's hand. In this way, data about the recognized matter can be transmitted to the trajectory detection unit 220.

In the trajectory detection unit 220, the trajectory of the hand signal performed in the space of the virtual canvas 201 can be processed in a form similar to Point Cloud data and input into a classification model, and this classification model can be used as a classification model based on the previously learned hand signal trajectory. Based on the information, the meaning of the trajectory hand signal, which is a hand signal drawn in the space of the virtual canvas 201, can be inferred.

Specifically, the trajectory detection unit 220 learns the data transmitted from the trajectory detection unit 220 using an artificial neural network such as CNN and recognizes it using a multi-stage detection/classification model, thereby forming trajectory detection data. .

In Figure 7, the trajectory detection data detected/classified and generated by the trajectory detection unit 220, details on the images of such trajectory detection data, and the configuration (56K images) in which the trajectory detection data are stored are disclosed. there is.

The hand signal display unit 420 displays an image of the user's hand captured by the imaging unit 410, and at the same time, the space of the virtual canvas 201 may be expressed as an image in an opaque color. In addition, the user can check whether the user's hand is located within the space of the virtual canvas 201 through the screen of the hand signal display unit 420. When the user creates a hand signal with his hand, the user's hand is displayed on the screen of the hand signal display unit 420. The trajectory can be displayed.

As described above, in the case of forming trace detection data, the hand signal recognition device of the present invention operates in the hand signal trace tracking mode (model), so that the user does not need to memorize the above-mentioned ASL and Gatti formal symbols, and commonly used You can replace input devices such as a keyboard by inputting the shape of the language into the air.

Figure 8 is a schematic diagram of the configuration of a hand signal recognition device according to an embodiment of the present invention, Figure 9 is a schematic diagram of a tracking unit 310 according to an embodiment of the present invention, and Figure 10 is an embodiment of the present invention. This is a conceptual diagram of hand trace tracking according to an example.

Figure 8 (a) shows software (S/W) when data is accumulated using initial learning data in the hand signal recognition device of the present invention, and Figure 8 (b) shows shape detection data and trajectory This shows the software (S/W) in the hand signal recognition device of the present invention in an advanced form in which learning data increases as the amount of detection data increases.

And, Figure 8(c) shows the learning of the learning module for shape detection data, and Figure 8(d) shows real-time hand signal tracking (Dynamic convex tracking) for the trajectory detection module 200.

As shown in Figure 8, as time passes, shape detection data and trajectory detection data can be formed by the hand signal recognition device of the present invention in an advanced form as shown in (b) of Figure 8.

The learning module includes a tracking unit 310 that performs real-time learning on images of shape detection data or trajectory detection data to infer tracking coordinates of a reaction map; and a data accumulation unit 320 that accumulates data received from the tracking unit 310.

And, the tracking unit 310 includes a feature extraction unit 311 that extracts features from images of shape detection data or trajectory detection data; a learning unit 312 that learns the data transmitted from the feature extraction unit 311 using an artificial neural network and generates a response map for the image; A correction unit 313 that performs correction on the response map transmitted to the learning unit 312; and a feature enhancement unit 314 that performs feature enhancement processing on the image object using the response map transmitted from the correction unit 313.

As shown in Figures 8 and 9, shape detection data or trajectory detection data is transmitted and input to the feature extraction unit 311, and the feature extraction unit 311 has a built-in DiMP (Discriminative Model Prediction) tracker, and The same DiMP tracker may be equipped with a ResNet-based discriminative model prediction feature extraction model, that is, a pre-learning feature extractor based on the ResNet50 backbone structure.

The learning unit 312 can perform calculations using an artificial neural network on the data transmitted from the feature extraction unit 311 and generate a response map, where the artificial neural network is a CNN (Convolution Neural Network). , DNN (Deep Neural Network), RNN (Recurrent Neural Network), etc. may be used, but are not limited thereto.

The correction unit 313 may have a built-in IoU-Net (Intersection over Union NETwork) model, and the correction unit 313 infers the maximum value of the reaction map for the received reaction map data and determines the maximum value of the reaction map. Object tracking coordinates can be updated according to the coordinates.

Tracking performance may deteriorate depending on the conditions of the input image (video) only by the operation of the feature extraction unit 311 and the learning unit 312. For example, when the characteristic elements of the object being tracked in the image are reduced or feature information is lacking, the coordinates of the maximum value of the response map are distributed to multiple local coordinates, or the difference between the maximum and minimum values is reduced, along with the reliability. The value may decrease.

Due to this problem, the performance error increases in the continuous object tracking operation of the tracker in the feature extraction unit 311. To overcome these limitations, the response map by the feature extraction unit 311 and the learning unit 312 is used. It is transmitted to the correction unit 313, and is a network learned to predict the IoU (Intersection over Union) between the tracked bounding box and the correct value. A more accurate bounding box can be estimated by performing precise correction of the tracking coordinates.

As described above, the tracking unit 310 includes DDiMP (Depth enhanced DiMP), which operates by fusing two complex sub-network models, the DiMP tracker and the IoU-Net model, to improve the inference accuracy for the tracking coordinates of the reaction map. You can do it.

Data that satisfies the IoU value between the bounding box tracked in the correction unit 313 and the correct answer value can be transmitted to the data accumulation unit 320, and data that does not meet the requirements can be classified as a localization response and transmitted to the feature enhancement unit 314. there is.

The feature enhancement unit 314 performs Reinforced Feature with Dynamic Search Area (RFDSA). Specifically, the feature enhancement unit 314 performs the output coordinates of the response map corrected by the correction unit 313. From this, a region of interest (ROI) can be designated and processing to adaptively enhance image features can be performed on the search region where the object has a high probability of being present.

In this case, the shape information of the object may be distorted due to the difference in quality between the image of the entire image and the image of the search area, so it can be corrected so that the feature enhancement area of the object is naturally processed through dynamic Gaussian masking. there is.

Next, the feature enhancement unit 314 performs image adaptive processing on the data to generate a feature-enhanced feature sequence, and this feature-enhanced sequence is again sent to the feature extraction unit 311. By transmitting the information to and using it to operate the initial tracking unit 310, the efficiency of extracting features of the object and generating tracking coordinates can be improved even if object information is insufficient.

The data accumulation unit 320 automatically accumulates the data transmitted from the tracking unit 310, and for the accumulated data, the data accumulation unit 320 checks the correlation between the input data and the existing data and determines the similarity. Saturation of accumulated data can be prevented by forgetting data higher than the reference value.

Figure 10 (a) shows a screen where a hand signal based on the user's hand trace is input to the trace tracking unit 210, and Figure 10 (b) shows the details on which real-time recognition is performed in the trace tracking part 210. It is an image, and FIG. 10(c) shows that information resulting from the hand signal recognition of the trace tracking unit 210 is displayed on the screen of the hand signal display unit 420.

As shown in Figure 8 (d) and Figure 10, the trajectory detection data is learned by the learning module and used to generate learning data, which is data by the learning module, and is recognized as a hand signal based on the user's hand trace in real time. It is possible to perform input functions such as keyboard input. And, of course, this same function can be performed equally in the case of ASL hand signals.

In addition, in order to implement the two modes (hand signal shape detection/recognition, hand signal trace tracking recognition mode) proposed by the learning module as above, data learning of a multi-stage artificial intelligence detection/classification model and a trajectory signal classification model is performed. The learning data generated accordingly is generated by the multi-domain precise tracking algorithm described above, and can be generated with minimal intervention by the developer.

Such learning data is also continuously transmitted to the shape detection module 100 and the trajectory detection module 200, and the trajectory detection module 200 and the shape detection module 100 that have learned this are used as shown in (a) of Figure 8. The same early version of the hand signal recognition device can be developed into a developed type of hand signal recognition device as shown in (b) of FIG. 8.

Specifically, the learning data is mainly used at times when imaging by the imaging unit 410 is not performed, and each of the shape detection module 100 and the trajectory detection module 200 performs learning using the learning data, and accordingly , the accuracy of each of the shape detection module 100 and the trajectory detection module 200 is improved, and the shape hand signal recognition by the shape detection module 100 and the accuracy of the trajectory hand signal by the trajectory detection module 200 are improved, such hand signals Input accuracy can be improved using .

Figures 11 and 12 are images of a hand signal recognition test using a hand signal recognition device according to an embodiment of the present invention. Specifically, FIG. 11 shows a case where the image capture unit 410 captures an ASL hand signal, and FIG. 12 shows a case where the image capture unit 410 captures a hand signal based on a hand trace.

Figure 11 (a) is about performing a shape hand signal toward the imaging unit 410, and Figure 11 (b) is about a screen displayed for shape hand signal recognition on the hand signal display unit 420. In addition, Figure 12 (a) is about performing a trace hand signal toward the imaging unit 410, and Figure 12 (b) is about the screen displayed for trace hand signal recognition on the hand signal display unit 420. .

As shown in Figure 11, it can be confirmed that the hand signal device of the present invention recognizes each numeric hand signal and displays it accurately on the screen, and as shown in Figure 12, the hand signal device of the present invention recognizes hand signals based on the trace of the hand. You can confirm that it is recognized and displayed in the recognized form on the screen.

Hand signal recognition device of the present invention; And an indoor game system including a game device that performs control by processing information about numbers, letters, or symbols transmitted from a vision artificial intelligence-based hand signal recognition device can be formed.

Specifically, the game device can perform online tour riding, etc., and when the user controls the game device by inputting the above numbers, letters, or symbols into the game device, the present invention recognizes hand signals. The device may be used.

Hereinafter, a hand signal recognition method using the hand signal recognition device of the present invention will be described.

First, in the first step, captured images can be collected by capturing the user's hand movements in the plurality of imaging units 410. And, in the second step, the captured image can be transmitted to the shape detection module 100 and the trajectory detection module 200.

Next, in the third step, shape detection data may be formed in the shape detection module 100, and trajectory detection data may be formed in the trajectory detection module 200. And, in the fourth step, an image based on shape detection data or an image based on trajectory detection data may be displayed on the hand signal display unit 420.

In the fourth step, the shape hand signal, which is an image based on shape detection data as described above, and the trace hand signal, which is an image based on trace detection data, are displayed simultaneously on the screen of the hand signal display unit 420, or in separate areas on the screen. can be displayed.

In the fifth step after the fourth step, learning can be performed in the shape detection module 100 and the trajectory detection module 200 using the learning data generated by the learning module. Specifically, among the components included in each module, in the component that performs learning by artificial neural network, learning can be performed by receiving learning data from the learning module.

The remaining details of the hand signal recognition method of the present invention are the same as those described for the hand signal recognition device of the present invention described above.

When using the hand signal recognition device of the present invention configured as described above, the hand signal is recognized through imaging by an imaging device such as a vision sensor without the need to attach a sensor-attached input device to the user's hand, making it convenient and economical. You can.

In addition, it can be used as an input device even at a distance from the imaging device, and can be used as a control technology for large displays, which has the advantage of increasing usability.

Additionally, there is no limit to the number of types of input signals, including simple gestures to complex text or signature information, so a variety of information can be input quickly.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as single may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the patent claims described below, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention.

100: Shape detection module
110: Shape tracking unit
120: Shape judgment unit
130: Shape detection unit
200: Trajectory detection module
201: Virtual Canvas
210: Trajectory tracking unit
220: Trajectory detection unit
300: Learning module
310: tracking unit
311: Feature extraction unit
312: Learning Department
313: Correction unit
314: Feature enhancement unit
320: Data accumulation unit
410: imaging unit
420: Hand signal display unit

Claims (13)

An imaging unit that captures the motion of the user's hand and generates a captured image;
a shape detection module that receives the captured image from the imaging unit and generates shape detection data, which is data generated by recognizing the shape and change of the user's hand;
a trajectory detection module that receives the captured image from the imaging unit and generates trace detection data, which is data generated by recognizing the trace of the user's hand; and
A learning module that receives the shape detection data from the shape detection module, receives the trajectory detection data from the trajectory detection module, and performs learning on the shape detection data or the trajectory detection data.
The learning module includes a tracking unit that performs real-time learning on the image of the shape detection data or the trajectory detection data to infer the tracking coordinates of the reaction map,
The tracking unit may include a feature extraction unit that extracts features from an image of the shape detection data or the trajectory detection data; and a learning unit that learns the data transmitted from the feature extraction unit using an artificial neural network and generates the response map for the image. a correction unit that performs correction on the response map transmitted to the learning unit; and a feature enhancement unit that performs feature enhancement processing on an image object using the response map delivered from the correction unit.
In claim 1,
A vision artificial intelligence-based hand signal recognition device comprising a plurality of imaging units, wherein the captured image includes a visible light image that is an image by visible light imaging, an infrared image that is an image by infrared imaging, or a three-dimensional image.
In claim 1,
The shape detection module is,
a shape tracking unit that receives the captured image from the imaging unit and generates data by recognizing the shape and change of the user's hand; and
A vision artificial intelligence-based hand signal recognition device comprising a shape determination unit that receives data from the shape tracking unit, determines whether the user's hand movement corresponds to a hand signal, classifies it, and generates data.
In claim 3,
The shape detection module is a vision artificial intelligence-based hand signal recognition device, characterized in that it further includes a shape detection unit that receives data from the shape determination unit and performs learning on the data to generate the shape detection data.
In claim 3,
The shape tracking unit is a vision artificial intelligence-based hand signal recognition device characterized in that it is provided with a convex-hull algorithm to perform convex tracking.
In claim 1,
The learning module is a vision artificial intelligence-based hand signal recognition device, characterized in that it further includes a data accumulation unit that accumulates data received from the tracking unit.
delete delete In claim 1,
The trajectory detection module,
a trajectory tracking unit that receives the captured image from the imaging unit and generates data by recognizing the movement trace of the user's hand formed on a virtual canvas, which is a predetermined three-dimensional area; and
A vision artificial intelligence-based hand signal recognition device comprising a trajectory detection unit that receives data from the trajectory tracking unit, performs learning on the data, and generates the trajectory detection data.
In claim 1,
A vision artificial intelligence-based hand signal recognition device further comprising a hand signal display unit that displays an image using the generated shape detection data or the trajectory detection data on a screen.
A vision artificial intelligence-based hand signal recognition device according to any one of claims 1 to 6, claims 9, and 10; and
An indoor game system comprising a game device that performs control by processing information about numbers, letters, or symbols transmitted from the vision artificial intelligence-based hand signal recognition device.
In the hand signal recognition method using the vision artificial intelligence-based hand signal recognition device of claim 1,
A first step of collecting the captured images by capturing the user's hand movements in a plurality of imaging units;
A second step in which the captured image is transmitted to the shape detection module and the trajectory detection module;
A third step in which the shape detection data is formed in the shape detection module and the trajectory detection data is formed in the trajectory detection module; and
A hand signal recognition method comprising a fourth step in which an image based on the shape detection data or an image based on the trajectory detection data is displayed on a hand signal display unit.
In claim 12,
A hand signal recognition method further comprising a fifth step of performing learning in the shape detection module and the trajectory detection module using learning data generated by the learning module.