KR101837482B1 - Image processing method and apparatus, and interface method and apparatus of gesture recognition using the same - Google Patents

Abstract

A for each predetermined period every time detecting the multi-frame-based motion model, a predetermined object using an in processed images in inputted by the image capturing device into a foreground image and calculates the blob was simplicity it with the position and size and then, the object that is the target of tracking and And an object estimation step of estimating a position of the object at this time using the information of the current foreground image if the block is not present at this time in the image processing method for tracking the position of the object by comparing the block And an image processing method is provided.

Description

TECHNICAL FIELD [0001] The present invention relates to an image processing method and apparatus, and a gesture recognition interface method and apparatus using the same.

The present invention relates to an image processing method and apparatus, and a gesture recognition interface method and apparatus using the same. More specifically, the present invention relates to a method and apparatus for verifying the validity of an input blob, An image processing method and apparatus for preventing erroneous judgment beyond a range and preventing omission or loss of tracking of an object when the object exists in reality and increasing the accuracy of tracking of the position of the object and shortening the time required for tracking, And a gesture recognition interface method and apparatus using the same.

With the rapid development of computer vision technology, the importance of contactless interfaces has become an issue. In a generally known non-contact interface,

(1) Hand Gesture Recognition

(2) Eye [Pupil] Recognition

(3) Voice Recognition

(4) Facial Expression Recognition (Expression Recognition)

(5) Behavior Recognition

And the like.

Such a contactless interface is useful as a tool for mutual relationship between a user and a device, and can be applied to a variety of applications such as administrative work, game, and exercise. As an example of a concrete application device,

(1) Mobile (controller like input device - up, down, left, right, front and back in three dimensional space)

(2) Television (remote controller role - zoom in, zoom out, volume and channel adjustment)

(3) Game machines (controllers like input devices - up, down, left, right, front and back in 3D space)

(4) Others (controller of all devices equipped with camera)

.

For example, as illustrated in Figs. 1 and 2, a target object (e.g., a hand) h1 and h2 of a user h is detected from an image acquired through a camera (image acquisition device) (Not shown) such as a computer or a smartphone connected to the camera according to the gesture, determines a predetermined gesture corresponding to the trajectory, (For example, outputting an occurrence event of forward, backward, leftward and rightward and upward and downward movement) is known. As shown in FIG. 3 (c), the image input 1, the object detection 2, the position tracking 3, the gesture recognition 4, and the post-process 5 proceed in this order.

As a conventional art in this field, there is a hand movement recognition technique using a method using an external device such as a sensor glove and a computer vision technique. Particularly, the computer vision technology based hand-

(1) Histogram matching technique using features such as color, shape, and texture

(2) Contour based template matching techniques such as Chamfer Distance

(3) Hidden Markov Model based on Prediction

(4) Machine Learning

And so on.

In particular, the conventional technology of the patent document relates to 'depth information-based posture and gesture recognition apparatus and method', and features hand region extraction and finger number determination using skin color detection and background removal. Here, the technique of the hand-

(1) Color image input (101)

(2) Color and texture-based hand region detection (102)

(3) overlapping region-based hand tracking (103)

(4) Analysis of hand movement (104)

(5) Event output (105)

.

Patent Publication 10-2014-0055172

However, in the case of an environment in which not only the user h but also various other things are imaged together, as shown in Figs. 1, 2, and 3, ) (b). Here, assuming that the image taken on the camera cam is placed in a virtual xy plane extending in the horizontal (x) and vertical (y) directions, the direction away from the camera cam is referred to as z direction ). Then, in the order of the shallow depth (near) to the deep depth (far) in the z direction from the camera cam, the order of the fly (f), the hexahedron (b), the user (h), the tent As shown in Fig. Here, it should be noted that the fly (f), the user (h), and the plane (p) are movable.

It is possible to detect the target objects h1 and h2 of the user h precisely from the processed image when not only the user h but also various other background objects coexist in the processed image as shown in Figs. 3 (a) and 3 (b) Tracking that change is not easy.

According to the conventional art, after a foreground image of a target object (a hand) and a blob, which briefly shows the foreground image and its position and size, are specified by the hand detection unit 102, It tracks the trajectory of the movement of the object by matching or matching the object that was being tracked in the system. However, since the hand tracking unit 103 merely performs hand tracking based on the overlapping area, it can not verify the validity of the inputted block, and if a wrong block is input, it makes a misjudgment that exceeds the error range there is a problem.

Particularly, since the hand tracking unit 103 merely tracks the hand based on the overlapping area, when the detection of the information on the block fails due to the presence of various background objects, illumination change, and color, There is a problem that the overlapping area with the object can not be calculated, and the result can not be calculated and idling occurs.

The important point here is that, according to the related art, it is a matter of course that the object does not exist on the screen and the block is not calculated. Therefore, when the object is on the screen and the block is not finally calculated for some reason, This is not confirmed or supplemented at the location tracking stage. This causes omission of tracking for objects that should have tracked their position. Therefore, in the process of tracking the position of the object, the accuracy of tracking is decreased, and when they are overlapped, the object is lost and the time required for tracking is increased.

SUMMARY OF THE INVENTION The present invention has been made in order to solve the problems of the prior art, and it is an object of the present invention to verify the validity of an inputted block, to prevent a false judgment exceeding an error range even when a wrong block is inputted, An object of the present invention is to provide a method and an apparatus for image processing that prevent missing or loss of tracking of an object, increase the accuracy of position tracking of the object, shorten the time required for tracking, and provide a gesture recognition interface method and apparatus using the same. will be.

According to an aspect of the present invention, there is provided an image processing method for processing a foreground object in a foreground image using a multi-frame-based (tN) motion model in a processed image input from an image acquisition apparatus every predetermined period, after calculating the blob was detected and simplifying it with the position and size, as compared to the object and the blob that is the target of tracking, in the image processing method to track the location of the object, if the blob is absent in the current time, the current time And an object estimation step of maintaining or estimating the position of the current object using the information of the foreground image (including information on a multi-frame based (tN) motion model).

Here, the object estimating step may include: (1) the foreground image or the block or a combination thereof, (2) the object, or (3) ) takes place when the overlap is preferably a combination, and at least pre-determined ratio.

The position of the object estimated by the object estimating step may be specified as an average position of the foreground image.

When the object is not present in the previous time, the current block is set as a new object when the current block is overlapped with the foreground image or block or combination thereof at least a predetermined ratio at least one time It is preferable that an object addition step is further provided.

When the current block is overlapped with at least one of (a) the foreground image or the block and a predetermined ratio or more, and (b) the object overlaps with the object at a predetermined ratio or more, Is determined.

Here, it is preferable that the position of the current object is calculated as an average position of the current block and the previous object.

According to another aspect of the present invention, there is provided a gesture recognition interface method for performing a gesture recognition using a multi-frame based (tN) motion model in a processed image input from an image acquisition device at predetermined intervals, A tracking step of tracking a trajectory from a current location and size of the past location and size of the object and a gesture determination step of determining a gesture corresponding to the pattern of the trajectory, And the tracking step is performed using the image processing method.

According to another aspect of the present invention, there is provided an image processing apparatus including a plurality of image processing apparatuses, each of which has a predetermined motion vector, after calculating the blob was detected in the foreground image and simplify them to position and size, as compared to the object and the blob that is the target of tracking, in the image processing apparatus for tracking the location of said object, if the blob is absent in the current time, And an object estimator for maintaining or estimating a position of the current object using the current foreground image information (including information on a multi-frame based (tN) motion model).

(2) the object, or (3) the objects (1) and (2) described above, the object estimating unit may determine that the current foreground image is the latest one or more this is preferably carried out when the combination of the to be superimposed above a predetermined rate.

It is preferable that the position of the object estimated by the object estimating unit is designated as an average position of the foreground image.

When the object is not present in the previous time, the current block is set as a new object when the current block is overlapped with the foreground image or block or combination thereof at least a predetermined ratio at least one time Object It is preferable that an additional portion is further provided.

When the current block is overlapped with at least one of (a) the foreground image or the block and a predetermined ratio or more, and (b) the object overlaps with the object at a predetermined ratio or more, Is determined.

Here, it is preferable that the position of the current object is calculated as an average position of the current block and the previous object.

According to another aspect of the present invention, there is provided a gesture recognition interface apparatus including: a gesture recognition unit configured to generate a predetermined object using a multi-frame-based (tN) motion model in a processed image input from an image acquisition device at predetermined intervals; And a gesture determining unit for determining a gesture corresponding to a pattern of the locus, wherein the gesture recognizing interface unit comprises: a gesture recognition unit for recognizing a gesture of the object based on the past position and the size of the object; The tracking unit is characterized by using the image processing apparatus.

According to the present invention, validity of the input block is verified to prevent misjudgment exceeding the error range even when an erroneous block is input. In the case where a target object exists, An image processing method and apparatus for shortening the time required for tracking, and a gesture recognition interface method and apparatus using the same.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a perspective view for explaining an acquisition condition and an obstacle factor of an image used in the image processing method and apparatus according to the present invention, and the gesture recognition interface method and apparatus using the same.
Fig. 2 is a plan view of Fig. 1. Fig.
3 is an overall block diagram of (a) (b) the image captured by the camera, and (c) the gesture recognition interface.
FIG. 4 is a conceptual diagram of a blob that extracts the position and size of a foreground image after background removal, (b) a foreground image and defines and labels it as a pie, and (c) A conceptual diagram, and (d) a table illustrating the updating conditions.
Fig. 5 is an exemplary view of the detailed policy of Fig. 4 (d).
Fig. 6 is an exemplary diagram of the detailed policy of Fig. 4 (d) (B).
Fig. 7 is a diagram showing a detailed policy example of Fig. 4 (d) (C).

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings. However, for members having the same function and performing the same role, the same reference numerals are retained even if the drawings are different, and detailed description thereof may be omitted.

In the present invention, predetermined objects (e.g., hands) (h1, h2) in the processed image (Figs. 3A and 3B) input from the image acquisition device There is provided an image processing method for detecting a BGS (Background Subtraction), calculating a Blob in which the BGS is simplified to a position and a size, and then comparing the object to be tracked with the BLOB to track the position of the object, It is an image processing device implemented by hardware. To this end, according to the present invention, as illustrated in Figs. 1 and 2, a processing image is acquired through a camera (image acquisition device) cam, and as shown in Fig. 3 (c) (2) for detecting a target object (h1, h2) from the input processed image, a position tracking (3) for tracking the trajectory of the movement, and a predefined gesture corresponding to the trajectory (For example, outputting occurrence events of front, rear, left, and right vertical movement) of an information processing device (not shown) such as a computer or a smart phone connected to the camera cam according to the gesture recognition 4 and the gesture And the post-process (5). Among them, the present invention has a remarkable difference from the conventional technique particularly in the position tracking (3).

First, in the present invention, when the foreground image BGS is detected in the object detection 2 as shown in FIG. 4A, a block Blob is defined as shown in FIG. 4B. These blops, with their positions and sizes, are labeled with identifiers. In the illustrated example, the right hand h1 and the left hand h2 are blurred to calculate center positions (X, Y, Z) and size (width and height) Indicates that it is labeled with an identifier. However, this is only an example, and the position may be, for example, the position of the upper left vertex under the condition that it is used uniformly. Instead of the position and the size, two positions, for example, positions of the two facing diagonal vertexes may be stored.

Therefore, in the position tracking (3), the foreground image BGS is not directly compared with the object, but the block Blob calculated from the foreground image is compared with the object. This is because every object whose position is updated by position tracking is stored and managed in a form simplified in position and size as in the case of a block.

O (t-3), O (t-2), and O (t-1) ), It is necessary to determine the current object O (t) at time t. At this time, when the current block B (t) is input, the current object O (t) is determined directly or by processing from the block B (t). Here, we examine the validity of the BLOB B (t) and think about why it should be supplemented.

Since the object detection (2) in Fig. 3 (c) depends on the vision technique, there is a possibility that the accuracy of the detection is significantly lowered depending on the illumination or color or the movement speed. Thereby, even when the foreground image BGS of Fig. 4 (a) is correctly detected, a case may occur in which the blob of Fig. 4 (b) is detected incorrectly or not at all. This causes an incorrect result in tracking the position of the object (e.g., a hand) in the subsequent position tracking 3, which eventually becomes an error in the gesture recognition interface. Therefore, it is necessary to compensate for these problems. In the present invention, based on a motion model, a reasonable estimation is performed to compensate for an inaccuracy, an error, or a recognition failure of a block.

First, as shown in Fig. 4 (d), the updating condition for calculating the position of the current object O (t) from the current block B (t) can be determined. The updating condition can be determined, for example, by taking the presence or absence of a block and the presence or absence of an object as an element.

4B. Fig. 4D shows a case where the current block Blob is present as a result of the input from the object detection 2 and Fig. Blob) does not exist. In addition, (A) and (C) show a case where an object being tracked exists, and (B) and (D) show a case where there is no object being tracked.

First, FIG. 4 (d) shows a case where a new block is input in a situation where an object being tracked exists. In this case, the position of the current object O (t) may be determined by calculating the overlapping relationship.

In this case, when the current block is overlapped with at least one recent (a) the foreground image or the block at a predetermined ratio or more, and (b) the object is overlapped with the object at a predetermined ratio or more, Is determined. Specifically, for example, as shown in FIG. 5, when B (t) overlaps with the latest blocks B (t-2) and B (t- ) And the latest object O (t-3) in O (t-3) is overlapped by 30% or more, the position of the current object O (t) can be determined therefrom. For example, the position of the current object can be calculated as the average position of the current block B (t) and the latest object O (t-3).

Thus, by verifying whether or not the current block B (t) overlaps with the latest block B (t-1), it can be verified that the current block B (t) is not the wrong data. Then, it can be verified whether or not the current block B (t) overlaps with the latest object O (t-3), so that it can be used for estimation of the current object O (t) by estimation.

Next, FIG. 4 (D) (B) shows a case where a new block is input in a situation where there is no object being tracked. In this case, after validating the validity of the block, it is possible to select it as a new object O (t).

In this case, when the object is not present in the previous time, when the current block B (t) is superimposed on the foreground image BGS or the block B or a combination thereof at least a predetermined ratio at least one time And an object adding step for setting the current block to the new object. Specifically, if, for example, B (t) overlaps with the latest blocks B (t-2) and B (t-1) by 25% or more based on 3 frames as shown in FIG. 6, And the position can be determined.

Thus, by verifying whether the current block B (t) overlaps with the latest blocks B (t-2) and B (t-1), it can be verified that the current block is not the wrong data by the estimation. Then, by setting the thus-validated current block B (t) to be a new object O (t), the validity of the object can be guaranteed.

Next, (C) in Fig. 4 (d) shows a case where a new block is not inputted even though an object being tracked exists. In this case, if the current foreground image BGS (t) information is present, the position of the current object O (t) can be maintained or estimated using this foreground image BGS (t) information. This processing step is referred to as an object estimation step , and it is a feature of the present invention. If the current foreground image (BGS) is also absent, it is very likely that the object is almost unrecognizable according to illumination or color, or that the object does not exist in the processed image range at all. Therefore, it may be excluded.

As described above, in the present invention, when there is no foreground image BGS as the basic information in the case where the block B (t) is not detected, rational reasoning that the position of the object O (t) is estimated on the basis of the foreground image BGS , It is possible to judge whether or not a false tracking is made, and in the case of a prospective case, the tracking can be stopped and the false tracking result can be eliminated.

However, if foreground image BGS exists, not all foreground images may be valid. There is a possibility that the foreground image BGS (t) of this time is also erroneously extracted if the current time block B (t) can not be calculated. Therefore, the validity of the foreground image to be used in the object estimation step needs to be verified.

In the present invention, the object estimating step may include: (1) the foreground image or the bluff or a combination thereof, (2) the object, or (3) the foreground image BGS (1) and (2) and overlapping at a predetermined ratio or more. This allows the foreground image to be used for estimation only if it is associated with any other foreground image or block or object or combination of the most recent foreground images. At this time, the position of the object estimated by the object estimation step may be designated as an average position of the foreground image, for example.

As described above, according to the present invention, the validity of the input block is verified to prevent misjudgment exceeding the error range even when an erroneous block is input. In the case where a target object exists, It is possible to prevent the loss phenomenon, increase the accuracy of tracking the position of the object, and shorten the time required for tracking.

According to another aspect of the present invention, there is provided a tracking method comprising: a detection step of detecting a predetermined object in a processed image input from an image acquisition device at predetermined intervals; a tracking step of tracking a trajectory from a current location and size of the past location and size of the object; And a gesture determining step of determining a gesture corresponding to the pattern of the locus, wherein the tracking step is performed using the image processing method or apparatus described above, Interface method or apparatus.

However, the technology of the present invention can be implemented by subdividing each step or incorporating various steps, which may be implemented in whole or in part as a device on a hardware-by-module basis, It is assumed that the present invention also includes an implementation method in which a software module that has been loaded into the memory is executed by a central processing unit such as a microprocessor.

While the present invention has been described with reference to specific embodiments thereof, it is to be understood that the embodiments are for illustrative purposes only and the present invention is not limited to these embodiments. Modifications, variations, and alterations made by those skilled in the art within the scope of the claims are to be construed as being within the scope of the present invention.

The present invention can be applied to an image processing method and apparatus, and a gesture recognition interface method and apparatus using the same.

h: User
h1, h2: object
b, f, p, t: background object
cam: camera (image acquisition device)
1: Video input module
2: object detection module
3: Positioning module
4: Gesture recognition module
5: post-processing module

Claims (8)

One for each predetermined period each time detects the multi-frame-based motion model, a predetermined object using an in processed images in inputted by the image capturing device into a foreground image and calculates the blob was simplicity it with the position and size and then, the object that is the target of tracking and The image processing method according to claim 1, further comprising:
In the image processing method, when the position of the current object is determined, the presence or absence of the current block and the presence or absence of the object being tracked are used as an element of positioning the current object based on the motion model, The position of the current object is determined while compensating for the recognition failure,
(1) When a new block is input at this time in a situation where an object being tracked exists,
(1-1) In a case where at least one of the foreground images or blops is overlapped with a predetermined ratio or more,
(1-2) When the object is overlapped with the at least one object more than a predetermined ratio recently,
An object verification determining step of determining a position of the object at present by the position of the current block;
(2) In the case where a new block is input at this time in a situation where there is no object being tracked, in order to verify the validity of the block, the current block is divided into the foreground image or the block, An object adding step of causing the current block to be set as a new object when overlapping at a predetermined ratio or more,
(3) If there is an object to be tracked but no new block is input, if there is information of the current foreground image, an object for estimating the position of the current object using the current foreground image information Estimation step
, ≪ / RTI >
Wherein, in the object estimation step, in order to verify the validity of the foreground image,
(3-1) the foreground image or block or a combination thereof, or
(3-2) The object, or
(3-3) The combination of (3-1) and (3-2)
And overlapping by more than a predetermined ratio.
The image processing method comprising the steps of: The method according to claim 1,
The current position of the object is calculated as the average position of the current position of the object and the position of the current block
Wherein the image processing method comprises the steps of: A detection step of detecting a predetermined object using a multi-frame-based motion model in a processed image input from an image acquisition device at predetermined intervals,
A tracking step of tracking a trajectory from a current location and size of the past location and size of the object;
A gesture determination step of determining a gesture corresponding to the pattern of the locus
A gesture recognition interface method comprising:
The tracking step is performed using the image processing method of claim 1
Wherein the gesture recognition method comprises the steps of: One for each predetermined period each time detects the multi-frame-based motion model, a predetermined object using an in processed images in inputted by the image capturing device into a foreground image and calculates the blob was simplicity it with the position and size and then, the object that is the target of tracking and An image processing apparatus for comparing the blocks and tracking the position of the object,
When the position of the current object is determined, the image processing apparatus determines whether or not the current block is present and the presence or absence of the object being tracked based on the motion model as an element of positioning the current object, The position of the current object is determined while compensating for the recognition failure,
(1) When a new block is input at this time in a situation where an object being tracked exists,
(1-1) In a case where at least one of the foreground images or blops is overlapped with a predetermined ratio or more,
(1-2) When the object is overlapped with the at least one object more than a predetermined ratio recently,
An object verification determination unit that determines the position of the object at present based on the position of the current block;
(2) In the case where a new block is input at this time in a situation where there is no object being tracked, in order to verify the validity of the block, the current block is divided into the foreground image or the block, An object adding unit for setting the current block to be a new object when overlapping at a predetermined ratio or more,
(3) If there is an object to be tracked but no new block is input, if there is information of the current foreground image, an object for estimating the position of the current object using the current foreground image information Estimation part
, ≪ / RTI >
In the object estimating section, in order to verify the validity of the foreground image,
(3-1) the foreground image or block or a combination thereof, or
(3-2) The object, or
(3-3) The combination of (3-1) and (3-2)
And overlapping by more than a predetermined ratio.
Wherein the image processing apparatus comprises: A detection unit for detecting a predetermined object using a multi-frame-based motion model in a processed image input from an image acquisition device at predetermined intervals,
A tracking unit for tracking a trajectory from a current location and size of the past location and size of the object;
A gesture determining unit for determining a gesture corresponding to the pattern of the locus,
A gesture recognition interface device comprising:
The tracking unit is configured using the image processing apparatus of claim 4
And a gesture recognition interface device. One for each predetermined period each time detects the multi-frame-based motion model, a predetermined object using an in processed images in inputted by the image capturing device into a foreground image and calculates the blob was simplicity it with the position and size and then, the object that is the target of tracking and And an image processing method for tracking the position of the object by comparing the blocks with each other, the method comprising the steps of:
In the image processing method, when the position of the current object is determined, the presence or absence of the current block and the presence or absence of the object being tracked are used as an element of positioning the current object based on the motion model, The position of the current object is determined while compensating for the recognition failure,
(1) When a new block is input at this time in a situation where an object being tracked exists,
(1-1) In a case where at least one of the foreground images or blops is overlapped with a predetermined ratio or more,
(1-2) When the object is overlapped with the at least one object more than a predetermined ratio recently,
An object verification determining step of determining a position of the object at present by the position of the current block;
(2) In the case where a new block is input at this time in a situation where there is no object being tracked, in order to verify the validity of the block, the current block is divided into the foreground image or the block, An object adding step of causing the current block to be set as a new object when overlapping at a predetermined ratio or more,
(3) If there is an object to be tracked but no new block is input, if there is information of the current foreground image, an object for estimating the position of the current object using the current foreground image information Estimation step
, ≪ / RTI >
Wherein, in the object estimation step, in order to verify the validity of the foreground image,
(3-1) the foreground image or block or a combination thereof, or
(3-2) The object, or
(3-3) The combination of (3-1) and (3-2)
And overlapping by more than a predetermined ratio.
Wherein the image processing method is characterized in that the image processing method comprises the steps of: A detection step of detecting a predetermined object using a multi-frame-based motion model in a processed image input from an image acquisition device at predetermined intervals,
A tracking step of tracking a trajectory from a current location and size of the past location and size of the object;
A gesture determination step of determining a gesture corresponding to the pattern of the locus
And a gesture recognition interface method for performing the steps of the gesture recognition interface method in an information processing device,
The tracking step is performed using the image processing method of claim 1
And a gesture recognition interface method for performing the steps of the gesture recognition interface method in an information processing device. delete

Images