TW200910221A - Method of determining motion-related features and method of performing motion classification - Google Patents

Abstract

The invention relates to a method of determining motion-related features (Fm) pertaining to the motion of an object (1), which method comprises obtaining a sequence of images (f1, f2, . . . , fn) of the object (1), processing at least parts of the images (f1, f2, . . . , fn) to extract a number of first-order features from the images (f1, f2, . . . , fn), computing a number of statistical values pertaining to the first-order features, combining the statistical values into a number of histograms (HLC, HG), and determining the motion-related features (Fm) for the object (1) based on the histograms (HLC, HG). Furthermore, the invention relates to a method for performing motion classification for the motion of an object (1) captured in a sequence of images (f1, f2, . . . , fn). Such a motion classification method comprises determining motion-related features (Fm) for the images (f1, f2, . . . , fn) using the described method of determining motion-related features (Fm), and using the motion-related features (Fm) to classify the motion of the object (1). The invention also relates to a system (3) for determining motion-related features (Fm) pertaining to an object (1) in an image (f1), and to a system (4) for performing motion classification for the motion of an object (1).

Description

200910221 IX. Description of the invention: [Technical field to which the invention pertains] The present invention relates to a method for judging an action-related feature relating to an action of an object, and a method for classifying an action for performing an action of an object. Furthermore, the present invention relates to a system for determining action related features relating to an object' and to a system for performing action classification. [Prior Art] One of the most natural means of communication among human beings is to swing with a hand or a head. It is also said that the potential is used as a communication to supplement, or as a means of communication itself. A simple amplification of one of the speeches can include, for example, giving another person a guide to move the hand and arm of the mouth to point to the _special direction. Communicate by communication & sample sign language alone, based entirely on gestures. Since gestures are the first instinct for human beings, for example, in a home dialogue system for automatic sign language interpretation, a dialogue system in an office environment, gestures will be a natural and easy means of communication between humans and computers. . Due to the development of 'in the past few years', the use of various dialogue systems will become more and more in the future, and the classification of users' swings (ie, gesture recognition) will play an increasingly important role. . Several gesture recognition systems have been proposed by the current technical methods. In general, it has been proposed that the system relies on image analysis to clamp a user in each of the image sequences and analyze the continuous pose to determine its hand gesture. Some systems require that the user wear a color suit or position sensor that will be a requirement that is most undesirable for a typical user of a home conversation system. For example, the other systems of the US 6,256, 〇33 B1 propose to capture a certain number of coordinates corresponding to a user's body from an image 130766.doc 200910221, and this:: should be known gestures The coordinates are compared. This is one of the obvious shortcomings. The computational effort required to analyze such images has a very high level, and the camera used to obtain such images must be sufficient to ensure a clear image. Other conventional systems attempt to solve the problem of gestures broadcasting "the problem of 労 労 4" which is operated by stereo images obtained with - or multiple cameras. Example Ο ^Touch 3/_218 proposes to use two cameras Obtaining a stereo image of the user as a gesture, and further filtering the background in the image, and tracking the segment using a statistical framework corresponding to segments of the image of the user's body The user is above the person, using this has been introduced, the calculation level required for gesture recognition is very high, so the potential: the knowledge will be very slow. Another shortcoming of such systems is related to Hard body (four) "_ machine or multiple cameras. For all conventional gestures, the user must slowly perform a handful of gestures to ensure that a certain image that maintains image quality is not blurred. I 茨等手(四)识说'The solution proposed by the current technology is too scary, too slow, or too cumbersome. In particular, the case is too intrusive, the user is easy to learn and use. #Need to be non-invasive to judge A user's swing: capital =,. Frequently, simply using the action of the action classification program to determine the appropriate effort is too expensive to be required for the b 4 system. [Invention] Therefore, the object of the present invention is to provide a judgment on the object in the image. 130766.doc 200910221 Action-related features and methods for classifying one object. Furthermore, one of the present inventions is easy and economical as a system. The present invention provides a method for characterizing a feature, the method comprising the act of obtaining an action of at least eight images of the images, and the order of the images. feature. "" to take a number from the image couch - whereby the term "image sequence", all captured images 2 = for example, borrowed * for example, for a selection with a fast shutter speed. The image may be sufficient, otherwise: the camera 'every nine cameras will be used' may use all images. The low level is determined by one of its action-related features I* , -3⁄4 ^ , an image of the object can handle 1 meal, or for example, when only the image ~, the image - some area. Obtaining only the shadow-related features, gradient values from the shadow, or 掏: as a modulo value, for example, any other meaning of a color y image that describes _pixels is basically in the calculation of the whistle term" The order "feature-feature n stage towel can be obtained from the image information and the value of the ladder value = * pixel color value can be directly taken from the image information, and the decorative value can be obtained by directly processing the image data. . The number of statistic values of the ΪΓ-element-order feature is calculated by combining the statistic value of the pre- or multiple dibasisms, and the correlation feature. The second-order action=the figure determines that the object's action-related feature can be given or retrieved from the straight-line 130766.doc 200910221 Γ=information' identifies the bar field in the image associated with the action of the object, for example, Made by a moving object, especially the 'ambiguous area of the histogram itself k. It is used here as a second-order action-related feature. For example, the term "object" is interpreted as meaning - the user, the body of the user, or the user - to book the body part of the kibs 4 - but can also be able to write ^ Any way to move any moving object, such as a pendulum clock, a machine" and a machine that makes = type. The following will be interchangeably slang "object Ο = == provide an implementation in an image The method of classifying the action of the action of the object captured in the sequence, and the object captured in the image is determined by the above method, and the action related feature is used, and the action of the object is performed using the action related feature knife. > +, + class, benefit J. The obvious advantage is that the action or gesture can be achieved. The image analysis required by the gesture recognition system is taken from @ and used in the method according to the present invention. The algorithm of the coucher is easy to obtain, and is familiar with this technology = two, for example, the set algorithm for image compression. It is easy to simply settle or count the statistical information of the feature in the histogram. Ground and very fast The analysis is carried out to obtain the relevant characteristics of the tracking operation, and the method of using the method of providing a clear advantage over the prior art identification method is not required. The method according to the invention does not require the modulo 1 = The object in the image (for example, - character), and not part of the body. Another advantage is that a network camera, such as a one-resolution camera, is quite sufficient for generating such images, allowing 130766.doc, 10. 200910221 The system according to the invention is implemented efficiently and efficiently. Furthermore, since there is no need to directly track the actual action of the object, a low frame rate is sufficient for the camera, so an inexpensive camera system is quite sufficient. A corresponding system for determining an action-related feature relating to an object includes an image source for providing an image sequence of the object, and a processing unit for processing at least a portion of the image to capture a number from the image First-order features; _ 士 + 瞀 S3 - 异 皂 , , , , , , , , , , , , , , , , , , , , , 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁 汁A statistical value is given to give a number of histograms, and & is used to determine the action-related features of the object based on the 4-histogram. Further, the corresponding system for performing the action classification includes determining the object! The action-related feature of this-system, and at the same time a classification as early as 'the use of the action-related features to classify the action of the object. Special σ's subsidiary request and subsequent descriptions of the disclosed embodiments and features. An image source (for example, a camera type of characterization information, such as Γ! like containing edge definition, etc., such as using grayscale values, round temples or - order features. For two: description = according to the present invention The square* knife is used to capture one of the uses in an image. When the object looks like a grayscale value of the cross-section m gold ^ noon action, for example, "moving time" may be for color or a... For example to track the color of the object. However, it is beneficial to take advantage of the ambiguity of the ~$景'---the & If the _ ° of an image is special - in front of the camera - gesture ❹ 二 2 ° this area corresponds to the hand or arm of a user like a gesture or other swing. In a particularly preferred embodiment of the invention, the order features are extracted from the image by fuzzy correlation features. For image processing techniques such as object or pattern recognition, the information given by the rim or edge in an image is particularly useful. Some edges in the image can be sharp (focus) or blurred (out of focus). For a camera with sufficient depth of field, the Τ blurred portion of an image area will indicate that the object in that portion of the image moves as the image is produced. Several technical systems are available for identifying edges in an image and in recent years techniques for wavelet transform have been developed for applications such as effective image compression. Those skilled in the art will be aware of this technique and will not be explained in detail here. In summary, the wavelet transform technique can be used to quickly and reliably identify the edges in the image and to obtain information as to the level of clarity of such edges. Therefore, in accordance with a preferred embodiment of the present invention, it is preferred that the fuzzy correlation feature is obtained from the image data by performing a wavelet transform with respect to a plurality of scales to determine the shadow material. - Point - series of small miles. Because the value of the wavelet coefficient looks at the difference between the values of the adjacent pixels and then the level of the difference is regarded as the level of the blur in the image, the = wavelet coefficient is a fuzzy correlation feature, which ultimately provides information about the image system. special. The use of fuzzy correlation features as a first-order feature is an important fact that will introduce vagueness into the image: the moving hand and the first class are used for positive effects. Blurring no longer provides a valuable information for a "effect" but for eight (for example, gesture recognition). "The type of wavelet transform used by the knife can look at the size of the image to a certain degree - I3〇 766.d〇c 12 200910221 Ο Ο Degree. For example, image data with a larger resolution of a larger image can be used as a round of the binary wavelet transform (DWT), whereas a continuous wavelet transform (plus) can be used to have only a relatively low resolution. Small size images, such as those produced by a typical webcam. The wavelet coefficients thus obtained can then be processed to obtain the blur related features of the image. Each pixel or point of the image corresponds to a wavelet coefficient' in each scale of the transform and each scale has the same number of points as the image. Therefore, after implementing the -10 scale conversion, there will be ten coefficients for each pixel in the image. For a point or pixel on the edge of the image, the modulus associated with the coefficient of the pixel will be a maximum value in each scale. The evolution of the ten coefficients across the scale of the edge pixels provides information about the derivability of the image at that point. The image gradient can be calculated for the pixel using the wavelet coefficients of one of the pixels in the image. If the pixel in question is the blurring edge of the moving object, the direction and intensity or magnitude of the image gradient provides information about the direction of the blur and the speed of an object. Since the /ladder direction is orthogonal to the 4 edge or wheel temple, it provides information about the shape of the object. Similarly, a low step value of a pixel from a column or row implicitly diffuses or smoothes the edge, and a high step value implies a clear or distinct edge. Thus, the histogram of the intensity and direction of the image gradient provides information about the shape and motion field in the area of consideration of the images. • The slope of the evolution across the coefficients of the scale 曰 always provides an estimate of the so-called =pSChltz coefficient or index for that pixel. The index is measured by a measure of the degree to which the pixel is: derivable, and thus the degree of continuity. If the Lipschitz exponent is positive, then the pixel is at a smooth 130766.doc -13 - 200910221 or blurred edge. Similarly, if the Lipsehitz index is zero or negative, the image is at the sharp edge. Performing the wavelet transform on several scales effectively provides a third dimension other than the two-dimensionality of the image, and the coefficients Ο

Guangjin is observed in this second dimension. In general, a wavelet transform is performed on a series of pixels (e.g., a column or a row of pixels in a group of images). Because, if applicable, the LipSehitz money from the resulting wavelet coefficients and the intensity and direction of the image gradient provide information on the degree of blurring of the neighboring pixels in the (four)-column or row and the direction of the blur. For example, a set of most negative Lipschhz indices for a column of pixels would mean that the columns of the column include well-defined edges. On the one hand, the Llpschltz index of the most positive value will indicate that there is a smooth transition between the pixels in this column, which implies that it lacks a sharp edge, so it is possible that this column belongs to the ambiguous region of the image. The wavelet transform can be performed across consecutive columns of pixels and consecutive rows of pixels. The wavelet transform is for each image or part of the image, such as the index and wavelet coefficient gradient described above. The -th order feature is calculated for each-wavelet transform operation (i.e., for each - processed column and row). The effective way to collect the information provided by the -order feature is to carry the value of the -order feature to some discrete value and simply count the number of occurrences of each value. Thus, in accordance with the present invention, statistical values relating to the -order features of a certain type of at least a portion of the image (eg, the number of occurrences of the special coefficients) are combined in the histogram of the "order" = after In a system for obtaining an action related to an object in an image - 130766.doc -14- 200910221 related features, a counter can be assigned to each discrete coefficient value, and when this value occurs, Increase the counter by one. The value thus accumulated is settled or collected in the histogram, which can be visualized into a simple bar graph. In order to determine the swing or gesture made by a user, a plurality of images (also referred to as frames) can be captured, for example, by a camera following the swing of the user from start to finish. Generally speaking, this action or swing is referred to as a gesture, and in general, the various stages or positions in the swing are referred to as postures and are not intended to limit the scope of the invention in any way. The overall gesture is then given sequentially by a gesture, and then each gesture is captured in the image sequence. For the user to move their right hand and arm-gesture or swing, the images will display the user and a blurred area to the left of the image, which effectively "moves across the time-lapse image If the order of an image or frame is imaged on the upper right and one is stacked on the other, the resulting three-dimensional block is the virtual volume given by the sequence of images undergoing time. For this - All of the images in the volume capture first-order features and combine them to give action-related information about the overall sequence. Thus in a preferred embodiment of the invention, for example, by performing the image of the = a wavelet transform to take a first-order feature for the virtual volume' and combine the statistical values of the features into one or more graphs. And, m, however, *only a portion of the image is actually of interest Therefore, in order to reduce unnecessary calculations, it is preferred that only those regions in the image of actual interest can be processed. For example, as described above, if the user is only one by one, 130766.doc -15- 200910221 The hand and the arm make a gesture, and the area in the image including the hand and the arm is only a part of the overall image, and the area or the specific area can be identified in an image, and only the continuous image in the image sequence needs to be processed. Corresponding to a specific area. This takes advantage of the fact that a user will more or less stay in the same place when making a gesture in front of a camera. If the images or frames are imaged on the upper right and one is stacked The other is followed by 'and the specific area of each of the successive frames is imaged as "

Ο Connected ", the result is a virtual sub-volume. Accordingly, in a particularly preferred embodiment of the present invention, a particular region sequence' is identified in an image sequence and a first order feature is obtained for a child volume given by a particular region sequence of elapsed time. The identification of a particular region of interest in the image is easily achieved by the first-order features of the image, for example by analyzing the number of occurrences of a first-order feature of the various regions of one of the image sequences. For example, the overall image can be cut or subdivided into a plurality of segments or illuminating blocks, preferably in such a manner that the segments or illuminating blocks overlap. For each segment or illuminating block, a sub-image histogram of the species-order features can be compiled. For example, a sub-image histogram for most of the zero or positive values of the _ gamma coefficient indicates that this region of the image contains a tracing. m is blurred so that all areas of the image containing the blur can be quickly identified. A selection may be formed that includes such zones, which may be visualized by an imaginary rectangle drawn around a moving arm of one of the images, and for example by the image of the rectangle Coordinate = Image coordinates can be used to locate the same in all images in the order; Temple field. $, etc., for example, for a moving object having a color different from the background 130766.doc • 16 - 200910221 in the images, the per-lighting square or segment can be analyzed by pixel color values. A light-emitting block containing only pixels of a certain color can be: interested, so the remaining light-emitting blocks can be ignored. - The steps of a particular area of interest can be observed at regular intervals so that the special area tracks the moving objects in the images. In this way, it is even possible to analyze the motion or gesture of the user involved in the overall image. Any action or gesture can be broken down into a series of substantially different Γ 连 sequential stages that, when used together, give the overall action or gesture. Therefore, the following § 吾 "Location" refers to any stage in the action of capturing an object or user in the _image sequence. A histogram of one or more first-order statistical values can be compiled for each particular region of the image sequence. Therefore, when considered together, the information in these histograms is characteristic of the order of the particular region. Moreover, since an image can be used to determine a position held or a gesture made, the information of the histogram of the order of the particular region is the position maintained or the characteristics of the resulting security. Thus, in a preferred embodiment of the present invention, a leaf value relating to a type of order from a particular region in the sequence of images is combined in a volume histogram of the feature. Description The position of the cow, a pose, or a volumetric one of a gesture will be used in the action classification procedure described in more detail below. Although the region of the image defined by the particular region contains useful information about the type of creation: or gesture type, the image can also be used to complement the ''region). The complementary region basically includes an image that does not contain, the region. n A犄疋

Pixel Thus, in a preferred embodiment of the invention, A I30766.doc -17- 200910221 identifies that a particular one of the images, pr ° Π domain #, also identifies its complementary region, and one of - _ value of the genre-order feature = 顺序 from the complementary region sequential volume histogram. , value, and. A complement to this-order feature ★ A particular region and/or complementary region of an image is identified by a - order feature that is judged for the second:: interpreted image. Example 2 Γ Ο Jing) area. Various increments, and calculations::#, in order to identify the interested parties know. Action-related features that can be obtained and are familiar to this 2-image sequence can be used to classify == actions that are captured in order t. For example, an action is made as a "win" between two histograms. This distance can be calculated between one of the four images V and the histogram of the corresponding complementary region, or it can be consecutive* in the image sequence. ~ standard statistical distribution techniques, either by comparing the main properties of the = (for example in the case of fuzzy correlation-order features and = degree direction or main Lipschitz coefficients), or any "_ early nature of the histogram. Finally 'The distances are an indication of the degree of change in the relevant features relative to the image areas. As described above, the visible order corresponds to - the particular position or fixed portion of the object = trace, or corresponds to a use In other words, according to the present invention, the sub-volume histogram and the complementary volume histogram (hereinafter collectively referred to as "volume histogram" generated by using the image-order features of the image are described as follows. The most line γ I30766.doc -18- 200910221 describes or characterization of the position of the object. Therefore, in a preferred method according to the present invention, such a volume histogram is analyzed, Obtaining a position characteristic feature that can be used to classify the action. For example, a position characteristic feature such as a ratio of a positive histogram value to a negative histogram value in a volume histogram of an image sequence can give a position and direction of the action of the object An indication of another type of positional property information may be a distance between two volume histograms of different image sequences. Ο u Using a volumetric histogram may be for various sequences of actions or for a wide variety of Gesture, comparing the volume histogram (or the derivative of the volume histogram) with the previously generated data. For example, one of the volume histograms and gesture models obtained by tracking one of the poses by the shirt image sequence Several "prototype poses" are collected (for example, collected in one of several state transition models). In this way, one or more candidates can be determined: a gesture that includes a gesture corresponding to the volume histogram. By recognizing the continuous prototype of the volume histogram of the pose order, the number of candidate gestures can be narrowed down until the gesture most likely to correspond to the pose order has been identified. Thus, in a preferred embodiment of the invention, the positional characteristic of an action (obtained from the analysis of the image sequence) is used to identify a position or pose sub-star based on an action or gesture - a unit to generate a model (For example, one of the states of the state transitions die-cut 1Μ __ U &). This state corresponds to a special position in the action, and you, I, Jifel, corresponds to a certain posture of a user, or corresponds to a certain stage of the action φ of an object. It is familiar with one known to the skilled person. The preferred state transition model is a hidden Markov model (ΗΜΜ), and it is used to judge a position in a gesture or swing and then another position: 130766.doc •19· 200910221 Rate histogram to derive simpler features For example, the above-distance. The single Li between the volume histogram and its complementary volume histogram can be applied to any suitable method for obtaining a statistical distribution comparison of real numbers from a series of volume histograms. Method (eg, - incremental algorithm established - difference (four). ... / / input, in the other embodiment of the invention - particularly preferred embodiment, the calculation uses an image sequence as described above The volume histogram and the sequence of the corresponding complementary regions in the order of the images produced by the order of the singular, Γ·丄, and the corresponding complementary regions in the image sequence are generated between the complementary bodies '', + M ^ 10,000 graphs - To give: a location characteristic feature, which can then be used in the action classification program. , , ^ ^ , Gan + ^ § 4 and other histograms (second-order action-related features), which can also be regarded as a class if the prototype position is By the characteristics of this type, the action related features. The simple third-order feature can identify the corresponding use of this special j. (4) The position of the action - the action described above for obtaining the action-related features and heart The steps may be implemented in the form of a software module, which is operated in a system for performing action classification, for performing a motion-related feature, for example, first, or as a certain These image processing steps, such as wavelet transforms in the case of fuzzy correlation features, can be implemented in the form of hardware (such as image-month/intermediate circuit (ASIC) or field programmable gate array (FP(3A)). ', the use of integrated electrical power 130766.doc • 20- 200910221 From the above detailed description with reference to the drawings, the purpose and features of the present invention will be readily understood. However, it should be understood that the drawings are only for Purpose 7 is not intended to limit the definition of the present invention. [Embodiment] FIG. 1 shows that the user 1 gestures in front of a camera 2 (for example, a network camera 2). Here, the user i is acting The direction of M (such as __ gesture or a gesture) moves its arm. The webcam generates the user i

The scene v is like fl, and this is forwarded to a suitable processing unit (not shown in the figure), such as a personal computer. Any suitable type of camera can be used, and even a low cost network camera 2 with a typical resolution of -32G pixels is sufficient for this purpose. The user 1 can act as a gesture associated with a command that is commonly present in a conversation between the user and a dialog system, for example, "stop interaction", "etc.", "return to continuation, help", etc. For example. In a noisy environment, the user 1 is connected by sign language, which eliminates the need for voice interaction. The user can also provide additional gesture information to supplement a spoken command, for example, by pointing to - Special direction, and verbally instruct the camera to look in that direction', for example, by saying "seeing this." The webcam 2 can generate the image of the user's work with more or less regular intervals' and gives an image sequence as shown in Fig. 2a ^ f f2, ..., fn. The following examples of the proposed method are explained The first-order features of the order are fuzzy correlation features, which are selected because of the advantages of the above detailed description. Without limiting the invention in any way, it is assumed that a method of gesture classification is described 130766.doc 21 200910221 As illustrated in Figure 2a, the user has been captured in the sequence of fi, 匕, ., . 1 of all kinds of fast 一把 一把 a aunt to h 呎 起 起 起 起 起 起 起 起 起 起 起 起 起 起 起 私 私 私 私 私 私 私 私 私 私 私 私 私 私 私 私 私 私 私 私 私An edge (four) is then performed on an image 6 by performing a wavelet transform on the columns and rows of pixels in the image fl. It may be sufficient to look at the size of the image £1 and the resolution of the camera 2 using every nine pixels, or it is necessary to use each pixel to obtain a set of wavelet coefficients per pixel in this way, which contains

Information about the difference between adjacent pixels, that is, the motion information given by the fuzzy level. For each-group wavelet coefficient, a discrete fuzzy correlation feature is derived, in this case the Lipsehitz exponent and image gradient. The occurrence of each value of the count-order feature', e.g., the number of Upschitz coefficients of a 1>5 or the number of image gradient values of 〇, and is collected by the histogram I of the first-order feature. Therefore, the obtained histogram Hlc, % contains the fuzzy related information about the overall image f丨, such as the ratio of the blurred image ,, or the level of sharpness of the edge in the image fi. Repeating this procedure for each image of the remaining images f2, ..., fn in the sequence, while performing edge analysis on the overall image is a waste of resources, and will be easy to flow around. For example, a type of "sliding window" can be used to virtually break into the image data, which gives a smaller sub-view to achieve the J-wave transform. The dimension of the sliding window can also vary over time. For ease of classification, Figure 2b shows another technique for reducing computational effort and easier visual presentation. Here, an image L is virtually divided into smaller segments, light-emitting blocks, or sub-images. Sub-images of the overall image fi can be Each of the 130766.doc •22· 200910221 performs an edge (four). It displays a sub-image of 2., = its histogram histogram SHLC, SHG. Since the program is for the image f丨Each of these sub-images: in such a histogram system: the image is derived from each of them - and then can be analyzed to determine which of the person is actually interested. As one of the gestures Use = Yes: The element is the limb of its movement, such as the arm, the hand, etc., and because of the octave; 蕤Γ accompanied by a certain level of blur, so the image fl "interest" Γ Ο: 2 ° a check Equivalent-order feature histogram To determine /, which is - fuzzy sub-image prostitute h A τ1 As described above, the binary of the image can be easily located by using the type of "sliding window" for the image data, effectively minimizing the calculation Efforts. = The map has positioned two such sub-images 21'22 in the image f. ..., ground' can subdivide the image into more sub-images than shown here: and = sub-images can also be Overlap. However, for the sake of simplicity, only the two sub-images 21, 22 are not of interest. These sub-images 21, 22 are combined with a special area A, which is indicated by a thick rectangle in the figure. Camera or internet camera 彡 her I 2 堂7... The camera is fascinating and gesturing - the user usually = or less will stand in the same place, and only move one or both of his hands and arms. Therefore, you can The specific area identified in a contention or frame is simply propagated from an image in ::: to the next, in order to define - a specific area of the sequence 2 or each of the frames. Counting resources' because of the time, only the shadows in these specific areas Changing the rest of the image (the " complementary " region). All intent remains the same and the hit the nose, and may be considered as a fixed 130766.doc -23- 200910221.

This system is illustrated in Fig. 3a, which shows the image sequence 匕, f2, ..., fn which are vertically stacked after the other. If a dashed line is drawn to connect the corners of the frames L f2'...' fn, a virtual volume V is obtained. The specific area A1 shown in the first frame G is propagated to all subsequent frames ^..., I thus the frame f 2, f PI J- ba 2, ·, Π has its specific position at the relevant position region

Ah..., An. Again, draw a dashed line to connect to these specific areas A. A corner of A2, ..., An, a virtual sub-volume % is created which excludes the complementary areas Al, A, ... of the images L f2'..., fn. As described in the figure, the first-order feature histogram is calculated for each of the frames k ..., the specific regions Als A2, ·.., and the sub-images in the frame. By combining these histograms into corresponding volume histograms I, %, the information contained in these histograms is expanded to include time. In other words, it also experiences time to track changes in the level of blur in the special areas of the images. This diagram is illustrated in the figure, in which the image data of the frame sequence ^ L..., fn has been superimposed on the image 3〇, and it is found in the zone Ο that the user made the gesture All "fun" information should be in the specific region A, A of the image sequence fi, U: In Figure 4, a state transition model G of an action or hand heat is shown, in this case A discrete hidden Mark〇v model (Hmm ^ c ^ T - finite number of 1::12, S3, S4 corresponds to several prototype positions of an action, or: model of the type A of the genre is used for classification - action For example, a gesture made by the user. The transition is by the probability of being the other one (as shown, for example, by probability p(s > 33), 叩 4.. 84) It is weighted separately from ^4), ring-state 4, from state 1 130766.doc •24·200910221 to state 3, and from state 4 to state 4 stepping or a measure of the probability of making the transition. For example, a volume histogram created by the user and captured in a particular sequence of images can be compared to a collection of prototype gestures associated with various gestures. The graph shows the state of the state transition model G ~, S2, S3, S4 four prototype pose histograms PH丨, pH2, pH3, pH4, but any number of such state transition models can be obtained, each of which Can be associated with various prototype poses. These prototype pose histograms pHi, pH:, PH3, PH* can be generated previously using a suitable learning algorithm. Ο When comparing a volume histogram of a pose with the prototype pose, the prototype pose PH that is most similar to, for example, the state & associated with the state transition model can be found. This relatively simply involves calculating a distance, as described above, between the volume histogram and the prototype histogram, and converting this distance to a probability, which is a number between _. It can be found that the subsequent volume histogram collected by the elapsed time corresponds to the sequential state I, & s4' so that the gesture made by the user will likely correspond to the state transition model by this state (3 In the event that another volume histogram does not correspond to the lower-like fS2, the classification procedure may include that the transition model G does not correspond to a candidate gesture. ^ Figure 5 shows a block diagram of one of the systems 4 for performing gesture classification. Without limiting the scope of the present invention in any way, the system includes - camera 2, used - user 1 - image sequence..., (10) in the graph, = not - single - image fl). Each image is first acquired in a processing unit $ to obtain a plurality of fuzzy correlation features 12. In this particular embodiment, the 130766.doc -25- 200910221 牵 牵 牵 牵 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 对该 。 。 。 。 。 。 。 。 。 。 Then, the fuzzy correlation-order features ^ are forwarded to - δ ten counts early x 6, where the statistical value 14 is calculated for the first-order features 在 in this case, if appropriate, in counting or settlement Before the occurrence of each discrete value, the first-order features are first carried or rounded to the nearest set of predefined discrete values (eg _〇.5, 〇·〇, 〇5, 1〇, etc.) ',,, = give = counts... In the - combination unit 15, the counts of 14 groups are given to a number of histograms HLC, hg, each for each type of blur, and for - ( The Llpschltz index and the wavelet coefficient ladder can be regarded as an action-related feature of the image. The equal histogram can be used to derive the additional action correlation. The total number of positive values in the histogram HLC, HG is zero or negative.

The ratio of the total can be supplied as the M of the histogram HLC, HQ. Equivalently, the histogram (4) is used as a correlation feature. The histograms Hlc and HG can be used as a β-feature Fm. The combining unit 15 may include a separate analyzing unit for performing a histogram analysis of the required requirements, but in the commentary, it is displayed as a single unit and blocks 2, 5, and 6 described so far. Features, while - system 3' system 4 front end. The histograms HLC, H (for forwarding to a first to collect, and combined with the previous histogram, for:: early: 'in which the image data of the previous image in the image sequence is added or: :: Straight = is derived from this 。. Then the 130130.doc • 26- 200910221 The output of the second combining unit 8 is a set of volume histograms VHlc, VHg, VHLC', VHG, wherein the volume histogram VHlc, VHg corresponds to a particular region of interest identified in the sequence of images produced by the camera 2, and the complementary volume histograms VHlc, VHg correspond to complementary regions in the sequence of images. Ο Ο In the second combining unit 8, the volume histograms VHlc, V%, vhlc, VHg corresponding to certain postures made by the user when performing the gesture are processed to obtain the position characteristic feature Fp, for example, the volume. The distance between the histograms VHlc and VHg, or the distance between a volume histogram VHLC, VHG and its complementary volume histogram VHu:|, VHg. The volume histograms VHlc, VHg, VHlc, vtv and these positions Feature feature Fp is forwarded to one Class 7 is analyzed in order to classify the gestures made by the user. Finally, from the candidate gesture - the database type 9 pose histogram ΡΗ ι, ΡΗ 2, 叫, 么 prototype bit: A 13 ° can be some suitable Mode comparison - volume histograms ρ Η =, VHG, VHLC,, VHg, and a corresponding prototype histogram PH1, 4 卩 _ number of candidate handcuffs. By comparing the continuous volume histogram, the limbs experience time irAvs The number of gestures in which the gesture is made, Kyrgyz, narrows the number of candidate gestures from the database down to τ until the recognition is most likely, a .gg <#作-suitable signal 10 is forwarded to the other two. The result of the classification is as a module. Processing block 11, for example, a gesture interpretation, although the second seedling-part & unit 8 is used to perform the action classification system 4, it is also possible to assume that the second combination unit 8 Integrated in a system 3 for determining the relevant characteristics of the motion 130766.doc • 27· 200910221. Figure 6 shows a block diagram of the main steps in the method of action classification according to the present invention for first order based on fuzzy correlation One of the characteristics of the analysis In a first step 5, the image 600 is obtained from an image source such as a camera. Edge detection is performed on the image 6〇〇 in an edge detection block 5〇1, for example by Performing wavelet transform on the image data to give fuzzy correlation information 6〇1 M columns such as a set of wavelet coefficients 6〇1. Ay is processed in a special_fetch block to give each group of wavelet coefficients 6 (H- Group - 'order feature 602. In the histogram compilation block 503, calculate the system value of the first-order feature 602, * and compile the number of histograms or volume histograms of the order-like features of the image. , an image-specific area' - image order, or - image-specific area order. The histograms 6〇3 are input to a histogram analysis block 5〇4, from which the motion-related features or positional characteristics 6〇4 are derived. These features are then input to an action classification block 505, which is used to determine the operation of the corresponding location. Once the classification procedure is completed, the output signal 605 can indicate the identified action, e.g., a gesture made by a user, or whether the action classification has expired. The present invention has been disclosed in its preferred embodiments and variations thereof, but it will be appreciated that numerous additional modifications may be made thereto without departing from the scope of the invention. The present invention can be used in any system that makes it impossible to: and has a computer with sufficient computing resources. For example, the method and system for gesture recognition can be used to recognize sign language gestures ^ to convert this to text. Alternatively, the implementation may be a monitoring system that uses I30766.doc -28- 200910221 to classify and identify moving objects of the type (5), such as people, cars, and the like. Another possible implementation may be in an automated production process in which systems and methods in accordance with the present invention can be used to track mobile projects and to control different stages of production. For the sake of clarity, please understand that this case is all in use. • • • • • • • • “ does not exclude plurals, and "including ^ does not exclude other steps or components. A "unit" or " The module " can include Zhao 4 _,,. 匕 数 疋 数 模组 模组 模组 模组 模组 模组 , , , , , 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 【 User-image-intentional representation; Ο Figure 2a is a person-image sequence-schematic representation, basin with associated motion information and fuzzy correlation features; Figure 8b is based on the invention. 2~ _. ', a schematic representation of one of the images of the Knife sub-image, Figure 2c is according to the present invention; a schematic representation of one of the specific areas of the Guanyin shirt image by the image sequence a schematic representation of the = product and its associated volume histogram given by the order of a particular region; a representation of a virtual system, a singularity, a ;;, 1 ^ » view, which displays an image sequence Superimpose a specific area; a state of the body/one gesture-state transition model Figure, and a plurality of crowbar histograms; Figure 5 is a block 130766.doc • 29-200910221 of the system for gesture recognition in accordance with the present invention; Figure 6 shows the execution of gesture recognition in accordance with the present invention. A flow chart of the steps of the method. In the drawings, the same number from the beginning to the end refers to the same object. [Description of main component symbols] 1 User 2 Camera 3 System 4 System 5 Processing unit 6 Calculation unit 7 Classification unit 8 Combination unit 9 Database 10 Signal 11 Processing block 12 Fuzzy correlation feature 13 Posture information 14 Statistics 15 Combination unit 20 Image 21 > 22 Sub-image 30 Image 31 Area of image 130766.doc -30- 200910221

A 1,A 2,...,A η Ai ',A2',...,An' fl ,f2, ... ,fn G specific region complementary region image state transition model

130766.doc -31 -

Claims (1)

200910221 X. Patent application scope: 1. A method for judging the action-related features (5) of an action of an object (1) 'This method includes - obtaining the image sequence (fbU) of the object (1); - the image of the king ( At least a portion of fl, f2, ., fn) to extract a number of first-order features from the V image (fI 'f2, 'fn); • calculate a number of statistical values relating to the first-order features; __ combine And other statistical values to give a plurality of histograms (HO); and determining the action-related features of the object (1) based on the histograms (HLC, HG) (FJ. IX 2 2_, as in the method of claim 1, Where #征. 〒 The specific first-order features include fuzzy correlations. 3. As in the method of claim 2, which extracts fuzzy related features from the relevant 牯Λ 4 (h, f2, ·.., fn) including the image w w wavelet coefficient, monthly ♦w wavelet transform 'to determine a number of ο characteristics. U coefficient 'to give such fuzzy correlations 4. As described in the above request, any box by the time of the image two:, \, one of Step: It is the one given by borrowing (V). , ,, fn) 5. The method of the above-mentioned request item " 疋 疋 a special region sequence A2, ..., An) is identified in the image sequence, (^, 4 2 ' ', . And _ τ Θ 藉 藉 经历 经历 f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f f ;t will have _ taken from - image sequence... 130766.doc 200910221 2' '··' From - the sequence of specific regions in the image sequence (f}, heart, fn) (AA 1 ' 2,...,Αη) The statistical value of a certain order characteristic of a certain kind ▲ in a volume histogram (vhlc, vhg) of this order feature. The method of finding item 6, wherein a complementary region is ordered (f:, in image order (fl, Identifyed in f2, ·.., fn), where a complementary region (αλ ",...,An,) in the image silly: f V··, fn) is associated with the shadow and will have the specific one in the LA The region (Αΐ, Α2,...,Αη)·4 Γ (A, A ', the statistic value of the first-order feature of the species (taken from the complementary region order 2'.··, An')) is combined here First order feature A complementary volume histogram ⑽ qe iv) in. Supplementing the method of claim 5, wherein the complementary region of the image (f,, f2 f "2, ..., An) and / or ., . . . , fn) (Α,, Α 2,, ..., Αη,) is performed by the first-order features judged by the image (f-missing recognition, 2', ·, U) - an action performed in the image sequence ^ f The party of the classification; I. The object of capture (1) to 8 of the method ^ The method includes classifying the action (1) of the object (1) using the object (6) in the action related special feature 14) of claim 1 (1) The action-related features 10. As in the case of the item 9 Λ Λ / υ , ' 'the volume histogram, VHLC,, VHg,) is the map (VHlc, VHq, position characteristic feature (F) 4 item 6 Or the method of 7 is generated to obtain the P action of classifying the object (1) and analyze the positional characteristic features (Fp), 11 · as in the case of the request item 丨0, where the five tenth volume histograms (VH, 130766. Doc 200910221 VHg) with a complementary volume histogram (vhlc, VHg,) ~ feature to obtain position characteristics (Fp), and analyze this distance (FP), to classify the action of the object (1). 12. If the position feature is used to identify a state transition in a state transition model a state (s], , / position ~), the state (s, s2, s "S4) corresponds to the motion two S" W - special system 13: species:: r an object (1) ... phase _-... source (7), for providing the image sequence of the object (1) • processing the early element (5) 1 at (4), etc. (f, f2 f. = part 'to extract several images from the image (..."): The number of first-order features 'one computational unit (6) statistical values; the square graph (H-combined unit 〇5)' is used to combine the statistical values into several illusions) and is used for The histogram (hm, Hg) such as Hai is used to determine the action-related features (Fm). 14=: The system (4) that performs the action classification of the action of the object (1), and the system (3) of the second uncle 13 of Judging the relevant = feature (6), and a classification unit (7) for classifying the object using the action feature (Fm). 15. - Computer readable media 'its storage - computer programs can be loaded directly 130766.doc 200910221 Iinto a memory of a programmable device for use in a system for determining action-related features of an object, and/or for performing a classification of actions of an action of an object, including a software code portion The step of performing the method of claim 1 when the computer readable medium is run on the device. 130766.doc