WO2015112194A2 - Processeur d'image comprenant un système de reconnaissance de geste avec reconnaissance de pose de main statique sur la base d'une déformation dynamique - Google Patents

Processeur d'image comprenant un système de reconnaissance de geste avec reconnaissance de pose de main statique sur la base d'une déformation dynamique Download PDF

Info

Publication number
WO2015112194A2
WO2015112194A2 PCT/US2014/047744 US2014047744W WO2015112194A2 WO 2015112194 A2 WO2015112194 A2 WO 2015112194A2 US 2014047744 W US2014047744 W US 2014047744W WO 2015112194 A2 WO2015112194 A2 WO 2015112194A2
Authority
WO
WIPO (PCT)
Prior art keywords
image
contour
hand
interest
points
Prior art date
Application number
PCT/US2014/047744
Other languages
English (en)
Other versions
WO2015112194A3 (fr
Inventor
Alexander A. PETYUSHKO
Ivan L. MAZURENKO
Dmitry N. BABIN
Aleksey A. LETUNOVSKIY
Alexander B. KHOLODENKO
Original Assignee
Lsi Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Lsi Corporation filed Critical Lsi Corporation
Priority to US14/374,392 priority Critical patent/US20160026857A1/en
Publication of WO2015112194A2 publication Critical patent/WO2015112194A2/fr
Publication of WO2015112194A3 publication Critical patent/WO2015112194A3/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G06V40/11—Hand-related biometrics; Hand pose recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features

Definitions

  • the field relates generally to image processing, and more particularly to image processing for recognition of gestures.
  • Image processing is important in a wide variety of different applications, and such processing may involve two-dimensional (2D) images, three-dimensional (3D) images, or combinations of multiple images of different types.
  • a 3D image of a spatial scene may be generated in an image processor using triangulation based on multiple 2D images captured by respective cameras arranged such that each camera has a different view of the scene.
  • a 3D image can be generated directly using a depth imager such as a structured light (SL) camera or a time of flight (ToF) camera.
  • SL structured light
  • ToF time of flight
  • raw image data from an image sensor is usually subject to various preprocessing operations.
  • the preprocessed image data is then subject to additional processing used to recognize gestures in the context of particular gesture recognition applications.
  • Such applications may be implemented, for example, in video gaming systems, kiosks or other systems providing a gesture-based user interface.
  • These other systems include various electronic consumer devices such as laptop computers, tablet computers, desktop computers, mobile phones and television sets.
  • an image processing system comprises an image processor having image processing circuitry and an associated memory.
  • the image processor is configured to implement a gesture recognition system comprising a static pose recognition module.
  • the static pose recognition module is configured to identify a hand region of interest in at least one image, to extract a contour of the hand region of interest, to compute a feature vector based at least in part on the extracted contour, and to recognize a static pose of the hand region of interest utilizing a dynamic warping operation based at least in part on the feature vector.
  • inventions include but are not limited to methods, apparatus, systems, processing devices, integrated circuits, and computer-readable storage media having computer program code embodied therein.
  • FIG. 1 is a block diagram of an image processing system comprising an image processor implementing a static pose recognition module in an illustrative embodiment.
  • FIG. 2 is a flow diagram of an exemplary static pose recognition process performed by the static pose recognition module in the image processor of FIG. 1.
  • FIG. 3 shows an example of an extracted contour comprising an ordered list of points.
  • FIGS. 4A and 4B illustrate respective left hand and right hand versions of a given hand region of interest.
  • FIG. 5 illustrates the generation of a feature vector using an extracted contour.
  • FIG. 6 is a flow diagram of a process for determining a centroid of a static pose class.
  • FIG. 7 is a flow diagram of a process for determining pattern statistics for a static pose class using a centroid determined by the process of FIG. 6.
  • Embodiments of the invention will be illustrated herein in conjunction with exemplary image processing systems that include image processors or other types of processing devices configured to perform gesture recognition. It should be understood, however, that embodiments of the invention are more generally applicable to any image processing system or associated device or technique that involves recognizing static poses in one or more images.
  • FIG. 1 shows an image processing system 100 in an embodiment of the invention.
  • the image processing system 100 comprises an image processor 102 that is configured for communication over a network 104 with a plurality of processing devices 106-1, 106-2, . . . 106- M.
  • the image processor 102 implements a recognition subsystem 108 within a gesture recognition (GR) system 1 10.
  • the GR system 110 in this embodiment processes input images 1 1 1 from one or more image sources and provides corresponding GR-based output 112.
  • the GR- based output 1 12 may be supplied to one or more of the processing devices 106 or to other system components not specifically illustrated in this diagram.
  • the recognition subsystem 108 of GR system 1 10 more particularly comprises a static pose recognition module 114 and one or more other recognition modules 1 15.
  • the other recognition modules may comprise, for example, respective recognition modules configured to recognize cursor gestures and dynamic gestures.
  • the operation of illustrative embodiments of the GR system 110 of image processor 102 will be described in greater detail below in
  • the recognition subsystem 108 receives inputs from additional subsystems 1 16, which may comprise one or more image processing subsystems configured to implement functional blocks associated with gesture recognition in the GR system 1 10, such as, for example, functional blocks for input frame acquisition, noise reduction, background estimation and removal, or other types of preprocessing.
  • additional subsystems 1 16 may comprise one or more image processing subsystems configured to implement functional blocks associated with gesture recognition in the GR system 1 10, such as, for example, functional blocks for input frame acquisition, noise reduction, background estimation and removal, or other types of preprocessing.
  • the background estimation and removal block is implemented as a separate subsystem that is applied to an input image after a preprocessing block is applied to the image.
  • Exemplary background estimation and removal techniques suitable for use in the GR system 1 10 are described in Russian Patent Ap lication No. 2013135506, filed July 29, 2013 and entitled "Image Processor Configured for Efficient Estimation and Elimination of Background Information in Images,” which is commonly assigned herewith and incorporated by reference herein.
  • the recognition subsystem 108 generates GR events for consumption by one or more of a set of GR applications 118.
  • the GR events may comprise information indicative of recognition of one or more particular gestures within one or more frames of the input images 1 11, such that a given GR application in the set of GR applications 1 18 can translate that information into a particular command or set of commands to be executed by that application.
  • the recognition subsystem 108 recognizes within the image a gesture from a specified gesture vocabulary and generates a corresponding gesture pattern identifier (ID) and possibly additional related parameters for delivery to one or more of the applications 118.
  • ID gesture pattern identifier
  • the configuration of such information is adapted in accordance with the specific needs of the application.
  • the GR system 110 may provide GR events or other information, possibly generated by one or more of the GR applications 118, as GR-based output 1 12. Such output may be provided to one or more of the processing devices 106. In other embodiments, at least a portion of the set of GR applications 118 is implemented at least in part on one or more of the processing devices 106.
  • Portions of the GR system 110 may be implemented using separate processing layers of the image processor 102. These processing layers comprise at least a portion of what is more generally referred to herein as "image processing circuitry" of the image processor 102.
  • the image processor 102 may comprise a preprocessing layer implementing a preprocessing module and a plurality of higher processing layers for performing other functions associated with recognition of gestures within frames of an input image stream comprising the input images 1 1 1.
  • Such processing layers may also be implemented in the form of respective subsystems of the GR system 110.
  • embodiments of the invention are not limited to recognition of static or dynamic hand gestures, but can instead be adapted for use in a wide variety of other machine vision applications involving gesture recognition, and may comprise different numbers, types and arrangements of modules, subsystems, processing layers and associated functional blocks.
  • processing operations associated with the image processor 102 in the present embodiment may instead be implemented at least in part on other devices in other embodiments.
  • preprocessing operations may be implemented at least in part in an image source comprising a depth imager or other type of imager that provides at least a portion of the input images 1 1 1.
  • one or more of the applications 1 18 may be implemented on a different processing device than the subsystems 108 and 1 16, such as one of the processing devices 106.
  • image processor 102 may itself comprise multiple distinct processing devices, such that different portions of the GR system 1 10 are implemented using two or more processing devices.
  • image processor as used herein is intended to be broadly construed so as to encompass these and other arrangements.
  • the GR system 1 10 performs preprocessing operations on received input images 11 1 from one or more image sources.
  • This received image data in the present embodiment is assumed to comprise raw image data received from a depth sensor, but other types of received image data may be processed in other embodiments.
  • Such preprocessing operations may include noise reduction and background removal.
  • the raw image data received by the GR system 110 from the depth sensor may include a stream of frames comprising respective depth images, with each such depth image comprising a plurality of depth image pixels.
  • a given depth image D may be provided to the GR system 1 10 in the form of a matrix of real values.
  • a given such depth image is also referred to herein as a depth map.
  • image is intended to be broadly construed.
  • the image processor 102 may interface with a variety of different image sources and image destinations.
  • the image processor 102 may receive input images 111 from one or more image sources and provide processed images as part of GR-based output 1 12 to one or more image destinations. At least a subset of such image sources and image destinations may be implemented as least in part utilizing one or more of the processing devices 106.
  • At least a subset of the input images 11 1 may be provided to the image processor 102 over network 104 for processing from one or more of the processing devices 106.
  • processed images or other related GR-based output 1 12 may be delivered by the image processor 102 over network 104 to one or more of the processing devices 106.
  • processing devices may therefore be viewed as examples of image sources or image destinations as those terms are used herein.
  • a given image source may comprise, for example, a 3D imager such as an SL camera or a ToF camera configured to generate depth images, or a 2D imager configured to generate grayscale images, color images, infrared images or other types of 2D images. It is also possible that a single imager or other image source can provide both a depth image and a corresponding 2D image such as a grayscale image, a color image or an infrared image. For example, certain types of existing 3D cameras are able to produce a depth map of a given scene as well as a 2D image of the same scene. Alternatively, a 3D imager providing a depth map of a given scene can be arranged in proximity to a separate high-resolution video camera or other 2D imager providing a 2D image of substantially the same scene.
  • An image source is a storage device or server that provides images to the image processor 102 for processing.
  • a given image destination may comprise, for example, one or more display screens of a human-machine interface of a computer or mobile phone, or at least one storage device or server that receives processed images from the image processor 102.
  • the image processor 102 may be at least partially combined with at least a subset of the one or more image sources and the one or more image destinations on a common processing device.
  • a given image source and the image processor 102 may be collectively implemented on the same processing device.
  • a given image destination and the image processor 102 may be collectively implemented on the same processing device.
  • the image processor 102 is configured to recognize hand gestures, although the disclosed techniques can be adapted in a straightforward manner for use with other types of gesture recognition processes.
  • the input images 11 1 may comprise respective depth images generated by a depth imager such as an SL camera or a ToF camera.
  • a depth imager such as an SL camera or a ToF camera.
  • Other types and arrangements of images may be received, processed and generated in other embodiments, including 2D images or combinations of 2D and 3D images.
  • image processor 102 in the FIG. 1 embodiment can be varied in other embodiments.
  • an otherwise conventional image processing integrated circuit or other type of image processing circuitry suitably modified to perform processing operations as disclosed herein may be used to implement at least a portion of one or more of the components 1 14, 115, 116 and 118 of image processor 102.
  • image processing circuitry that may be used in one or more embodiments of the invention is an otherwise conventional graphics processor suitably reconfigured to perform functionality associated with one or more of the components 114, 115, 1 16 and 118.
  • the processing devices 106 may comprise, for example, computers, mobile phones, servers or storage devices, in any combination. One or more such devices also may include, for example, display screens or other user interfaces that are utilized to present images generated by the image processor 102.
  • the processing devices 106 may therefore comprise a wide variety of different destination devices that receive processed image streams or other types of GR-based output 112 from the image processor 102 over the network 104, including by way of example at least one server or storage device that receives one or more processed image streams from the image processor 102.
  • the image processor 102 may be at least partially combined with one or more of the processing devices 106.
  • the image processor 102 may be implemented at least in part using a given one of the processing devices 106.
  • a computer or mobile phone may be configured to incorporate the image processor 102 and possibly a given image source.
  • Image sources utilized to provide input images 111 in the image processing system 100 may therefore comprise cameras or other imagers associated with a computer, mobile phone or other processing device.
  • the image processor 102 may be at least partially combined with one or more image sources or image destinations on a common processing device.
  • the image processor 102 in the present embodiment is assumed to be implemented using at least one processing device and comprises a processor 120 coupled to a memory 122.
  • the processor 120 executes software code stored in the memory 122 in order to control the performance of image processing operations.
  • the image processor 102 also comprises a network interface 124 that supports communication over network 104.
  • the network interface 124 may comprise one or more conventional transceivers, In other embodiments, the image processor 102 need not be configured for communication with other devices over a network, and in such embodiments the network interface 124 may be eliminated.
  • the processor 120 may comprise, for example, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor (DSP), or other similar processing device component, as well as other types and arrangements of image processing circuitry, in any combination.
  • a "processor” as the term is generally used herein may therefore comprise portions or combinations of a microprocessor, ASIC, FPGA, CPU, ALU, DSP or other image processing circuitry.
  • the memory 122 stores software code for execution by the processor 120 in implementing portions of the functionality of image processor 102, such as the subsystems 108 and 1 16 and the GR applications 118.
  • a given such memory that stores software code for execution by a corresponding processor is an example of what is more generally referred to herein as a computer- readable storage medium having computer program code embodied therein, and may comprise, for example, electronic memory such as random access memory (RAM) or read-only memory (ROM), magnetic memory, optical memory, or other types of storage devices in any combination.
  • RAM random access memory
  • ROM read-only memory
  • magnetic memory magnetic memory
  • optical memory or other types of storage devices in any combination.
  • Articles of manufacture comprising such computer-readable storage media are considered embodiments of the invention.
  • the term "article of manufacture” as used herein should be understood to exclude transitory, propagating signals.
  • embodiments of the invention may be implemented in the form of integrated circuits.
  • identical die are typically formed in a repeated pattern on a surface of a semiconductor wafer.
  • Each die includes an image processor or other image processing circuitry as described herein, and may include other structures or circuits.
  • the individual die are cut or diced from the wafer, then packaged as an integrated circuit.
  • One skilled in the art would know how to dice wafers and package die to produce integrated circuits. Integrated circuits so manufactured are considered embodiments of the invention.
  • image processing system 100 as shown in FIG. 1 is exemplary only, and the system 100 in other embodiments may include other elements in addition to or in place of those specifically shown, including one or more elements of a type commonly found in a conventional implementation of such a system.
  • the image processing system 100 is implemented as a video gaming system or other type of gesture-based system that processes image streams in order to recognize user gestures.
  • the disclosed techniques can be similarly adapted for use in a wide variety of other systems requiring a gesture-based human-machine interface, and can also be applied to other applications, such as machine vision systems in robotics and other industrial applications that utilize gesture recognition.
  • embodiments of the invention are not limited to use in recognition of hand gestures, but can be applied to other types of gestures as well.
  • the term "gesture” as used herein is therefore intended to be broadly construed.
  • the input images 11 1 received in the image processor 102 from an image source comprise input depth images each referred to as an input frame.
  • this source may comprise a depth imager such as an SL or ToF camera comprising a depth image sensor.
  • Other types of image sensors including, for example, grayscale image sensors, color image sensors or infrared image sensors, may be used in other embodiments.
  • a given image sensor typically provides image data in the form of one or more rectangular matrices of real or integer numbers corresponding to respective input image pixels. These matrices can contain per-pixel information such as depth values and corresponding amplitude or intensity values. Other per-pixel information such as color, phase and validity may additionally or alternatively be provided.
  • a process 200 performed by the static pose recognition module 1 14 in an illustrative embodiment is shown.
  • the process is assumed to be applied to preprocessed image frames received from a preprocessing subsystem of the set of additional subsystems 1 16.
  • the preprocessing subsystem performs noise reduction and background estimation and removal, using techniques such as those identified above.
  • the image frames are received by the preprocessing system as raw image data from an image sensor of a depth imager such as a ToF camera or other type of ToF imager.
  • the image sensor comprises a variable frame rate image sensor, such as a ToF image sensor configured to operate at a variable frame rate.
  • the static pose recognition module 114 or at least portions thereof can operate at a lower frame rate than other recognition modules 1 15, such as recognition modules configured to recognize cursor gestures and dynamic gestures.
  • use of variable frame rates is not a requirement, and a wide variety of other types of sources supporting fixed frame rates can be used in implementing a given embodiment.
  • the process 200 includes the following steps:
  • ROI detection Region of interest detection
  • This step in the present embodiment more particularly involves defining an ROI mask for a hand in the input image.
  • the ROI mask is implemented as a binary mask in the form of an image, also referred to herein as a "hand image,” in which pixels within the ROI are have a certain binary value, illustratively a logic 1 value, and pixels outside the ROI have the complementary binary value, illustratively a logic 0 value.
  • the ROI corresponds to a hand within the input image, and is therefore also referred to herein as a hand ROI.
  • ROI masks each comprising a hand ROI can be seen in FIGS. 3, 4A, 4B and 5 in the context of various steps of the FIG. 2 process.
  • the ROI mask is shown with 1 -valued or "white” pixels identifying those pixels within the ROI, and 0-valued or "black” pixels identifying those pixels outside of the ROI.
  • the input image in which the hand ROI is identified in Step 1 may be supplied by a ToF imager.
  • a ToF imager typically comprises a light emitting diode (LED) light source that illuminates an imaged scene.
  • Distance is measured based on the time difference between the emission of light onto the scene from the LED source and the receipt at the image sensor of corresponding light reflected back from objects in the scene.
  • speed of light one can calculate the distance to a given point on an imaged object for a particular pixel as a function of the time difference between emitting the incident light and receiving the reflected light. This distance is more generally referred to herein as a depth value.
  • the hand ROI can be identified in the preprocessed image using any of a variety of techniques. For example, it is possible to utilize the techniques disclosed in the above-cited Russian Patent Application No. 2013135506 to determine the hand ROI. Accordingly, the first step of the process 200 may be implemented in a preprocessing block of the GR system 1 10 rather than in the static pose recognition module 114.
  • the hand ROI can be determined using threshold logic applied to depth and amplitude values of the image. This can be more particularly implemented as follows:
  • amplitude values are known for respective pixels of the image, one can select only those pixels with amplitude values greater than some predefined threshold. This approach is applicable not only for images from ToF imagers, but also for images from other types of imagers, such as infrared imagers with active lighting. For both ToF imagers and infrared imagers with active lighting, the closer an object is to the imager, the higher the amplitude values of the corresponding image pixels, not taking into account reflecting materials. Accordingly, selecting only pixels with relatively high amplitude values allows one to preserve close objects from an imaged scene and to eliminate far objects from the imaged scene.
  • pixels with lower amplitude values tend to have higher error in their corresponding depth values, and so removing pixels with low amplitude values additionally protects one from using incorrect depth information.
  • Opening or closing morphological operations utilizing erosion and dilation operators can be applied to remove dots and holes as well as other spatial noise in the image.
  • the output of the above-described ROI determination process is a binary ROI mask for the hand in the image. It can be in the form of an image having the same size as the input image, or a sub-image containing only those pixels that are part of the ROI. For further description below, it is assumed that the ROI mask is an image having the same size as the input image. As mentioned previously, the ROI mask is also referred to herein as a "hand image” and the ROI itself within the ROI mask is referred to as a "hand ROI.”
  • the output may include additional information such as an average of the depth values for the pixels in the ROI.
  • This step in the present embodiment more particularly involves defining the palm boundary and removing from the ROI any pixels below the palm boundary, leaving essentially only the palm and fingers in a modified hand image.
  • Such a step advantageously eliminates, for example, any portions of the arm from the wrist to the elbow, as these portions can be highly variable due to the presence of items such as sleeves, wristwatches and bracelets, and in any event are typically not useful for static hand pose recognition.
  • the palm boundary may be determined by taking into account that the typical length of the human hand is about 20-25 centimeters (cm), and removing from the ROI all pixels located farther than a 25 cm threshold distance from the uppermost fingertip, possibly along a determined main direction of the hand.
  • the uppermost fingertip can be identified simply as the uppermost 1 value in the binary ROI mask.
  • the 25 cm threshold can be converted to a particular number of image pixels by using an average depth value determined for the pixels in the ROI as mentioned in conjunction with the description of Step 1 above.
  • the contour of the hand ROI is determined, so as to permit the contour to be used in place of the hand ROI in subsequent processing steps.
  • the contour is represented as ordered list of points characterizing the general shape of the hand ROI. The use of such a contour in place of the hand ROI itself provides substantially increased processing efficiency in terms of both computational and storage resources.
  • FIG. 3 A more particular example of an extracted contour comprising an ordered list of points selected from the hand ROI is shown in FIG. 3.
  • the contour of a hand ROI for a pointing finger gesture comprises the ordered list of points denoted 1, 2, 3, 4, 5, 6, 7, 8, 9 in the figure.
  • the contour in this example generally characterizes the border of the hand ROI in a clockwise direction.
  • a given extracted contour determined in this step of the process 200 can be expressed as an ordered list of n points ci, c 2 , c n .
  • Each of the points includes both an x coordinate and a y coordinate, so the extracted contour can be represented as a vector of coordinates ((c ] x , c !y ), (c 2x , c 2y ), .. ., (c nx , c ny )).
  • the contour extraction may be implemented at least in part utilizing known techniques such as S. Suzuki and K. Abe, “Topological Structural Analysis of Digitized Binary Images by Border Following,” CVGIP 30 1, pp. 32-46 (1985), and C.H. The and R.T. Chin, “On the Detection of Dominant Points on Digital Curve,” PAMI 11 8, pp. 859-872 (1989). Also, algorithms such as the Ramer-Douglas-Peucker (RDP) algorithm can be applied in extracting the contour from the hand ROI.
  • RDP Ramer-Douglas-Peucker
  • the particular number of points included in the contour can vary for different types of hand ROI masks and associated static poses. Contour simplification not only conserves computational and storage resources as indicated above, but can also provide enhanced recognition performance. Accordingly, in some embodiments, the number of points in the contour is kept as low as possible while maintaining a shape close to the actual hand ROI.
  • a given extracted contour is normalized to a predetermined left or right hand configuration.
  • This normalization may involve, for example, flipping the contour points horizontally, as illustrated for corresponding hand ROIs in FIG. 4A and 4B.
  • FIGS. 4A and 4B show respective left hand and right hand versions of a given hand ROI from which a contour has been extracted. It is apparent that the left hand version in FIG. 4A can be obtained by horizontally flipping the right hand version in FIG. 4B, and vice-versa.
  • the static pose recognition module 1 14 in the present embodiment is assumed to be configured to operate on either right hand versions or left hand versions. For example, if it is determined in this step that a given extracted contour or its associated hand ROI is a left hand ROI when the static pose recognition module 1 14 is configured to process right hand ROIs, then the normalization involves horizontally flipping the points of the extracted contour, such that all of the extracted contours subject to further processing correspond to right hand ROIs. For subsequent description below, it is assumed that the static pose recognition module 114 operates using the right hand versions only, and that any detected left hand versions are converted to right hand versions prior to further processing. This is not a requirement, however, and it is possible in some embodiments to process both left hand and right hand versions, for example, using respective distinct sub-classes of a static pose class.
  • the normalization in Step 4 can alternatively be performed prior to the contour extraction step, utilizing the hand ROI itself rather than the contour points, although the normalization process is generally much more efficient when applied to the extracted contour than to the corresponding hand ROI.
  • the horizontal flipping of the contour points can be achieved by reversing the order of the ordered list of contour points.
  • the left hand and right hand versions can be distinguished from one another using a number of different techniques.
  • Pr is more particularly determined as the center of the maximal-circumference circle that can be inscribed within the extracted contour.
  • the point Pr can be approximately determined using the following computationally-efficient iterative process:
  • step 5 If the new center point is sufficiently close to the previous center point, or if a designated number of iterations (e.g., 2 iterations) is reached, the process is complete, and otherwise the process returns to step 2. Other convergence properties can be used to terminate the iterative process.
  • the above iterative process generates a point that is close to the center of the maximal- circumference inscribed circle, but involves significantly less computational complexity than determining the actual center.
  • Such an approximate point is considered an example of what is more generally referred to herein as a center of a maximal-circumference circle that can be inscribed within an extracted contour.
  • the current version is assumed to be a right hand version and no normalization is required.
  • Pc x > Pr x then the current version is assumed to be a left hand version, and the contour points should be flipped horizontally in order to generate the corresponding right hand version for use in subsequent processing. More particularly, the horizontal flipping of the contour points is achieved in the present embodiment by reversing the order of the contour points such that the normalized contour is given by Cn, Cn- l j . . . , C l .
  • the left hand and right hand versions can be distinguished using both x and y coordinates of the Pc and Pr points.
  • information such as a main direction of the hand can be determined and utilized to facilitate distinguishing left hand and right hand versions of the extracted contours.
  • Exemplary techniques for determining hand main direction are disclosed in Russian Patent Application Attorney Docket No. L13-0959RU1, filed October 30, 2013 and entitled "Image Processor Comprising Gesture Recognition System with Computationally- Efficient Static Hand Pose Recognition," which is commonly assigned herewith and incorporated by reference herein.
  • This particular patent application further discloses additional relevant techniques, such as skeletonization operations for determining a hand skeleton in a hand image, that may be applied in conjunction with distinguishing left hand and right hand versions of an extracted contour in a given embodiment. For example, a skeletonization operation may be performed on a hand ROI, and a main direction of the hand ROI determined utilizing a result of the skeletonization operation.
  • Other information that may be taken into account in distinguishing left hand and right hand versions of an extracted contour includes, for example, a mean x coordinate of points of intersection of the hand ROI and a bottom row or other designated row of the frame, with the mean x coordinate being determined prior to removing from the hand ROI any pixels below the palm boundary in Step 2 described above.
  • a classification engine of the static pose recognition module 1 14 may involve use of a database of training images in which the training images are predetermined as left hand or right hand versions.
  • Step 5 Feature vector computation
  • features are computed from the extracted contour in this step and utilized in subsequent steps to facilitate recognition of static hand poses.
  • the recognition by dynamic warping in Step 7 of process 200 can be applied directly to the vector of coordinates ((ci x , ci y ), (c 2x , c 2y ), (c nx , c ny )), such that Steps 5 and 6 are eliminated.
  • the feature vectors may be viewed as parameterizations of the corresponding contours.
  • FIG. 5 shows a pointing figure gesture of the type previously described in conjunction with FIG. 3.
  • a pair of x and y coordinate axes is shown having an origin O.
  • the origin O may correspond to a center point of the extracted contour, such as one of the points Pc or Pr described above, or another point with similar characteristics.
  • the contour points ci and c 2 in FIG. 5 represent two consecutive points from an extracted contour ci, c 2 , c n .
  • Arrowed solid lines emanating from origin O of the coordinate system in the figure are more particularly referred to herein as radius vectors n and r 2 and denote respective distances between contour points ci and c 2 and the origin O.
  • the feature vector in such an arrangement illustratively comprises an ordered list of radius vectors n, r 2 , r n corresponding to respective ones of the contour points ci , C 2 , . . . , Cn.
  • the feature vector computed in Step 5 can further include, for each of the radius vectors, the angle in a clockwise direction between the positive x axis and that radius vector.
  • This angle for radius vector n is illustrated by the dashed line in FIG. 5, and is denoted as ⁇ ⁇ .
  • the feature vector in this example comprises an ordered list of pairs (radius vector, angle), and is more particularly given by ((n , ⁇ p ⁇ ), (r 2 , ⁇ ), (r n , ⁇ )), where is the angle in the clockwise direction between the positive x axis and rk.
  • the feature vector can utilize relative angles ⁇ .
  • the feature vector in this example comprises an ordered list of pairs (radius vector, relative angle), and is more particularly given by ((n, ⁇ i), (r 2 , ⁇ 2 ), (r n , ⁇ ⁇ )).
  • the feature vector ((n, ⁇ ), (r 2 , ⁇ ), (r n , ⁇ )) tends to provide better recognition results than the other two in some embodiments of the exemplary process 200.
  • feature vectors that are computed from an extracted contour in Step 5 of the process 200.
  • a wide variety of other types of features vectors comprising respective different parameterizations of an extracted contour can be used in other embodiments.
  • the term "feature vector” as used herein is therefore intended to be broadly construed, and should not be viewed as being limited in any way to any particular aspects of the above examples.
  • the number and spacing of the contour points may be adjusted in order to improve the regularity of the point distribution over the contour. Such adjustment is useful in that different types of contour extraction can produce different and potentially irregular point distributions, which can adversely impact recognition quality. This is particularly true for embodiments in which the contour is simplified after or in conjunction with extraction. In some embodiments, it has been found that recognition quality generally increases with increasing regularity in the distribution of the contour points.
  • an initial extracted contour comprising the ordered list of points ci ,...,c n is converted into a processed list of points CCl , . . , ,CCm, where distances
  • are approximately equal for all i 1 ... m - 1, where m may, but need not, be equal to n.
  • the number of points in the contour is changed in this conversion process.
  • An exemplary technique for converting an initial extracted contour to a contour with improved regularity of point distribution is as follows:
  • a predetermined number m-1 of equal sub-segments is desired.
  • a particular sub-segment length is desired, rather than a particular number of sub-segments.
  • len the desired length
  • points cq and ccj + i as the nearest points of the contour in the foregoing can utilize not only the points from the initial contour ci, ..., c n , but also interpolated points. This is possible, for example, in the case of simplified contours, because the simplified contour values D(n)*(j-l)/(m-l) and D(n)*j/(m-l) will typically lie sufficiently far from the points of the initial contour.
  • the interpolated points can be determined using linear interpolation, spline interpolation or other types of interpolation.
  • d(i) sqrt( (x(i-l)-x(i) ) A 2 + ( ( i-1 ) -y ( i ) ) ⁇ 2 ) ;
  • phi(i) atan2 (y (i) -my, x(i)-mx); % (mx, my) - the center of the hand
  • Step 6 Feature vector normalization
  • the feature vector computed in Step 5 is normalized. Assuming by way of example that the feature vector is given by ((n, ⁇ p ⁇ ), (r 2 , ⁇ ), (r m , ⁇ 1 ⁇ 2)), the feature vector can be normalized in the following manner:
  • steps 1 and 2 of this exemplary feature vector normalization process may be interpreted as division in the complex number space.
  • Step 6 The particular normalization applied in Step 6 will generally vary depending upon the type of feature vector and other factors.
  • recognition involves comparing two time-series signals each of which comprises a contour in the form of an ordered list of points.
  • si (pi, .., p n i)
  • si (qi, . ..,q n 2)
  • An exemplary training process utilized to obtain such patterns for all classes will be described in detail below.
  • contour feature vector is given by ((n, ⁇ p ⁇ ), (r 2 , (pi), (r m , i»m)), although as indicated previously, numerous other types of feature vectors may be used.
  • the distance between the i-th element of s and j-th element of a given pattern can be determined as follows using a Mahalanobis distance metric:
  • the previously-described dynamic warping is then applied to determine the distance between s and pat e as F(s, pat e ). This distance may be subject to a final correction taking into account the lengths of both s and pate by dividing F(s, pate) by the sqrt(m 2 + len_e 2 ).
  • static pose classes examples include pointing finger, palm with fingers, hand edge, pinch, fist, fingergun and many others.
  • training database illustratively incorporates training images that include respective known static hand poses and may be implemented at least in part using one or more storage devices associated with the memory 122 of the image processor 102.
  • the training process can be implemented as follows.
  • centroid is determined for each class. This centroid may be determined, for example, by computing argmin(max(F(sj, Sj) for all ij) where F(si, Sj) denotes all pairwise dynamic warping distances between sample images within the class.
  • process 600 includes steps 602, 604 and 606, as well as multiple parallel instances of Steps 1 through 6 of the FIG. 2 process.
  • step 602 a particular class e is selected.
  • the Lc sample images s e i, s e Lc of the extracted subset are utilized to estimate the centroid.
  • the multiple parallel instances of Steps 1 through 6 of the FIG. 2 process are applied to respective ones of the Lc sample images, and so there are Lc parallel instances in the process 600. Each instance generates a normalized feature vector in the manner previously described in conjunction with FIG. 2.
  • step 606 the normalized feature vectors received from the respective instances of Steps 1 through 6 of the FIG. 2 process are further processed in the manner described below to determine the centroid for the class e.
  • centroid determination in step 600 of the training process does not utilize means and standard deviations.
  • the process 600 is repeated for each of the classes in the training database, with a different class e being selected on each iteration.
  • centroids may be determined for each of one or more of the classes, with each such centroid for a given class corresponding to a primary dissimilar hand pose variation within that class. For example, if the training images within a class have not all been normalized to either left hand or right hand versions of the corresponding static hand pose, separate centroids may be determined for the left hand and right hand versions. Other dissimilar hand pose variations may be treated in a similar manner.
  • each class for which multiple centroids are determined can be separated into multiple sub-classes each corresponding to one of the multiple centroids.
  • the recognition in Step 7 can then be configured to generate a recognition result that indicates not only the class but also the sub-class for a given input image.
  • the separation of classes into sub-classes can be implemented, for example, using clustering techniques, such as the k-means algorithm.
  • the patterns for each class are obtained using the process 700 of FIG. 7.
  • step 702 a particular class e is selected.
  • step 704 all of the sample images in class e are obtained.
  • step 706 the centroid for class e is obtained, as previously determined in process 600 of FIG. 6.
  • Each such processing path includes an instance of step 708 followed by an instance of step 710,
  • the figure shows only the first and final parallel processing paths, although it is assumed that there are train e such parallel processing paths, one for each of the sample images to be used for pattern training in class e, where train e ⁇ nc e .
  • the first of these multiple parallel processing paths includes steps 708-1 and 710-1, and the final one includes steps 708-traine and 710-train e .
  • step 708 prepares the corresponding sample using Steps 1 through 6 of FIG. 2 to generate a normalized feature vector from that sample.
  • Step 710 determines the correspondence between that normalized feature vector and the previously-determined centroid for class e.
  • distance F(s e i, cntr e ) is obtained using a technique similar to that used to determine the centroid in FIG. 6, and correspondence between elements s e and cntr e which leads to this distance is determined.
  • s e i is denoted as z
  • cntr e is denoted as x.
  • the process 700 is repeated for each of the classes in the training database, with a different class e being selected on each iteration.
  • processing blocks shown in the embodiments of FIGS. 2, 6 and 7 are exemplary only, and additional or alternative blocks can be used in other embodiments.
  • blocks illustratively shown as being executed serially in the figures can be performed at least in part in parallel with one or more other blocks or in other pipelined configurations in other embodiments.
  • the illustrative embodiments provide significantly improved gesture recognition performance relative to conventional arrangements. For example, these embodiments provide significant enhancement in the computational efficiency of static pose recognition through the use of dynamic warping of contour feature vectors. Accordingly, the GR system performance is accelerated while ensuring high precision in the recognition process.
  • the disclosed techniques can be applied to a wide range of different GR systems, using depth, grayscale, color infrared and other types of imagers which support a variable frame rate, as well as imagers which do not support a variable frame rate.
  • Different portions of the GR system 110 can be implemented in software, hardware, firmware or various combinations thereof.
  • software utilizing hardware accelerators may be used for some processing blocks while other blocks are implemented using combinations of hardware and firmware.
  • At least portions of the GR-based output 1 12 of GR system 110 may be further processed in the image processor 102, or supplied to another processing device 106 or image destination, as mentioned previously.

Abstract

Un système de traitement d'image comprend un processeur d'image comportant des circuits de traitement d'image et une mémoire associée. Le processeur d'image est configuré pour mettre en œuvre un système de reconnaissance de geste comprenant un module de reconnaissance de pose statique. Le module de reconnaissance de pose statique est configuré pour identifier une région de main d'intérêt dans au moins une image, pour extraire un contour de la région de main d'intérêt, pour calculer un vecteur de caractéristique sur la base au moins en partie du contour extrait, et pour reconnaître une pose statique de la région de main d'intérêt en utilisant une opération de déformation dynamique basée au moins en partie sur le vecteur de caractéristique.
PCT/US2014/047744 2014-01-22 2014-07-23 Processeur d'image comprenant un système de reconnaissance de geste avec reconnaissance de pose de main statique sur la base d'une déformation dynamique WO2015112194A2 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US14/374,392 US20160026857A1 (en) 2014-07-23 2014-07-23 Image processor comprising gesture recognition system with static hand pose recognition based on dynamic warping

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
RU2014101965 2014-01-22
RU2014101965/08A RU2014101965A (ru) 2014-01-22 2014-01-22 Процессор изображений, содержащий систему распознавания жестов с распознаванием статической позы кисти руки, основанным на динамическом изменении масштаба времени

Publications (2)

Publication Number Publication Date
WO2015112194A2 true WO2015112194A2 (fr) 2015-07-30
WO2015112194A3 WO2015112194A3 (fr) 2015-11-05

Family

ID=53682086

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/047744 WO2015112194A2 (fr) 2014-01-22 2014-07-23 Processeur d'image comprenant un système de reconnaissance de geste avec reconnaissance de pose de main statique sur la base d'une déformation dynamique

Country Status (2)

Country Link
RU (1) RU2014101965A (fr)
WO (1) WO2015112194A2 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113128324A (zh) * 2020-01-16 2021-07-16 舜宇光学（浙江）研究院有限公司 基于深度数据的手势分割方法及其系统和电子设备
CN114499655A (zh) * 2021-11-23 2022-05-13 烽火通信科技股份有限公司 一种提高otdr事件识别的方法及装置

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5845002B2 (ja) * 2011-06-07 2016-01-20 ソニー株式会社 画像処理装置および方法、並びにプログラム
CN103294996B (zh) * 2013-05-09 2016-04-27 电子科技大学 一种3d手势识别方法
CN103455794B (zh) * 2013-08-23 2016-08-10 济南大学 一种基于帧融合技术的动态手势识别方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113128324A (zh) * 2020-01-16 2021-07-16 舜宇光学（浙江）研究院有限公司 基于深度数据的手势分割方法及其系统和电子设备
CN114499655A (zh) * 2021-11-23 2022-05-13 烽火通信科技股份有限公司 一种提高otdr事件识别的方法及装置
CN114499655B (zh) * 2021-11-23 2023-05-16 烽火通信科技股份有限公司 一种提高otdr事件识别的方法及装置

Also Published As

Publication number Publication date
WO2015112194A3 (fr) 2015-11-05
RU2014101965A (ru) 2015-07-27

Similar Documents

Publication Publication Date Title
US20150253864A1 (en) Image Processor Comprising Gesture Recognition System with Finger Detection and Tracking Functionality
US20160026857A1 (en) Image processor comprising gesture recognition system with static hand pose recognition based on dynamic warping
US20220383535A1 (en) Object Tracking Method and Device, Electronic Device, and Computer-Readable Storage Medium
US20150253863A1 (en) Image Processor Comprising Gesture Recognition System with Static Hand Pose Recognition Based on First and Second Sets of Features
US20150278589A1 (en) Image Processor with Static Hand Pose Recognition Utilizing Contour Triangulation and Flattening
Tang et al. A real-time hand posture recognition system using deep neural networks
US9959462B2 (en) Locating and tracking fingernails in images
US20180068431A1 (en) Video processing system and method for object detection in a sequence of image frames
US20190188460A1 (en) Method and device for use in hand gesture recognition
US10140513B2 (en) Reference image slicing
US20150286859A1 (en) Image Processor Comprising Gesture Recognition System with Object Tracking Based on Calculated Features of Contours for Two or More Objects
US20150161437A1 (en) Image processor comprising gesture recognition system with computationally-efficient static hand pose recognition
US9269018B2 (en) Stereo image processing using contours
WO2010135617A1 (fr) Systèmes de reconnaissance des gestes et procédés associés
US20130106994A1 (en) Depth Image Compression
US20150269425A1 (en) Dynamic hand gesture recognition with selective enabling based on detected hand velocity
US20150310264A1 (en) Dynamic Gesture Recognition Using Features Extracted from Multiple Intervals
US20170249503A1 (en) Method for processing image with depth information and computer program product thereof
WO2015012896A1 (fr) Procédé et appareil de reconnaissance gestuelle sur la base de l'analyse de multiples limites candidates
US20150262362A1 (en) Image Processor Comprising Gesture Recognition System with Hand Pose Matching Based on Contour Features
US9256780B1 (en) Facilitating dynamic computations for performing intelligent body segmentations for enhanced gesture recognition on computing devices
Vishwakarma et al. An efficient approach for the recognition of hand gestures from very low resolution images
CN117581275A (zh) 眼睛注视分类
US20150139487A1 (en) Image processor with static pose recognition module utilizing segmented region of interest
US20150278582A1 (en) Image Processor Comprising Face Recognition System with Face Recognition Based on Two-Dimensional Grid Transform

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14880005

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14880005

Country of ref document: EP

Kind code of ref document: A2