WO2020048213A1 - 指尖检测方法、指尖检测装置、指尖检测设备及介质 - Google Patents

指尖检测方法、指尖检测装置、指尖检测设备及介质 Download PDF

Info

Publication number
WO2020048213A1
WO2020048213A1 PCT/CN2019/094733 CN2019094733W WO2020048213A1 WO 2020048213 A1 WO2020048213 A1 WO 2020048213A1 CN 2019094733 W CN2019094733 W CN 2019094733W WO 2020048213 A1 WO2020048213 A1 WO 2020048213A1
Authority
WO
WIPO (PCT)
Prior art keywords
depth
hand
point
fingertip
vertex
Prior art date
Application number
PCT/CN2019/094733
Other languages
English (en)
French (fr)
Inventor
徐楚
Original Assignee
京东方科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司 filed Critical 京东方科技集团股份有限公司
Priority to EP19857813.0A priority Critical patent/EP3848778B1/en
Priority to US16/647,765 priority patent/US11315265B2/en
Publication of WO2020048213A1 publication Critical patent/WO2020048213A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/0416—Control or interface arrangements specially adapted for digitisers
    • G06F3/0418—Control or interface arrangements specially adapted for digitisers for error correction or compensation, e.g. based on parallax, calibration or alignment
    • G06F3/04186—Touch location disambiguation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
    • G06F3/0426—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking fingers with respect to a virtual keyboard projected or printed on the surface
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects

Definitions

  • the present disclosure relates to the field of touch technology, and more particularly, to a fingertip detection method, a fingertip detection device, a fingertip detection device, and a medium.
  • fingertip detection and positioning as the key technology of virtual measurement and control, are also facing higher requirements.
  • a fingertip algorithm based on geodesic distance and a coordinate algorithm based on geometric features can be used.
  • the fingertip algorithm based on geodesic distance is limited by the gesture posture position when detecting the fingertip, and cannot detect the fingertip when the finger is facing the camera.
  • the coordinate algorithm based on geometric features has the disadvantages of slow calculation speed and poor calculation effect. These problems affect the accuracy and efficiency of fingertip detection and positioning, and make fingertip detection less robust in practical applications.
  • the present disclosure provides a fingertip detection method and device.
  • the fingertip detection method provided by the present disclosure can effectively locate the position of the fingertip, solve the influence of the gesture posture on the detection result during the detection of the fingertip, improve the detection accuracy and efficiency, and the method has good robustness.
  • a method for detecting fingertips including: determining a minimum depth point in a hand region to be detected based on a depth image; and determining a hand vertex in the hand region to be detected based on the depth image. Determining a fingertip position based on the minimum depth point and the hand apex.
  • determining a fingertip position based on the minimum depth point and the hand vertex includes: in a case where the minimum depth point and the hand vertex are the same pixel point, determining the pixel point as Fingertip position; if the minimum depth point and the hand vertex are different pixels, the minimum depth point and the hand vertex are processed based on a preset rule, and the fingertip position is determined based on the processing result .
  • the minimum depth point and the hand vertex are different pixel points, the minimum depth point and the hand vertex are processed based on a preset rule, and a finger is determined based on a processing result.
  • the tip position includes: determining a vertical axis distance between the minimum depth point and the hand vertex in the depth image; determining a depth distance between the minimum depth point and the hand vertex in the depth image; based on the depth distance Distance from vertical axis to determine fingertip position.
  • determining the position of the fingertip based on the depth distance and the vertical axis distance includes: when the difference between the depth distance and the vertical axis distance is greater than or equal to a distance threshold, determining the minimum depth point as the fingertip position ; In a case where a difference between the depth distance and the vertical axis distance is less than a distance threshold, determining a vertex of the hand as a fingertip position.
  • determining the minimum depth point in the hand region to be detected based on the depth image includes: for each pixel point in the hand region to be detected, using a given area around the pixel point as a determination region and using the pixel point as The center pixel point of the determination area; the number of pixels in the determination area where the depth difference between the center pixel point and the center pixel point is less than the depth difference threshold value; if the depth difference value between the center pixel point and the center pixel point is less than the depth difference threshold value If the number is greater than the threshold, the center pixel point is determined to be a stable point; the stable point with the smallest depth in the depth image is determined as the minimum depth point of the hand region to be detected.
  • determining the hand vertices in the hand region to be detected based on the depth image includes: for each pixel point in the hand region to be detected, a given area around the pixel point is used as a determination area, and the pixel point is used as The center pixel point of the determination area; the number of pixels in the determination area where the depth difference between the center pixel point and the center pixel point is less than the depth difference threshold value, and if the depth difference value with the center pixel point is less than the depth difference threshold value If the number is greater than the threshold, the center pixel point is determined to be a stable point; the stable point with the maximum height in the depth image is determined as the hand vertex of the hand region to be detected.
  • determining the hand vertices in the hand region to be detected based on the depth image includes: extracting a foreground hand effective depth map; and determining the Hand apex.
  • the extracting the foreground hand effective depth map comprises: determining a plurality of hand depths and a hand volume corresponding to each hand depth based on the depth image; according to the determined plurality of hand depths and each The volume of the hand corresponding to the depth of the hand determines the image capture position and obtains the effective depth map of the foreground hand.
  • the determining a plurality of hand depths and a hand volume corresponding to each hand depth includes: using the minimum depth point as a starting point, and obtaining the Depth section; for each depth section, determine the hand depth corresponding to the depth section, and determine the number of pixels in the depth section; based on the number of pixels in each depth section, determine the corresponding hand depth Hand volume.
  • determining the image interception position and obtaining the foreground hand effective depth map according to the determined hand depth and hand volume include: the hand volume corresponding to a depth section is greater than the volume threshold and the corresponding hand When the depth is not greater than the depth threshold, the hand depth corresponding to the previous depth section of the depth section is determined as the target arm depth; or the hand depth corresponding to a depth section is greater than the depth threshold and the calculated hand When the volume is not greater than the volume threshold, the depth of the hand corresponding to the previous depth section of the depth section is determined as the target arm depth; the image is intercepted according to the target arm depth to obtain the foreground hand effective depth map.
  • a fingertip detection device including: a minimum depth point detection module configured to determine a minimum depth point in a hand region to be detected based on a depth image; a hand vertex detection module configured to Based on the depth image, a hand vertex in the hand region to be detected is determined; a fingertip position determination module is configured to determine a fingertip position based on the minimum depth point and the hand vertex.
  • the fingertip position determination module determining the fingertip position based on the minimum depth point and the hand vertex includes: when the minimum depth point and the hand vertex are the same pixel point, Determine the pixel point as the position of a fingertip; if the minimum depth point and the hand vertex are different pixels, process the minimum depth point and the hand vertex based on a preset rule, based on The result of the process determines the position of the fingertips.
  • the fingertip position determination module processes the minimum depth point and the hand vertex based on a preset rule and Determining the position of the fingertip based on the processing result includes: determining a vertical axis distance between the minimum depth point and the hand vertex in the depth image; determining a depth between the minimum depth point and the hand vertex in the depth image Distance; determines fingertip position based on depth distance and vertical axis distance.
  • determining the position of the fingertip based on the depth distance and the vertical axis distance includes: when the difference between the depth distance and the vertical axis distance is greater than or equal to a distance threshold, setting the fingertip position The minimum depth point is determined as the position of the fingertip; when the difference between the depth distance and the vertical axis distance is less than the distance threshold, the vertex of the hand is determined as the position of the fingertip.
  • a fingertip detection device characterized in that the device includes a processor and a memory, the memory contains a set of instructions, and the set of instructions is set by the processor When executed, the fingertip detection device is caused to perform the method described above.
  • a computer-readable storage medium on which computer-readable instructions are stored, and the method as described above is performed when the instructions are executed by a computer.
  • the method By using the method, device and device for detecting fingertips provided by the present disclosure, noise interference in an image can be well removed.
  • the fingertips are in different orientations, detection and positioning of the position of the fingertips is achieved.
  • the method disclosed in the present disclosure can well overcome the problem of difficult fingertip recognition when the user gesture is rotated around the center of the hand on a plane perpendicular to the depth direction, further improving the robustness of fingertip detection and its Detection efficiency.
  • FIG. 1A illustrates an exemplary block diagram of a fingertip detection apparatus according to an embodiment of the present disclosure
  • FIG. 1B illustrates a schematic diagram of a user gesture according to an embodiment of the present disclosure
  • FIG. 1C illustrates a schematic diagram of another user gesture according to an embodiment of the present disclosure
  • FIG. 2 illustrates an exemplary flowchart of a fingertip detection method according to an embodiment of the present disclosure
  • FIG. 3 illustrates an exemplary flowchart of determining a minimum depth point in a hand region to be detected according to an embodiment of the present disclosure
  • FIG. 4 illustrates an exemplary flowchart of determining a hand vertex in a hand region to be detected according to an embodiment of the present disclosure
  • FIG. 5 illustrates an exemplary flowchart for extracting a foreground hand effective depth map (ED) according to an embodiment of the present disclosure
  • FIG. 6 illustrates an example of determining a fingertip position based on the minimum depth point N and the hand vertex T in a case where the minimum depth point (N) and the hand vertex (T) extracted according to the embodiment of the present disclosure are different pixel points.
  • Sex flow chart
  • FIG. 7 shows a schematic block diagram of a fingertip detection device according to an embodiment of the present disclosure.
  • modules in a system according to embodiments of the present disclosure, any number of different modules may be used and run on a user terminal and / or server.
  • the modules are merely illustrative, and different aspects of the systems and methods may use different modules.
  • a flowchart is used in the present disclosure to explain operations performed by a system according to an embodiment of the present disclosure. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the various steps can be processed in reverse order or simultaneously, as needed. At the same time, you can add other operations to these processes, or remove a step or steps from these processes.
  • FIG. 1A illustrates an exemplary block diagram of a fingertip detection apparatus according to an embodiment of the present disclosure.
  • the fingertip detection device 100 shown in FIG. 1A may be implemented as one or more dedicated or general-purpose computer system modules or components, such as a personal computer, a notebook computer, a tablet computer, a mobile phone, and a personal digital assistant (PDA). , Smart glasses, smart watches, smart rings, smart helmets and any smart portable devices.
  • the fingertip detection device 100 may include at least one processor 110 and a memory 120.
  • the at least one processor is configured to execute a program instruction.
  • the memory 120 may exist in the fingertip detection device 100 in different forms of a program storage unit and a data storage unit, such as a hard disk, a read-only memory (ROM), and a random access memory (RAM), which can be used to store a processor Processes and / or executes various data files used during fingertip detection and possible program instructions executed by the processor.
  • the fingertip detection device 100 may further include an input / output component to support input / output data flow between the fingertip detection device 100 and other components (such as the image acquisition device 130).
  • the fingertip detection apparatus 100 may also send and receive information and data from a network through a communication port.
  • the fingertip detection device 100 may receive a depth image collected from the image acquisition device 130 and perform a fingertip detection method described below on the received depth image data.
  • the image acquisition device may be, for example, a depth camera, and specifically may be, for example, a Realsense camera; it may also be a binocular camera, such as a Bumblebee binocular camera.
  • the fingertip detection device may output the obtained fingertip position, for example, the fingertip position may be output to a computer processing module for subsequent processing.
  • the fingertip detection device may output the obtained fingertip position, for example, the fingertip position may be output to a computer processing module for subsequent processing.
  • Embodiments of the present disclosure are not limited The specific manner in which the fingertip detection device outputs the position of the fingertip and the limitation of the specific device to which it is output.
  • the fingertip detection device may display the obtained position of the fingertip, for example, display it on a display panel or a control screen to achieve good human-computer interaction.
  • Embodiments of the present disclosure are not limited by the specific manner of displaying the position of the fingertip.
  • processor 110 and the memory 120 are presented as separate modules, those skilled in the art can understand that the above-mentioned device modules may be implemented as separate hardware devices or integrated into one or more hardware devices. As long as the principles described in this disclosure can be implemented, the specific implementation of different hardware devices should not be taken as a factor limiting the scope of protection of this disclosure.
  • the fingertip position is located at the minimum depth point N
  • the fingertip position is located at the vertex T of the hand.
  • the gesture gestures taken by the user are not limited by the embodiments of the present disclosure, and they can adopt various possible touch gestures.
  • the minimum depth point N and the hand vertex T change with different gestures of the user, and the position of the fingertip is also located in both the minimum depth point N of the hand to be detected and the hand vertex T according to the different gesture posture of the user One of the locations.
  • FIG. 2 illustrates an exemplary flowchart of a fingertip detection method according to an embodiment of the present disclosure.
  • a minimum depth point N in a hand region to be detected is determined based on the acquired depth image.
  • the depth image may be a depth image obtained by a depth camera, or may be a depth image obtained in other ways in advance.
  • the source of the depth image and the method of obtaining the depth image are not limited. For example, it may be a depth image directly acquired by a depth camera or a binocular camera, or it may be a depth image processed in advance by a computer.
  • a hand vertex T in the hand region to be detected is determined based on the acquired depth image. Determining the hand vertex T can be determined, for example, by comparing the height of each pixel in the hand region to be detected, that is, comparing the vertical axis of each pixel in the hand region to be detected in the direction of the vertical axis of the acquired depth image. Coordinate value.
  • steps S201 and S202 may be performed in parallel, or performed sequentially, and no limitation is imposed here. Further, as required, steps S201 and S202 may be performed based on different depth images obtained after preprocessing or local feature extraction, as long as these depth images are all derived from the same original depth image.
  • a fingertip position is further determined based on the minimum depth point N and the hand vertex T.
  • the determination of the position of the fingertip may be performed by logically judging the coordinates of the two points N and P, and selecting a point representing the position of the fingertip as the fingertip.
  • the system presets threshold conditions for comparison, and then determines the point representing the position of the fingertip from the two points of the minimum depth point N and the hand apex T.
  • FIG. 3 illustrates an exemplary flowchart of determining a minimum depth point in a hand region to be detected according to an embodiment of the present disclosure.
  • the minimum depth point N can be obtained through an area determination algorithm.
  • the area determination algorithm firstly, through step S301, for each pixel point in the hand region to be detected, the pixel point is determined.
  • the surrounding given area is used as the determination area and the pixel point is used as the central pixel point of the determination area.
  • the given region represents a pixel region having a preset size, which may be, for example, a pixel region formed by 5 * 5 pixels, or may be a pixel region formed by 8 * 8 pixels.
  • Embodiments of the present disclosure are not limited by the size of a given area that is set.
  • step S302 the stability judgment for a given pixel point is further implemented through step S302.
  • This operation first calculates the distance between each pixel point and the central point of the pixel point in the judgment area via S3021. The depth difference, and then, in step S3022, it is further determined that the number of pixels M whose depth difference between the determination area and the central pixel is less than the depth difference threshold, where the depth difference threshold is used to screen for stable conditions that meet the conditions Pixel center point. Finally, in step S3023, the discrimination is performed based on the previously calculated number M.
  • the center pixel is determined to be a stable point. ; If the number of pixels whose depth difference from the center point is less than the number threshold is less than or equal to the number threshold, the center point of the pixel is determined as a noise point.
  • the depth difference threshold is intended to represent a preset value of the absolute value of the depth difference between two pixels, and is used to filter the center point of a pixel that meets the conditions as a stable point, which can be set based on actual requirements and calculation accuracy.
  • the depth difference threshold may be 2 mm or 5 mm, and the embodiments of the present disclosure are not limited by the specific value of the set depth difference threshold.
  • the number threshold is intended to represent a preset number of values.
  • the number threshold can be the same value for all sizes of the decision area, for example, it is set to 15; or it can also be based on the size of the decision area.
  • the number threshold may be 8, for example, and when the size of the determination area is an 8 * 8 pixel point area, the number threshold may be 20, for example.
  • Embodiments of the present disclosure are not limited by the specific numerical value of the number threshold.
  • the depth difference threshold is set to 2mm, and the number threshold is set to 15. If the number of pixels whose depth difference is less than 2mm is greater than 15, the center pixel is determined to be a stable point. If the number of pixels whose depth difference from the center pixel is less than 2mm is less than or equal to 15 , It is determined that the central pixel point is a noise point.
  • One or more stable points will be obtained through the foregoing steps, and thereafter, they will be further screened in step S303.
  • it is the minimum depth point by default; in the case where there are multiple stable points, based on the depth value of each stable point, it will have the minimum depth stability in the acquired depth image
  • the point is determined as the minimum depth point N of the hand region to be detected.
  • the depth values are 9mm, 13mm, 15mm, 21mm, and 25mm, respectively.
  • a stable point with a depth value of 9mm is selected as the minimum depth point.
  • FIG. 4 illustrates an exemplary flowchart of determining a hand vertex T in a hand region to be detected according to an embodiment of the present disclosure.
  • the hand vertex T can be obtained by using the area determination algorithm.
  • the area determination algorithm will be further explained. For example, first through step S401, for each pixel point in the hand region to be detected, the pixel point is determined. The surrounding given area is used as the determination area and the pixel point is used as the central pixel point of the determination area.
  • the given region represents a pixel region having a preset size, which may be, for example, a pixel region formed by 5 * 5 pixels, or may be a pixel region formed by 8 * 8 pixels.
  • Embodiments of the present disclosure are not limited by the size of a given area that is set.
  • step S402. the depth between each pixel point in the judgment area and the center point of the pixel point is obtained through step S4021.
  • step S4022 it is further determined that the number M of pixel points in the determination area whose depth difference with the central pixel point is less than the depth difference threshold.
  • step S4023 the discrimination is performed based on the previously calculated number M. If the number of pixels M whose depth difference from the center pixel is less than the depth difference threshold is greater than the number threshold, the center pixel is determined to be a stable point. ; If the number of pixels whose depth difference from the center point is less than the number threshold is less than or equal to the number threshold, the center point of the pixel is determined as a noise point.
  • the depth difference threshold is intended to represent a preset value of the absolute value of the depth difference between two pixels, and is used to filter the center point of a pixel that meets the conditions as a stable point, which can be based on actual needs and calculation accuracy Make settings.
  • the depth difference threshold may be 2 mm or 5 mm, and the embodiments of the present disclosure are not limited by the specific value of the set depth difference threshold.
  • the number threshold is intended to represent a preset number of values.
  • the number threshold can be the same value for all sizes of the decision area, for example, it is set to 15; or it can also be based on the size of the decision area.
  • the number threshold may be 8, for example, and when the size of the determination area is an 8 * 8 pixel point area, the number threshold may be 20, for example.
  • Embodiments of the present disclosure are not limited by the specific numerical value of the number threshold.
  • One or more stable points will be obtained through the foregoing steps, and thereafter, they will be further screened in step S403. For example, in the case where there is only one stable point, it is the hand vertex T by default; in the case where there are multiple stable points, based on the vertical coordinate value of each stable point in the vertical axis direction of the depth image, The stable point having the maximum height in the acquired depth image, that is, the largest vertical axis coordinate value is determined as the hand vertex T of the hand region to be detected.
  • the height in the depth image may be represented by the number of pixels, spatial coordinates, or calibration coordinates, etc., and the present disclosure is not limited by the manner of obtaining the height of the pixels.
  • the foreground hand effective depth map may also be extracted from the depth image before the hand vertices in the region to be detected are determined based on the depth image.
  • the foreground hand effective depth map is a target depth image obtained after removing background pixel points in the current depth image that are not related to the hand region to be detected.
  • a hand vertex is determined based on the region of the hand to be detected and then based on the foreground hand effective depth map. It filters out background pixels that have low correlation with the area of the hand to be detected, and eliminates the interference of the environmental background on identifying the vertex T of the hand to be detected, thereby improving the accuracy of subsequent detection of the top T of the hand.
  • extracting the foreground hand effective depth map may include: determining a plurality of hand depths and a hand volume corresponding to each hand depth based on the depth image; according to the determined plurality of hand depths and each hand The volume of the hand corresponding to the depth of the hand, determine the image capture position, and obtain the effective depth map of the foreground hand.
  • the hand depth L represents a depth value at a current depth section in the depth image.
  • the multiple hand depths correspond to the depth values of multiple different depth sections in the depth image.
  • FIG. 5 illustrates an exemplary flowchart of extracting a foreground hand effective depth map (ED) according to an embodiment of the present disclosure.
  • ED foreground hand effective depth map
  • step S5011 using the depth value of the minimum depth point N that has been obtained as a starting point, according to a predetermined depth increment ⁇ m (unit: millimeter), traverse different sections of the depth map to obtain multiple depth points.
  • the predetermined depth increment ⁇ m can be set to 1 mm or 2 mm.
  • step S5012 the number of pixels P i in the current depth section is calculated, and the hand depth L i corresponding to the section is obtained.
  • the depth L i of the hand is the vertical distance from the minimum depth point N to the current depth section, that is, the difference between the depth value of the current depth section and the depth value of the minimum depth point N.
  • Li is the depth of the hand corresponding to the section
  • P i is the number of pixels on the section, where i is determined by the number of times the depth increment ⁇ m is performed, and its value is Positive integers 1, 2, 3 ...
  • the depth value of the minimum depth point N is 7mm, and the entire image is traversed in a depth increment of 2mm ⁇ m, then the fifth depth section reached after the fifth time depth increment is performed, that is, the depth value For a depth section of 17 mm, the corresponding hand depth L 5 is 10 mm.
  • the hand volume corresponding to each hand depth L i is determined based on the number of pixels P i in each depth section. V i .
  • the hand volume corresponding to each depth section can be split into the sum of the volume value corresponding to the previous section of the current depth section and the new volume value of the current depth section. It is specifically expressed as:
  • V i V i-1 + ⁇ V i
  • V i is the hand volume value corresponding to the current depth section
  • V i-1 is the hand volume value corresponding to the previous section of the current depth section
  • ⁇ V i is the new hand volume value of the current depth section.
  • i is a positive integer 1, 2, 3, ...
  • the value of the initial volume V 0 is set to 0.
  • the new hand volume value ⁇ V i of the current depth section can be further calculated from the current depth section area and depth increment ⁇ m. It is specifically expressed as:
  • S i is the cross-sectional area corresponding to the current depth section
  • ⁇ m is a predetermined depth increment.
  • each section has the same cross-sectional area as the current depth section.
  • P i is the number of pixels corresponding to the current depth section
  • the distance between the pixel point and the pixel point in the axial direction, the distance between the pixel point and the pixel point refers to the distance from one edge of a certain pixel point to the same edge of an adjacent pixel point, for example, a certain point the distance between the upper side from the upper side pixels adjacent pixels
  • R i is a ratio corresponding to the current pixel depth value with the depth profile of the focal length of the camera, for pixels in terms of units of millimeters.
  • the same depth section has a unique R i corresponding to it.
  • i is determined by the number of times the depth increment ⁇ m is performed, and the values are positive integers 1, 2, 3, ....
  • the volume value of the hand corresponding to the current depth section can be expressed as:
  • the distance between the pixel point and the pixel point in the axial direction, the distance between the pixel point and the pixel point refers to the distance from one edge of a certain pixel point to the same edge of an adjacent pixel point, for example, a certain point
  • R i is the ratio of the depth of the pixel to the focal length of the camera corresponding to the current depth section. It is used to convert the pixel unit to millimeters, and ⁇ m is the depth increment. . Based on the above calculation formula, the value of the hand depth L and the hand volume V corresponding to each depth section can be obtained.
  • the value of R 5 corresponding to the current depth section can be calculated as 5. If the number of pixels P 5 on the section is 500, and if the section is between the pixel point and the pixel point in the horizontal axis direction The distance lx is 0.3mm, and the distance ly between the pixel point and the pixel point in the vertical axis direction is 0.4mm. According to the calculation, the area s of a single pixel point is 0.12mm 2 , and the hand volume V corresponding to the previous depth section is V The value of 4 is 5000mm 3 , which corresponds to the hand volume value of the current section:
  • the interception of the arm position is further performed in step S502.
  • the arm position can be intercepted by judging the two parameters of the hand depth L and the hand volume V.
  • the volume threshold Vgs and the depth threshold Lgs can be set, and the corresponding hand depth L and hand obtained at each depth section The volume V value is compared with the depth and volume thresholds Lgs, Vgs.
  • the volume threshold is intended to represent a hand volume value, and the volume threshold can be set based on actual needs, for example, it is set to the average volume value of a human hand, or it is set to a measurement value of the current user's hand volume ;
  • the volume threshold may be, for example, 15000 mm 3 , or it may also be 20000 mm 3 .
  • the embodiments of the present disclosure are not limited by the manner in which the volume threshold is set and the specific numerical value that is set.
  • the depth threshold is intended to represent the length value of the hand in the depth direction.
  • the depth threshold can be set based on actual needs, for example, it can be set to the average length of a human hand, or it can be set to the current A measured value of the user's hand length; the depth threshold may be, for example, 150 mm, or it may be 170 mm.
  • the embodiments of the present disclosure are not limited by the manner in which the depth threshold is set and the specific value set by the depth threshold.
  • the corresponding hand depth L is determined as the target arm depth; or if the hand depth L corresponding to a depth section is greater than the depth threshold Lgs and the calculated hand volume V is not greater than the volume threshold Vgs, The hand depth L corresponding to the previous depth section is determined as the target arm depth. Then, based on the determined target arm depth, arm position interception can be performed on the original depth image to obtain an effective depth map of the foreground hand. For example, the depth value of all pixels after the depth of the target arm can be set to 0.
  • the depth map is traversed through the different sections of the depth map from the minimum depth point in 1mm increments.
  • Section corresponds to a hand depth L of 151mm, which is greater than 150mm, and when the hand volume V corresponding to the current depth section (131st section) is less than or equal to 15000mm 3 , the previous depth section (130th section) corresponds
  • the arm depth L is used as the target arm depth.
  • the hand volume V corresponding to the current depth section (131st section) is 15004mm 3 , which is greater than 15000mm 3 , and the hand depth L corresponding to the current depth section (131st section) is less than or equal to 150mm, then the previous one
  • the arm depth L corresponding to the depth section (130th section) is taken as the target arm depth.
  • a fingertip position will be further determined based on the minimum depth point N and the hand vertex T.
  • the minimum depth point N and the hand vertex T extracted based on the depth image may have the same or different position coordinates. If in the depth image, based on the position of the user's gesture, the extracted minimum depth point N and the hand vertex T are the same pixel point, the pixel point can be directly determined as the position of the fingertip. For example, when a user only extends a single finger for a touch operation, his fingertip may be located at the minimum depth point of the image and the vertex of the hand at the same time. When the extracted minimum depth point N and the hand vertex T are different pixels, the minimum depth point and the hand vertex will be further processed based on a preset rule, and the fingertip position will be determined based on the processing result. .
  • the preset rule may be selected according to actual requirements.
  • the rule may be: by logically judging the coordinates of the two points N and P above, and selecting a point representing the position of the fingertip as a fingertip, or by using the two
  • the distance or positional relationship between the points N and P is compared with a preset threshold condition of the system, and then from the two points of the minimum depth point N and the vertex T of the hand, a point representing the position of the fingertip is determined.
  • the embodiments of the present disclosure are not limited by the specific content of the preset rule.
  • the preset rule can be set in advance before image processing, for example, the same rule is used for all input images at this time; or it can be dynamically changed during image processing, for example, it can be based on the characteristics of the input image. Changed to use different preset rules for different input images.
  • the embodiments of the present disclosure are not limited by the application range of the preset rule and the setting time of the preset rule.
  • FIG. 6 shows an exemplary flowchart for determining the position of a fingertip based on the minimum depth point N and the hand vertex T in a case where the minimum depth point N and the hand vertex T are different pixel points extracted according to an embodiment of the present disclosure.
  • the operation of determining the fingertip position 600 based on the obtained minimum depth point N and the hand vertex T when the minimum depth point N and the hand vertex T are different pixels will be described below with reference to FIG. 6.
  • step S601 coordinate values D T and D N of the minimum depth point N and the hand vertex T on the vertical axis of the depth image are obtained. That is, the vertical axis coordinates of its pixel points.
  • the vertical axis coordinates can be the distance coordinates from the set origin or the pixel coordinates of the set origin.
  • the vertical axis distance ⁇ Y in the vertical axis direction of the projection point of the minimum depth point N and the hand vertex T on a plane perpendicular to the depth direction can be calculated, and the unit is millimeter. It can be specifically expressed as:
  • step S602 the depth values Z N and Z T of the minimum depth point N and the hand vertex T are obtained, and further, the depth distance of the minimum depth point N and the hand vertex T in the depth direction can be calculated.
  • ⁇ Z in millimeters It can be specifically expressed as:
  • steps S601 and S602 may be performed in parallel, or performed sequentially, and no limitation is imposed here.
  • step S603 a difference Diff between the depth distance ⁇ Z of the minimum depth point N and the hand vertex T and the vertical distance ⁇ Y is obtained, which is specifically expressed as:
  • the difference Diff is compared with the distance threshold.
  • the minimum depth point N is determined as the fingertip position; the difference between the depth distance ⁇ Z and the vertical axis distance ⁇ Y
  • the hand vertex T is determined as the position of the fingertip.
  • the distance threshold is a preset distance value, and the distance threshold may be set based on actual requirements, and may be set to 30 mm or 45 mm, for example.
  • the embodiments of the present disclosure are not limited by the specific value of the distance threshold and its setting manner.
  • the distance threshold is set to 30 mm. If the difference Diff between the depth distance ⁇ Z and the vertical axis distance ⁇ Y is greater than or equal to 30 mm, the minimum depth point N is the position of the fingertip; If the difference Diff of the vertical axis distance ⁇ Y is less than 30 mm, the vertex T of the hand is the position of the fingertip.
  • the fingertip detection method further includes the step of outputting the detected position of the fingertip.
  • the position of the fingertip may be output to a computer processing module for subsequent processing.
  • Embodiments of the present disclosure are not limited by the specific manner of outputting the position of the fingertip and the specific device to which it is output.
  • the method for detecting a fingertip further includes a step of displaying the obtained position of the fingertip, for example, displaying the position of the fingertip on a display panel or a control screen to achieve good human-computer interaction.
  • a step of displaying the obtained position of the fingertip for example, displaying the position of the fingertip on a display panel or a control screen to achieve good human-computer interaction.
  • Embodiments of the present disclosure are not limited by the specific manner of displaying the position of the fingertip.
  • FIG. 7 shows a schematic block diagram of a fingertip detection device according to an embodiment of the present disclosure.
  • the fingertip detection device 700 may include a minimum depth point detection module 710, a hand vertex detection module 720, and a fingertip position determination module 730.
  • the minimum depth point detection module 710 is configured to determine the minimum depth point N in the hand region to be detected based on the depth image by processing the image, which can execute the process shown in FIG. 3 to achieve the minimum depth point Determination of the depth point N.
  • the hand vertex detection module 720 is configured to determine the hand vertex T in the hand region to be detected based on the depth image.
  • the hand vertex detection module 720 can perform the process shown in FIG. 4 to realize the hand vertex T OK.
  • the fingertip position determination module 730 is configured to determine a fingertip position based on the minimum depth point N and the hand apex T. Among them, the position of the fingertip can be determined according to the coordinate position relationship between the minimum depth point and the vertex of the hand. When the minimum depth point is the same pixel point as the vertex of the hand, the pixel point can be determined as the fingertip position. If the minimum depth point and the vertex of the hand are different pixels, the process shown in FIG. 6 may be performed to determine the position of the fingertip.
  • the fingertip detection device may further include an output device.
  • the output device is connected to an external computing device, and is designed to output the obtained fingertip position to the computing device for subsequent processing operations.
  • the computing device may be, for example, a computer or a mobile phone. Examples are not limited by the type of the computing device and the manner in which the output device is connected to the external computing device.
  • the fingertip detection device may further include a display device, and the display device may display the obtained position of the fingertip so as to achieve good human-computer interaction with the user.
  • the display device may be, for example, a display device.
  • a panel or a display screen, embodiments of the present disclosure are not limited by the specific type of the display device.
  • a computer-readable storage medium on which computer-readable instructions are stored, and when the instructions are executed by a computer, the method described above can be performed.
  • the program part in the technology may be considered as a “product” or “article of manufacture” existing in the form of executable code and / or related data, which is participated or realized through a computer-readable medium.
  • Tangible, permanent storage media may include memory or storage used by any computer, processor, or similar device or related module.
  • various semiconductor memories, magnetic tape drives, magnetic disk drives or similar devices capable of providing storage functions for software.
  • All software or parts of it may sometimes communicate over a network, such as the Internet or other communication networks.
  • This type of communication can load software from one computer device or processor to another.
  • another medium capable of transmitting software elements can also be used as a physical connection between local devices, such as light waves, radio waves, electromagnetic waves, etc., and is transmitted through cables, optical cables, or air.
  • the physical medium used for carrier waves, such as electrical cables, wireless connections, or optical cables, can also be considered as the medium that carries the software.
  • tangible "storage” media is restricted, other terms referring to computer or machine "readable media” refer to media that participates in the execution of any instruction by a processor.
  • the present disclosure uses specific words to describe embodiments of the present disclosure.
  • “one embodiment”, “an embodiment”, and / or “some embodiments” means a certain feature, structure, or characteristic related to at least one embodiment of the present disclosure. Therefore, it should be emphasized and noted that the “one embodiment” or “one embodiment” or “an alternative embodiment” mentioned two or more times in different places in this specification does not necessarily refer to the same embodiment .
  • certain features, structures, or characteristics in one or more embodiments of the present disclosure may be appropriately combined.
  • aspects of the present disclosure can be illustrated and described through several patentable categories or situations, including any new and useful process, machine, product, or combination of materials, or to them Any new and useful improvements. Accordingly, aspects of the present disclosure may be performed entirely by hardware, may be performed entirely by software (including firmware, resident software, microcode, etc.), or may be performed by a combination of hardware and software.
  • the above hardware or software can be called “data block”, “module”, “engine”, “unit”, “component” or “system”.
  • aspects of the present disclosure may manifest as a computer product located in one or more computer-readable media, the product including computer-readable program code.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)

Abstract

公开了一种指尖检测的方法、指尖检测装置、指尖检测设备及介质。其中指尖检测方法包括：基于深度图像，确定待检测手部区域中的最小深度点(N)；基于深度图像，确定待检测手部区域中的手部顶点(T)；基于所述最小深度点(N)与所述手部顶点(T)，确定指尖位置。基于手部顶点(T)和最小深度点(N)，在用户采取不同手势姿态下，准确高效地实现对指尖的检测。

Description

指尖检测方法、指尖检测装置、指尖检测设备及介质
相关申请的交叉引用
本公开要求于2018年09月03日提交的中国专利申请第201811021272.4号的优先权，该中国专利申请的全文通过引用的方式结合于此以作为本公开的一部分。
技术领域
本公开涉及触控技术领域，更具体地涉及一种指尖检测方法、指尖检测装置、指尖检测设备及介质。
背景技术
随着虚拟触控在民用和商用领域的不断发展，指尖检测定位作为虚拟测控的关键技术，也面临着更高的要求。对于指尖的检测定位，可以采用基于测地距离的指尖算法和基于几何特征的坐标算法。
然而，基于测地距离的指尖算法在检测指尖时受到手势姿态位置的限制，当手指正对摄像头时不能检测出指尖。基于几何特征的坐标算法则具有计算速度慢，计算效果差的缺陷。这些问题影响指尖检测定位的精度和效率，并使得指尖检测在实际应用中具有较低的鲁棒性。
因此，需要一种既能够不受手势姿态位置影响、且检测速度快的指尖检测方法。
发明内容
针对以上问题，本公开提供了一种指尖检测方法及装置。利用本公开提供的指尖检测方法可以有效地定位指尖位置，解决了指尖检测时手势姿态对于检测结果的影响，提高了检测精度及效率，且该方法具有良好的鲁棒性。
根据本公开的一方面，提出了一种指尖检测的方法，包括：基于深度图像，确定待检测手部区域中的最小深度点；基于深度图像，确定待检测手部区域中的手部顶点；基于所述最小深度点与所述手部顶点，确定指尖位置。
在一些实施例中，基于所述最小深度点与所述手部顶点确定指尖位置包括：在所述最小深度点和所述手部顶点为同一像素点的情况下，将该像素点确定为指尖位置；在所述最小深度点和所述手部顶点为不同像素点的情况下， 基于预设规则对所述最小深度点和所述手部顶点进行处理，基于处理结果确定指尖位置。
在一些实施例中，在所述最小深度点和所述手部顶点为不同像素点的情况下，基于预设规则对所述最小深度点和所述手部顶点进行处理并基于处理结果确定指尖位置包括：确定深度图像中所述最小深度点和所述手部顶点之间的纵轴距离；确定深度图像中所述最小深度点和所述手部顶点之间的深度距离；基于深度距离和纵轴距离，确定指尖位置。
在一些实施例中，基于深度距离和纵轴距离确定指尖位置包括：在所述深度距离与纵轴距离的差值大于等于距离阈值的情况下，将所述最小深度点确定为指尖位置；在所述深度距离与纵轴距离的差值小于距离阈值的情况下，将所述手部顶点确定为指尖位置。
在一些实施例中，基于深度图像确定待检测手部区域中最小深度点包括：对于待检测手部区域中每一个像素点，将该像素点周围给定区域作为判定区域并将该像素点作为该判定区域的中心像素点；判断该判定区域内与该中心像素点的深度差值小于深度差阈值的像素点个数，若与中心像素点的深度差值小于深度差阈值的像素点个数大于个数阈值，则判断该中心像素点为稳定点；将所述深度图像中具有最小深度的稳定点确定为待检测手部区域的最小深度点。
在一些实施例中，基于深度图像确定待检测手部区域中手部顶点包括：对于待检测手部区域中每一个像素点，将该像素点周围给定区域作为判定区域并将该像素点作为该判定区域的中心像素点；判断该判定区域内与该中心像素点的深度差值小于深度差阈值的像素点个数，若与中心像素点的深度差值小于深度差阈值的像素点个数大于个数阈值，则判断该中心像素点为稳定点；将所述深度图像中具有最大高度的稳定点确定为待检测手部区域的手部顶点。
在一些实施例中，所述基于深度图像确定待检测手部区域中的手部顶点包括：提取前景手有效深度图；基于所述前景手有效深度图(ED)，确定待检测手部区域中的手部顶点。
在一些实施例中，所述提取前景手有效深度图包括：基于深度图像，确定多个手部深度及与每个手部深度对应的手部体积；根据所确定多个手部深度和每个手部深度对应的手部体积，确定图像截取位置，获取前景手有效深 度图。
在一些实施例中，所述确定多个手部深度及与每个手部深度对应的手部体积包括：以所述最小深度点为起点，按照预定深度增量，获取多个深度点处的深度截面；对于每个深度截面，确定该深度截面所对应的手部深度，并且确定该深度截面中的像素点个数；基于各个深度截面中的像素点个数，确定各个手部深度所对应的手部体积。
在一些实施例中，根据所确定的手部深度和手部体积确定图像截取位置并获取前景手有效深度图包括：在一深度截面所对应的手部体积大于体积阈值且其所对应的手部深度不大于深度阈值的情况下，将该深度截面的前一深度截面所对应的手部深度确定为目标手臂深度；或者在一深度截面所对应的手部深度大于深度阈值且所计算的手部体积不大于体积阈值的情况下，将该深度截面的前一深度截面所对应的手部深度确定为目标手臂深度；按照该目标手臂深度进行图像截取，得到前景手有效深度图。
根据本公开的另一方面，提供了一种指尖检测装置，包括：最小深度点检测模块，配置成基于深度图像，确定待检测手部区域中的最小深度点；手部顶点检测模块，配置成基于深度图像，确定待检测手部区域中的手部顶点；指尖位置确定模块，配置成基于所述最小深度点与所述手部顶点，确定指尖位置。
在一些实施例中，指尖位置确定模块基于所述最小深度点与所述手部顶点，确定指尖位置包括：在所述最小深度点和所述手部顶点为同一像素点的情况下，将该像素点确定为指尖位置；在所述最小深度点和所述手部顶点为不同像素点的情况下，基于预设规则对所述最小深度点和所述手部顶点进行处理，基于处理结果确定指尖位置。
在一些实施例中，在所述最小深度点和所述手部顶点为不同像素点的情况下，指尖位置确定模块基于预设规则对所述最小深度点和所述手部顶点进行处理并基于处理结果确定指尖位置包括：确定深度图像中所述最小深度点和所述手部顶点之间的纵轴距离；确定深度图像中所述最小深度点和所述手部顶点之间的深度距离；基于深度距离和纵轴距离，确定指尖位置。
在一些实施例中，在该指尖检测装置中，基于深度距离和纵轴距离确定指尖位置包括：在所述深度距离与纵轴距离的差值大于等于距离阈值的情况下，将所述最小深度点确定为指尖位置；在所述深度距离与纵轴距离的差值 小于距离阈值的情况下，将所述手部顶点确定为指尖位置。
根据本公开的另一方面，还提供了一种指尖检测设备，其特征在于，所述设备包括处理器和存储器，所述存储器包含一组指令，所述一组指令在由所述处理器执行时使所述指尖检测设备执行如前所述的方法。
根据本公开的另一方面，还提供了一种计算机可读存储介质，其上存储有计算机可读的指令，当利用计算机执行所述指令时执行如前所述的方法。
利用本公开提供的指尖检测的方法、装置及设备，可以很好地除去图像中的噪声干扰，当指尖处于不同方位的情况下，对于指尖位置实现检测定位。特别地，本公开所述方法可以良好地克服用户手势在垂直于深度方向的平面上处于绕手部中心旋转的位置时指尖识别困难的问题，进一步提高了指尖检测的鲁棒性及其检测效率。
附图说明
为了更清楚地说明本公开实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员而言，在没有做出创造性劳动的前提下，还可以根据这些附图获得其他的附图。以下附图并未刻意按实际尺寸等比例缩放绘制，重点在于示出本公开的主旨。
图1A示出了根据本公开的实施例的指尖检测设备的示例性的框图；
图1B示出了根据本公开的实施例的用户手势的示意图；
图1C示出了根据本公开的实施例的另一种用户手势的示意图；
图2示出了根据本公开的实施例的指尖检测方法的示例性的流程图；
图3示出了根据本公开的实施例确定待检测手部区域中最小深度点的示例性流程图；
图4示出了根据本公开的实施例确定待检测手部区域中手部顶点的示例性流程图；
图5示出了根据本公开的实施例提取前景手有效深度图(ED)的示例性流程图；
图6示出了根据本公开的实施例提取的最小深度点(N)和手部顶点(T)为不同像素点的情况下，基于最小深度点N与手部顶点T确定指尖位置的示例性流程图；
图7示出了根据本公开的实施例的指尖检测装置的示意性的框图。
具体实施方式
下面将结合附图对本公开实施例中的技术方案进行清楚、完整地描述，显而易见地，所描述的实施例仅仅是本公开的部分实施例，而不是全部的实施例。基于本公开实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，也属于本公开保护的范围。
如本公开和权利要求书中所示，除非上下文明确提示例外情形，“一”、“一个”、“一种”和/或“该”等词并非特指单数，也可包括复数。一般说来，术语“包括”与“包含”仅提示包括已明确标识的步骤和元素，而这些步骤和元素不构成一个排它性的罗列，方法或者设备也可能包含其他的步骤或元素。
虽然本公开对根据本公开的实施例的系统中的某些模块做出了各种引用，然而，任何数量的不同模块可以被使用并运行在用户终端和/或服务器上。所述模块仅是说明性的，并且所述系统和方法的不同方面可以使用不同模块。
本公开中使用了流程图用来说明根据本公开的实施例的系统所执行的操作。应当理解的是，前面或下面操作不一定按照顺序来精确地执行。相反，根据需要，可以按照倒序或同时处理各种步骤。同时，也可以将其他操作添加到这些过程中，或从这些过程移除某一步或数步操作。
图1A示出了根据本公开的实施例的指尖检测设备的示例性框图。如图1A所示的指尖检测设备100可以实现为一个或多个专用或通用的计算机系统模块或部件，例如个人电脑、笔记本电脑、平板电脑、手机、个人数码助理(personal digital assistance，PDA)、智能眼镜、智能手表、智能指环、智能头盔及任何智能便携设备。其中，指尖检测设备100可以包括至少一个处理器110及存储器120。
其中，所述至少一个处理器用于执行程序指令。所述存储器120在指尖检测设备100中可以以不同形式的程序储存单元以及数据储存单元存在，例如硬盘、只读存储器(ROM)、随机存取存储器(RAM)，其能够用于存储处理器处理和/或执行指尖检测过程中使用的各种数据文件，以及处理器所执行的可能的程序指令。虽然未在图中示出，但指尖检测设备100还可以包括一个输入/输出组件，支持指尖检测设备100与其他组件(如图像采集装置130)之间的输入/输出数据流。指尖检测设备100也可以通过通信端口从网络发送 和接收信息及数据。
在一些实施例中，指尖检测设备100可以接收来自图像采集装置130所采集的深度图像，并对接收的深度图像的数据执行下文描述的指尖检测方法。所述图像采集装置可以例如为深度摄像头，具体地可以例如为实感(Realsense)摄像头；其也可以为双目摄像头，例如大黄蜂(Bumblebee)双目摄像头。
在一些实施例中，所述指尖检测设备可以将所得到的指尖位置输出，例如可以将所述指尖位置输出至计算机处理模块以进行后续的处理，本公开的实施例不受所述指尖检测设备将所述指尖位置输出的具体方式及其所输出至的具体设备的限制。
在一些实施例中，所述指尖检测设备可以将所得到的指尖位置显示，例如将其显示在显示面板或控制屏幕上，以实现良好的人机交互。本公开的实施例不受显示该指尖位置的具体方式的限制。
尽管在图1A中，处理器110、存储器120呈现为单独的模块，本领域技术人员可以理解，上述装置模块可以被实现为单独的硬件设备，也可以被集成为一个或多个硬件设备。只要能够实现本公开描述的原理，不同的硬件设备的具体实现方式不应作为限制本公开保护范围的因素。
在由图像采集装置130采集到的用户触控手势中，待检测的手部区域上存在与摄像头XY平面垂直距离最小的像素点，即待检测的手部区域中具有最小深度值的最小深度点N；以及在所采集的深度图像的纵轴方向上，待检测的手部区域中存在具有最大纵轴坐标值的像素点，其为待检测的手部区域中具有最大高度的像素点，即手部顶点T。图1B及1C中以示意图的方式给出了在用户采用两种不同触控手势时，最小深度点N和手部顶点T的相应位置。在图1B中，指尖位置位于最小深度点N处，在图1C中，指尖位置位于手部顶点T处。可以理解的是，用户所采取的手势姿态不受本公开实施例的限制，其可采取各种可能的触控手势。其中最小深度点N和手部顶点T随着用户手势姿态的不同而变化，指尖位置也根据用户不同的手势姿态情况，位于待检测手部的最小深度点N和手部顶点T二者其中之一的位置。
因此，为在用户采取各种触控手势的情况下，良好地捕获最小深度点N和手部顶点T的位置并进一步实现对于指尖位置的检测，本公开提出了一种指尖检测方法。图2示出了根据本公开的实施例的指尖检测方法的示例性流 程图。
如图2所示，根据指尖检测方法200，首先在步骤S201中，基于已取得的深度图像，确定待检测手部区域中的最小深度点N。如前所述，该深度图像可以是通过深度摄像头获取的深度图像，或者也可以是预先以其他方式获得的深度图像。在本公开实施例不受深度图像的来源及获取方式的限制。例如，可以为由深度摄像头或双目摄像头直接采集的深度图像，或者可以是经过计算机预先处理后的深度图像。
在步骤S202中，基于已取得的深度图像，确定待检测手部区域中的手部顶点T。确定手部顶点T例如可以通过比较待检测手部区域内各像素点所具有的高度来确定，即比较待检测手部区域内各像素点在所采集的深度图像的纵轴方向上的纵轴坐标值。
应了解，步骤S201和S202的操作可以并行进行，或者按照顺序执行，在此不对其作出任何限制。进一步地，根据需要，步骤S201及S202可基于经预处理或局部特征提取后得到的不同深度图像进行操作，只要这些深度图像皆来源于相同的原始深度图像即可。
在获取了最小深度点N和手部顶点T之后，在步骤S203中，将进一步基于所述最小深度点N与所述手部顶点T，确定指尖位置。该确定指尖位置例如可通过对上述两点N、P自身坐标进行逻辑判别，从中选择表征指尖位置的点作为指尖，也可以通过将上述两点N、P间的距离或位置关系与系统预设阈值条件进行比较，进而从最小深度点N和手部顶点T两点中，确定表征指尖位置的点。
图3示出了根据本公开的实施例确定待检测手部区域中最小深度点的示例性流程图。
参照图3中的流程图可知，可经由区域判定算法获取最小深度点N，所述区域判定算法具体而言，首先经由步骤S301，对于待检测手部区域中每一个像素点，将该像素点周围给定区域作为判定区域并将该像素点作为该判定区域的中心像素点。
所述给定区域表征具有预设尺寸的像素点区域，其例如可以为5*5个像素点所形成的像素点区域，或者也可以为8*8个像素点所形成的像素点区域。本公开的实施例不受所设置的给定区域的尺寸的限制。
完成中心像素点及其相应判定区域的划分后，通过步骤S302进一步地实 现对于给定像素点的稳定性判别，该操作首先经由S3021计算在判定区域内各像素点与像素点中心点之间的深度差值，其后，在步骤S3022中，进一步判断该判定区域内与该中心像素点的深度差值小于深度差阈值的像素点的个数M，其中深度差阈值用于筛选符合条件的稳定像素点中心点。最后，在步骤S3023中，基于先前计算的个数M进行判别，若与中心像素点的深度差值小于深度差阈值的像素点个数M大于个数阈值，则判断该中心像素点为稳定点；若与中心点的深度差值小于个数阈值的像素点个数小于或等于个数阈值，则该像素点中心点被判定为噪点。
所述深度差阈值旨在表征两个像素点的深度差的绝对值的预设数值，用于筛选符合条件的像素点中心点作为稳定点，其可以基于实际需求和计算精度进行设置。例如深度差阈值可以为2mm或5mm，本公开的实施例不受所设置的深度差阈值的具体数值的限制。
所述个数阈值旨在表征预设的个数数值，所述个数阈值例如可以对于所有尺寸的判定区域均为相同数值，例如设置其为15；或者其也可以基于判定区域的尺寸大小而变化，例如当判定区域的尺寸为5*5像素点区域时，所述个数阈值例如可以为8，当判定区域的尺寸为8*8像素点区域时，所述个数阈值例如可以为20。本公开的实施例不受所述个数阈值的具体数值的限制。
上述过程可更具体地描述，例如选取中心点周围5*5像素点区域为判定区域时，设置深度差阈值为2mm，设置个数阈值为15；则若在该5*5区域内，与中心像素点的深度差值小于2mm的像素点的个数大于15个，则判定该中心像素点为稳定点，若与中心像素点的深度差值小于2mm的像素点的个数小于或等于15个，则判定该中心像素点为噪点。
通过前述步骤将获得一个或多个的稳定点，其后，在步骤S303中对其进行进一步地筛选。例如，在仅存在一个稳定点的情况下，默认其为最小深度点；在存在多个稳定点的情况下，基于各稳定点的深度值，将在已取得的深度图像中具有最小深度的稳定点确定为待检测手部区域的最小深度点N。例如筛选后的待检测手部区域中的稳定点有5个，其深度值分别为9mm、13mm、15mm、21mm、25mm，则选择深度值为9mm的稳定点作为最小深度点。
图4示出了根据本公开的实施例确定待检测手部区域中手部顶点T的示例性流程图。
参照该流程图可知，可利用区域判定算法获取手部顶点T，接下来将对 于区域判定算法进一步说明，例如，首先经由步骤S401，对于待检测手部区域中每一个像素点，将该像素点周围给定区域作为判定区域并将该像素点作为该判定区域的中心像素点。
所述给定区域表征具有预设尺寸的像素点区域，其例如可以为5*5个像素点所形成的像素点区域，或者也可以为8*8个像素点所形成的像素点区域。本公开的实施例不受所设置的给定区域的尺寸的限制。
完成中心像素点及其相应判定区域的划分后，通过步骤S402进一步地实现对于给定像素点的稳定性判别，首先经由步骤S4021获取该判定区域内各像素点与像素点中心点之间的深度差值，其后，在步骤S4022中，进一步判断该判定区域内与该中心像素点的深度差值小于深度差阈值的像素点的个数M。最后，在步骤S4023中，基于先前计算的个数M进行判别，若与中心像素点的深度差值小于深度差阈值的像素点个数M大于个数阈值，则判断该中心像素点为稳定点；若与中心点的深度差值小于个数阈值的像素点个数小于或等于个数阈值，则该像素点中心点被判定为噪点。
如前所述，所述深度差阈值旨在表征两个像素点的深度差的绝对值的预设数值，用于筛选符合条件的像素点中心点作为稳定点，其可以基于实际需求和计算精度进行设置。例如深度差阈值可以为2mm或5mm，本公开的实施例不受所设置的深度差阈值的具体数值的限制。
所述个数阈值旨在表征预设的个数数值，所述个数阈值例如可以对于所有尺寸的判定区域均为相同数值，例如设置其为15；或者其也可以基于判定区域的尺寸大小而变化，例如当判定区域的尺寸为5*5像素点区域时，所述个数阈值例如可以为8，当判定区域的尺寸为8*8像素点区域时，所述个数阈值例如可以为20。本公开的实施例不受所述个数阈值的具体数值的限制。
通过前述步骤将获得一个或多个稳定点，其后，在步骤S403中对其进一步地筛选。例如，在仅存在一个稳定点的情况下，默认其为手部顶点T；在存在多个稳定点的情况下，基于各稳定点在深度图像的纵轴方向上的纵轴坐标值，将在已取得的深度图像中具有最大高度，即纵轴坐标值最大的稳定点确定为待检测手部区域的手部顶点T。例如筛选后的待检测手部区域中的稳定点有3个，其在已取得的深度图像中的纵轴方向上的坐标，亦即其高度分别为H1、H2、H3，且H1<H2<H3，则选择高度为H3的稳定点作为待检测手部区域手部顶点。应了解，深度图像中的高度可以通过像素点数量、空间坐 标、或者标定坐标等表示，本公开不受得到像素点高度的方式的限制。
在一些实施例中，在基于深度图像确定待检测区域中的手部顶点之前，还可以从深度图像中提取前景手有效深度图。其中，所述前景手有效深度图即为去除当前深度图像中与待检测手部区域无关的背景像素点之后所得到的目标深度图像。基于待检测手部区域的然后再基于所述前景手有效深度图来确定手部顶点。其通过过滤与待检测手部区域相关度较低的背景像素点，排除了环境背景对识别待检测手部顶点T的干扰，进而提升后续手部顶尖T检测的精确度。
在一些实施例中，提取前景手有效深度图可以包括：基于深度图像，确定多个手部深度及与每个手部深度对应的手部体积；根据所确定多个手部深度和每个手部深度对应的手部体积，确定图像截取位置，获取前景手有效深度图。其中所述手部深度L表示深度图像中当前深度截面处所具有的深度值。多个手部深度对应于深度图像中多个不同深度截面所具有的深度值。
图5示出了根据本公开的实施例提取前景手有效深度图(ED)的示例性流程图。作为示例，下面将参考图5，进一步说明提取前景手有效深度图500的步骤。
如图5所示，首先通过步骤S5011，以已经得到的最小深度点N的深度值作为起点，按照预定深度增量Δm(单位：毫米)，遍历深度图的不同截面以获取多个深度点处的深度截面。其中预定深度增量Δm可设置为1mm或2mm。进一步地，在每一截面(第i截面)上，根据步骤S5012，计算当前深度截面中所具有的像素点个数P i，并得到该截面所对应的手部深度L i。其中，所述手部深度L i即为最小深度点N到当前深度截面的垂直距离，即当前深度截面所具有的深度值与最小深度点N的深度值之间的差值。简言之，对于第i个截面，L i为对应于该截面的手部深度，P i为该截面上的像素点个数，其中i由执行深度增量Δm的次数确定，其取值为正整数1、2、3……。例如，例如在最小深度点N的深度值为7mm的情况下，以2mm的深度增量Δm历遍整个图像，则针对第五次执行深度增量后到达的第五个深度截面，即深度值为17mm的深度截面而言，其对应的手部深度L 5为10mm。
据此得到当前深度截面所对应的手部深度L i后，进一步地，在步骤S5013中，基于各个深度截面中的像素点个数P i，确定各个手部深度L i所对应的手部体积V i。例如，各深度截面所对应的手部体积，可拆分为当前深度截面的 前一截面所对应的体积值与当前深度截面新增的体积值的和。其具体表示为：
V i＝V i-1+ΔV i
其中，V i为当前深度截面所对应的手部体积值，V i-1为当前深度截面的前一截面所对应的手部体积值；ΔV i为当前深度截面新增的手部体积值。其中i取正整数1、2、3……，且设定初始体积V 0的值为0。当前深度截面新增的手部体积值ΔV i又可进一步地由当前深度截面面积与深度增量Δm进行计算。其具体表示为：
ΔV i＝S i*Δm
其中S i为当前深度截面所对应的截面面积，Δm为预定的深度增量。此处近似地认为，在深度增量Δm的长度内，每一个截面均与当前深度截面具有相同的截面面积。其中，当前深度截面的截面面积S i可以根据该深度截面中的像素点个数进行计算。例如针对深度摄像机或摄像头所捕获的深度图像，在计算当前深度截面面积S i中，需要将像素点单位换算为毫米单位，其换算公式为“毫米单位＝像素点单位*l*R(其中R＝像素点的深度值/相机焦距，l为像素点与像素点之间的距离”，基于该换算公式，其截面面积计算公式如下：
Si＝P i*s*R i*R i
其中P i为对应于当前深度截面的像素点个数；s为单个像素点的面积，其中，s＝lx*ly，lx为横轴方向上像素点与像素点之间的距离，ly为纵轴方向上像素点与像素点之间的距离，所述像素点与像素点之间的距离指的是某一像素点的一个边到相邻像素点的同一边之间的距离，例如某一像素点的上边距离相邻像素点上边之间的距离；R i为对应于当前深度截面的像素点深度值与相机焦距的比值，用于将像素点单位换算为毫米单位。由于对同一深度截面而言，位于其上的像素点具有相同的深度值及相机焦距值，因此同一深度截面具有与之对应的唯一R i。如前所述，i由执行深度增量Δm的次数确定，取值为正整数1、2、3……。综上所述，则对应于当前深度截面的手部的体积值可表示为：
V i＝V i-1+ΔV i＝V i-1+P i*s*R i*R i*Δm
其中P i为对应于当前深度截面的像素点个数；s为单个像素点的面积，其中，s＝lx*ly，lx为横轴方向上像素点与像素点之间的距离，ly为纵轴方向上像素点与像素点之间的距离，所述像素点与像素点之间的距离指的是某一 像素点的一个边到相邻像素点的同一边之间的距离，例如某一像素点的上边距离相邻像素点上边之间的距离；R i为对应于当前深度截面的像素点深度值与相机焦距的比值，用于将像素点单位换算为毫米单位，Δm为深度增量。基于以上计算公式，可得到对应于每个深度截面的手部深度L的数值和手部体积V的数值。
以上步骤可以更具体地描述，例如在最小深度点的深度值为5mm时，深度增量Δm取2mm，则对于当前的第5截面，当前深度截面的深度值为15mm，摄像头的相机焦距值为3mm，则计算可得当前深度截面对应的R 5的值为5，若截面上的像素点个数P 5为500个，且若该截面上，在横轴方向像素点与像素点之间上的距离lx为0.3mm，在纵轴方向上像素点与像素点之间的距离ly为0.4mm，则计算可知单个像素点的面积s为0.12mm 2，前一深度截面对应的手部体积V 4的值为5000mm 3，则对应于当前截面的手部体积值：
V 5＝5000mm 3+500*0.12*5*5*2＝5000+3000＝8000mm 3
其后，基于在当前深度截面下，已确定的手部深度L和手部体积V，进一步地通过步骤S502来实现截取手臂位置的截取。其中可以通过对手部深度L和手部体积V两个参数的判别来截取手臂位置，例如可以设定体积阈值Vgs和深度阈值Lgs，将在每一深度截面获得的对应手部深度L和手部体积V数值与深度及体积阈值Lgs、Vgs进行比较。
所述体积阈值旨在表征手部体积数值，该体积阈值可以基于实际需要进行设置，例如将其设置为人体手部的平均体积数值，或者将其设置为当前使用者的手部体积的测量值；所述体积阈值例如可以为15000mm 3，或者其也可以为20000mm 3。本公开的实施例不受所述体积阈值的设置方式及其所设置的具体数值的限制。
所述深度阈值旨在表征手部在深度方向上所具有的长度数值，该深度阈值例如可以基于实际需要进行设置，例如将其设置为人体手部的平均长度，或者也可以将其设置为当前使用者的手部长度的测量值；所述深度阈值例如可以为150mm，或者其也可以为170mm。本公开的实施例不受所述深度阈值的设置方式及其所设置的具体数值的限制。
在一些实施例中，当在一深度截面所对应的手部体积V大于体积阈值Vgs且其所对应的手部深度L不大于深度阈值Lgs的情况下，将该深度截面的前一深度截面所对应的手部深度L确定为目标手臂深度；或者在一深度截面所 对应的手部深度L大于深度阈值Lgs且所计算的手部体积V不大于体积阈值Vgs的情况下，将该深度截面的前一深度截面所对应的手部深度L确定为目标手臂深度。而后，可基于所确定的目标手臂深度，在原始的深度图像上进行手臂位置截取，得到前景手有效深度图。例如可将目标手臂深度之后的所有像素点的深度值设为0。
上述步骤可更具体地描述，例如当深度阈值Lgs为150mm且体积阈值Vgs为15000mm 3的情况下，以1mm深度增量从最小深度点开始历遍深度图不同截面，若当前深度截面(第131截面)对应的手部深度L为151mm，其大于150mm，且当前深度截面(第131截面)对应的手部体积V小于或等于15000mm 3时，则将前一深度截面(第130截面)所对应的手臂深度L作为目标手臂深度。或者若当前深度截面(第131截面)对应的手部体积V为15004mm 3，其大于15000mm 3，且当前深度截面(第131截面)对应的手部深度L小于或等于150mm时，则将前一深度截面(第130截面)所对应的手臂深度L作为目标手臂深度。
实现最小深度点N和手部顶点T的获取后，在本公开实施例中，将进一步基于所述最小深度点N与所述手部顶点T，确定指尖位置。
基于用户手势姿态的不同，基于深度图像所提取的最小深度点N和手部顶点T可能具有相同或不同的位置坐标。若在深度图像中，基于用户手势位置，所提取的最小深度点N和手部顶点T为同一像素点，则可直接将该像素点确定为指尖位置。例如当用户仅伸出单指进行触控操作时，其指尖可同时位于图像最小深度点和手部顶点。当所提取的最小深度点N和手部顶点T为不同像素点的情况下，将进一步地，基于预设规则对所述最小深度点和所述手部顶点进行处理，基于处理结果确定指尖位置。
所述预设规则可以根据实际需求进行选取，该规则例如可以为：通过对上述两点N、P自身坐标进行逻辑判别，从中选择表征指尖位置的点作为指尖，也可以通过将上述两点N、P间的距离或位置关系与系统预设阈值条件进行比较，进而从最小深度点N和手部顶点T两点中，确定表征指尖位置的点。本公开的实施例不受所述预设规则的具体内容的限制。
所述预设规则例如可以在图像处理之前预先设定，此时例如对于所有输入图像均采用相同的规则；或者其也可以在图像处理的过程中动态变化，例如其可以基于输入图像的特征而改变，使得对不同的输入图像采用不同的预 设规则。本公开的实施例不受所述预设规则的应用范围及所述预设规则的设定时间的限制。
图6示出了根据本公开的实施例提取的最小深度点N和手部顶点T为不同像素点的情况下，基于最小深度点N与手部顶点T确定指尖位置的示例性流程图。作为示例，下面将参考图6来说明当最小深度点N与手部顶点T为不同像素点时，基于已获取的最小深度点N和手部顶点T确定指尖位置600的操作。
如图6所示，基于已获取的最小深度点N和手部顶点T，首先参照步骤S601，得到最小深度点N和手部顶点T在深度图像纵轴上的坐标数值D T及D N，即为其像素点的纵轴坐标，该纵轴坐标可以是距离设定的原点的距离坐标也可以是距离设定的原点的像素点坐标；当该纵轴坐标为像素点坐标时，基于得到的像素点的纵轴坐标进一步应用转换公式“毫米单位＝像素点单位*ly*R(其中R＝像素点的深度值/相机焦距，ly为纵轴方向上，像素点与像素点之间的距离)”，将上述像素点单位下两像素点的纵轴坐标数据转换为毫米单位下两点在与深度方向垂直的平面上的投影点的纵轴坐标数据，其具体表示为：
Y T＝D T*ly*R T(其中R T＝手部顶点T的深度值/相机焦距)
Y N＝D N*ly*R N(其中R N＝最小深度点N的深度值/相机焦距)
基于此可计算出上述最小深度点N和手部顶点T在与深度方向垂直的平面上的投影点的纵轴方向上的纵轴距离ΔY，单位为毫米。其可具体表示为：
ΔY＝|Y T-Y N|
其后，根据步骤S602，获取最小深度点N和手部顶点T的深度值Z N及Z T，则进一步地，可计算出上述最小深度点N和手部顶点T在深度方向上的深度距离ΔZ，单位为毫米。其可具体表示为：
ΔZ＝|Z T-Z N|
应了解，步骤S601和S602的操作可以并行进行，或者按照顺序执行，在此不对其作出任何限制。
基于以上所取得的纵轴距离和深度距离，参照步骤S603，求取最小深度点N和手部顶点T的深度距离ΔZ和纵轴距离ΔY的差值Diff，其具体表示为：
Diff＝ΔZ-ΔY
并将上述差值Diff与距离阈值进行比较。在深度距离ΔZ与纵轴距离ΔY的差值Diff大于等于距离阈值的情况下，如步骤S6042所示，将最小深度点N确定为指尖位置；在深度距离ΔZ与纵轴距离ΔY的差值Diff小于距离阈值的情况下，如步骤S6041所示，将所述手部顶点T确定为指尖位置。
所述距离阈值为预设的距离数值，所述距离阈值可以基于实际需求进行设置，其例如可被设置为30mm或45mm。本公开的实施例不受所述距离阈值的具体数值及其设置方式的限制。
上述步骤可以更具体地描述，例如设置距离阈值为30mm，若深度距离ΔZ与纵轴距离ΔY的差值Diff大于等于30mm，则最小深度点N为指尖位置；反之，若在深度距离ΔZ与纵轴距离ΔY的差值Diff小于30mm，则手部顶点T为指尖位置。
在一些实施例中，所述指尖检测方法还包括将检测得到的指尖位置输出的步骤。例如可以将所述指尖位置输出至计算机处理模块以进行后续的处理，本公开的实施例不受将所述指尖位置输出的具体方式及其所输出至的具体设备的限制。
在一些实施例中，所述指尖检测方法还包括将所得到的指尖位置显示的步骤，例如将所述指尖位置显示在显示面板或控制屏幕上，以实现良好的人机交互。本公开的实施例不受显示该指尖位置的具体方式的限制。
图7示出了根据本公开的实施例的指尖检测装置的示意性的框图。
如图7所示，指尖检测装置700可以包括最小深度点检测模块710、手部顶点检测模块720、指尖位置确定模块730。其中，最小深度点检测模块710配置成基于深度图像，通过对于图像进行处理，确定其中待检测手部区域中的最小深度点N，其可执行如图3中所示出的流程，实现对于最小深度点N的确定。手部顶点检测模块720配置成基于深度图像，通过对于图像进行处理，确定待检测手部区域中的手部顶点T，其可执行如图4中所示出的流程，实现对于手部顶点T的确定。指尖位置确定模块730配置成基于所述最小深度点N与所述手部顶点T，确定指尖位置。其中对于指尖位置的确定，可依据最小深度点与手部顶点的坐标位置关系实现。在最小深度点与手部顶点为同一像素点时，其可将该像素点确定为指尖位置。若最小深度点与手部顶点为不同像素点时，则可执行如图6所示出的流程，确定指尖位置。
在一些实施例中，所述指尖检测装置还可以包括输出装置。所述输出装 置与外部计算设备相连接，其旨在将所得到的指尖位置输出至该计算设备以进行后续处理运算，其中所述计算设备例如可以为计算机或移动电话等，本公开的实施例不受所述计算设备的类型及所述输出装置与所述外部计算设备的连接方式的限制。
在一些实施例中，所述指尖检测装置还可以包括显示装置，所述显示装置可以将所得到的指尖位置进行显示，从而实现与用户的良好人机交互，该显示装置例如可以为显示面板或显示屏，本公开的实施例不受所述显示装置的具体类型的限制。
根据本公开的另一方面，还提供了一种计算机可读存储介质，其上存储有计算机可读的指令，当利用计算机执行所述指令时可以执行如前所述的方法。
技术中的程序部分可以被认为是以可执行的代码和/或相关数据的形式而存在的“产品”或“制品”，通过计算机可读的介质所参与或实现的。有形的、永久的储存介质可以包括任何计算机、处理器、或类似设备或相关的模块所用到的内存或存储器。例如，各种半导体存储器、磁带驱动器、磁盘驱动器或者类似任何能够为软件提供存储功能的设备。
所有软件或其中的一部分有时可能会通过网络进行通信，如互联网或其他通信网络。此类通信可以将软件从一个计算机设备或处理器加载到另一个。例如：从指尖检测设备的一个服务器或主机计算机加载至一个计算机环境的硬件平台，或其他实现系统的计算机环境，或与提供指尖检测所需要的信息相关的类似功能的系统。因此，另一种能够传递软件元素的介质也可以被用作局部设备之间的物理连接，例如光波、电波、电磁波等，通过电缆、光缆或者空气等实现传播。用来载波的物理介质如电缆、无线连接或光缆等类似设备，也可以被认为是承载软件的介质。在这里的用法除非限制了有形的“储存”介质，其他表示计算机或机器“可读介质”的术语都表示在处理器执行任何指令的过程中参与的介质。
同时，本公开使用了特定词语来描述本公开的实施例。如“一个实施例”、“一实施例”、和/或“一些实施例”意指与本公开至少一个实施例相关的某一特征、结构或特点。因此，应强调并注意的是，本说明书中在不同位置两次或多次提及的“一实施例”或“一个实施例”或“一替代性实施例”并不一定是指同一实施例。此外，本公开的一个或多个实施例中的某些特征、结构或特点可 以进行适当的组合。
此外，本领域技术人员可以理解，本公开的各方面可以通过若干具有可专利性的种类或情况进行说明和描述，包括任何新的和有用的工序、机器、产品或物质的组合，或对他们的任何新的和有用的改进。相应地，本公开的各个方面可以完全由硬件执行、可以完全由软件(包括固件、常驻软件、微码等)执行、也可以由硬件和软件组合执行。以上硬件或软件均可被称为“数据块”、“模块”、“引擎”、“单元”、“组件”或“系统”。此外，本公开的各方面可能表现为位于一个或多个计算机可读介质中的计算机产品，该产品包括计算机可读程序编码。
除非另有定义，这里使用的所有术语(包括技术和科学术语)具有与本公开所属领域的普通技术人员共同理解的相同含义。还应当理解，诸如在通常字典里定义的那些术语应当被解释为具有与它们在相关技术的上下文中的含义相一致的含义，而不应用理想化或极度形式化的意义来解释，除非这里明确地这样定义。
上面是对本公开的说明，而不应被认为是对其的限制。尽管描述了本公开的若干示例性实施例，但本领域技术人员将容易地理解，在不背离本公开的新颖教学和优点的前提下可以对示例性实施例进行许多修改。因此，所有这些修改都意图包含在权利要求书所限定的本公开范围内。应当理解，上面是对本公开的说明，而不应被认为是限于所公开的特定实施例，并且对所公开的实施例以及其他实施例的修改意图包含在所附权利要求书的范围内。本公开由权利要求书及其等效物限定。

Claims (16)

  1. 一种指尖检测方法，包括：
    基于深度图像确定待检测手部区域中的最小深度点；
    基于深度图像确定待检测手部区域中的手部顶点；
    基于所述最小深度点与所述手部顶点确定指尖位置。
  2. 如权利要求1中所述的指尖检测方法，其中，基于所述最小深度点与所述手部顶点确定指尖位置包括：
    在所述最小深度点和所述手部顶点为同一像素点的情况下，将该像素点确定为指尖位置；
    在所述最小深度点和所述手部顶点为不同像素点的情况下，基于预设规则对所述最小深度点和所述手部顶点进行处理，基于处理结果确定指尖位置。
  3. 如权利要求2所述的指尖检测方法，其中，在所述最小深度点和所述手部顶点为不同像素点的情况下，基于预设规则对所述最小深度点和所述手部顶点进行处理并基于处理结果确定指尖位置包括：
    确定深度图像中所述最小深度点和所述手部顶点之间的纵轴距离；
    确定深度图像中所述最小深度点和所述手部顶点之间的深度距离；
    基于深度距离和纵轴距离，确定指尖位置。
  4. 如权利要求3所述的指尖检测方法，其中，基于深度距离和纵轴距离确定指尖位置包括：
    在所述深度距离与纵轴距离的差值大于等于距离阈值的情况下，将所述最小深度点确定为指尖位置；
    在所述深度距离与纵轴距离的差值小于距离阈值的情况下，将所述手部顶点确定为指尖位置。
  5. 如权利要求1所述的指尖检测方法，其中，基于深度图像确定待检测手部区域中最小深度点包括：
    对于待检测手部区域中每一个像素点，将该像素点周围给定区域作为判定区域并将该像素点作为该判定区域的中心像素点；
    判断该判定区域内与该中心像素点的深度差值小于深度差阈值的像素点个数，若与中心像素点的深度差值小于深度差阈值的像素点个数大于个数阈值，则判断该中心像素点为稳定点；
    将所述深度图像中具有最小深度的稳定点确定为待检测手部区域的最小深度点。
  6. 如权利要求1所述的指尖检测方法，其中，基于深度图像确定待检测手部区域中手部顶点包括：
    对于待检测手部区域中每一个像素点，将该像素点周围给定区域作为判定区域并将该像素点作为该判定区域的中心像素点；
    判断该判定区域内与该中心像素点的深度差值小于深度差阈值的像素点个数，若与中心像素点的深度差值小于深度差阈值的像素点个数大于个数阈值，则判断该中心像素点为稳定点；
    将所述深度图像中具有最大高度的稳定点确定为待检测手部区域的手部顶点。
  7. 如权利要求1所述的指尖检测方法，其中，所述基于深度图像确定待检测手部区域中的手部顶点之前，还包括：
    提取前景手有效深度图；
    基于所述前景手有效深度图，确定待检测手部区域中的手部顶点。
  8. 如权利要求7所述的指尖检测方法，其中提取前景手有效深度图包括：
    基于深度图像，确定多个手部深度及与每个手部深度对应的手部体积；
    根据所确定多个手部深度和每个手部深度对应的手部体积，确定图像截取位置，获取前景手有效深度图。
  9. 如权利要求8所述的指尖检测方法，其中，确定多个手部深度及与每个手部深度对应的手部体积包括：
    以所述最小深度点为起点，按照预定深度增量，获取多个深度点处的深度截面；
    对于每个深度截面，确定该深度截面所对应的手部深度，并且确定该深度截面中的像素点个数；
    基于各个深度截面中的像素点个数，确定各个手部深度所对应的手部体积。
  10. 如权利要求8所述的指尖检测方法，其中，根据所确定的手部深度和手部体积确定图像截取位置并获取前景手有效深度图包括：
    在一深度截面所对应的手部体积大于体积阈值且其所对应的手部深度不大于深度阈值的情况下，将该深度截面的前一深度截面所对应的手部深度确 定为目标手臂深度；或者在一深度截面所对应的手部深度大于深度阈值且所计算的手部体积不大于体积阈值的情况下，将该深度截面的前一深度截面所对应的手部深度确定为目标手臂深度；
    按照该目标手臂深度进行图像截取，得到前景手有效深度图。
  11. 一种指尖检测装置，包括：
    最小深度点检测模块，配置成基于深度图像，确定待检测手部区域中的最小深度点；
    手部顶点检测模块，配置成基于深度图像，确定待检测手部区域中的手部顶点；
    指尖位置确定模块，配置成基于所述最小深度点与所述手部顶点，确定指尖位置。
  12. 如权利要求11所述的指尖检测装置，其中，指尖位置确定模块基于所述最小深度点与所述手部顶点，确定指尖位置包括：
    在所述最小深度点和所述手部顶点为同一像素点的情况下，将该像素点确定为指尖位置；
    在所述最小深度点和所述手部顶点为不同像素点的情况下，基于预设规则对所述最小深度点和所述手部顶点进行处理，基于处理结果确定指尖位置。
  13. 如权利要求12所述的指尖检测装置，其中，在所述最小深度点和所述手部顶点为不同像素点的情况下，指尖位置确定模块基于预设规则对所述最小深度点和所述手部顶点进行处理并基于处理结果确定指尖位置包括：
    确定深度图像中所述最小深度点和所述手部顶点之间的纵轴距离；
    确定深度图像中所述最小深度点和所述手部顶点之间的深度距离；
    基于深度距离和纵轴距离，确定指尖位置。
  14. 如权利要求13所述的指尖检测装置，其中，基于深度距离和纵轴距离确定指尖位置包括：
    在所述深度距离与纵轴距离的差值大于等于距离阈值的情况下，将所述最小深度点确定为指尖位置；
    在所述深度距离与纵轴距离的差值小于距离阈值的情况下，将所述手部顶点确定为指尖位置。
  15. 一种指尖检测设备，所述设备包括处理器和存储器，所述存储器包含一组指令，所述一组指令在由所述处理器执行时使所述指尖检测设备执行 权利要求1-10中任意一项所述的方法。
  16. 一种计算机可读存储介质，其中存储有计算机可读的指令，当利用计算机执行所述指令时执行权利要求1-10中任意一项所述的方法。
PCT/CN2019/094733 2018-09-03 2019-07-04 指尖检测方法、指尖检测装置、指尖检测设备及介质 WO2020048213A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19857813.0A EP3848778B1 (en) 2018-09-03 2019-07-04 Fingertip detection method, fingertip detection means, fingertip detection device, and medium
US16/647,765 US11315265B2 (en) 2018-09-03 2019-07-04 Fingertip detection method, fingertip detection device, and medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811021272.4 2018-09-03
CN201811021272.4A CN110874179B (zh) 2018-09-03 2018-09-03 指尖检测方法、指尖检测装置、指尖检测设备及介质

Publications (1)

Publication Number Publication Date
WO2020048213A1 true WO2020048213A1 (zh) 2020-03-12

Family

ID=69716776

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/094733 WO2020048213A1 (zh) 2018-09-03 2019-07-04 指尖检测方法、指尖检测装置、指尖检测设备及介质

Country Status (4)

Country Link
US (1) US11315265B2 (zh)
EP (1) EP3848778B1 (zh)
CN (1) CN110874179B (zh)
WO (1) WO2020048213A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220050432A1 (en) * 2020-08-14 2022-02-17 VTouch Co., Ltd. Method, system and non-transitory computer-readable recording medium for supporting object control

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050271279A1 (en) * 2004-05-14 2005-12-08 Honda Motor Co., Ltd. Sign based human-machine interaction
CN103226387A (zh) * 2013-04-07 2013-07-31 华南理工大学 基于Kinect的视频人手指尖定位方法
CN103294996A (zh) * 2013-05-09 2013-09-11 电子科技大学 一种3d手势识别方法

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102402680B (zh) * 2010-09-13 2014-07-30 株式会社理光 人机交互系统中手部、指示点定位方法和手势确定方法
EP2680228B1 (en) * 2012-06-25 2014-11-26 Softkinetic Software Improvements in or relating to three dimensional close interactions.
US8913037B1 (en) * 2012-10-09 2014-12-16 Rawles Llc Gesture recognition from depth and distortion analysis
US9524028B2 (en) * 2013-03-08 2016-12-20 Fastvdo Llc Visual language for human computer interfaces
WO2015037273A1 (ja) * 2013-09-12 2015-03-19 三菱電機株式会社 操作入力装置及び方法、並びにプログラム及び記録媒体
TWI499966B (zh) * 2013-10-08 2015-09-11 Univ Nat Taiwan Science Tech 互動式操作方法
CN104978012B (zh) * 2014-04-03 2018-03-16 华为技术有限公司 一种指向交互方法、装置及系统
US9501138B2 (en) * 2014-05-05 2016-11-22 Aquifi, Inc. Systems and methods for remapping three-dimensional gestures onto a finite-size two-dimensional surface
CN105096259B (zh) * 2014-05-09 2018-01-09 株式会社理光 深度图像的深度值恢复方法和系统
US20160078289A1 (en) * 2014-09-16 2016-03-17 Foundation for Research and Technology - Hellas (FORTH) (acting through its Institute of Computer Gesture Recognition Apparatuses, Methods and Systems for Human-Machine Interaction
JP2016091457A (ja) * 2014-11-10 2016-05-23 富士通株式会社 入力装置、指先位置検出方法及び指先位置検出用コンピュータプログラム
CN104881029B (zh) * 2015-05-15 2018-01-30 重庆邮电大学 基于一点ransac和fast算法的移动机器人导航方法
US9922244B2 (en) * 2015-09-03 2018-03-20 Gestigon Gmbh Fast and robust identification of extremities of an object within a scene
US10048765B2 (en) * 2015-09-25 2018-08-14 Apple Inc. Multi media computing or entertainment system for responding to user presence and activity
US10186081B2 (en) * 2015-12-29 2019-01-22 Microsoft Technology Licensing, Llc Tracking rigged smooth-surface models of articulated objects
CN105739702B (zh) * 2016-01-29 2019-01-22 电子科技大学 用于自然人机交互的多姿态指尖跟踪方法
EP3203412A1 (en) * 2016-02-05 2017-08-09 Delphi Technologies, Inc. System and method for detecting hand gestures in a 3d space
CN106502390B (zh) * 2016-10-08 2019-05-14 华南理工大学 一种基于动态3d手写数字识别的虚拟人交互系统及方法
WO2018157286A1 (zh) * 2017-02-28 2018-09-07 深圳市大疆创新科技有限公司 一种识别方法、设备以及可移动平台
CN107885327B (zh) * 2017-10-27 2020-11-13 长春理工大学 一种基于Kinect深度信息的指尖检测方法
CN109934065B (zh) * 2017-12-18 2021-11-09 虹软科技股份有限公司 一种用于手势识别的方法和装置
CN108257165B (zh) * 2018-01-03 2020-03-24 上海兴芯微电子科技有限公司 图像立体匹配方法、双目视觉设备
KR102455382B1 (ko) * 2018-03-02 2022-10-18 엘지전자 주식회사 이동단말기 및 그 제어 방법

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050271279A1 (en) * 2004-05-14 2005-12-08 Honda Motor Co., Ltd. Sign based human-machine interaction
CN103226387A (zh) * 2013-04-07 2013-07-31 华南理工大学 基于Kinect的视频人手指尖定位方法
CN103294996A (zh) * 2013-05-09 2013-09-11 电子科技大学 一种3d手势识别方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3848778A4

Also Published As

Publication number Publication date
EP3848778B1 (en) 2024-04-24
CN110874179B (zh) 2021-09-14
US11315265B2 (en) 2022-04-26
EP3848778A4 (en) 2022-05-18
EP3848778A1 (en) 2021-07-14
CN110874179A (zh) 2020-03-10
US20200286246A1 (en) 2020-09-10

Similar Documents

Publication Publication Date Title
KR101580914B1 (ko) 표시된 대상의 줌을 제어하는 전자 기기 및 방법
RU2662690C2 (ru) Устройство и способ управления объектом пользовательского прибора
WO2019233212A1 (zh) 文本识别方法、装置、移动终端以及存储介质
US9007321B2 (en) Method and apparatus for enlarging a display area
WO2015161653A1 (zh) 一种终端操作方法及终端设备
US20150026646A1 (en) User interface apparatus based on hand gesture and method providing the same
WO2019128639A1 (zh) 音频信号底鼓节拍点的检测方法以及终端
US9196055B2 (en) Method and apparatus for providing a mechanism for gesture recognition
WO2022222658A1 (zh) 一种凹槽深度测量方法、装置、系统及激光测量设备
KR102260805B1 (ko) 이미지 검색 장치 및 그 방법
CN111818050A (zh) 目标访问行为检测方法、系统、装置、设备及存储介质
EP2888716B1 (en) Target object angle determination using multiple cameras
WO2020048213A1 (zh) 指尖检测方法、指尖检测装置、指尖检测设备及介质
EP3214533B1 (en) Method and terminal for moving screen interface
CN115713616B (zh) 一种房源空间模型生成方法、装置、终端设备及存储介质
CN110852217B (zh) 人脸识别的方法及电子设备
KR102122793B1 (ko) 전자 디바이스 및 전자 디바이스에서 이미지 처리 방법
WO2019228106A1 (zh) 按压面积优化方法、装置、移动终端以及存储介质
WO2019042479A1 (zh) 基于陀螺仪的指纹操控方法、移动终端及存储介质
CN115329420B (zh) 一种标线生成方法、装置、终端设备及存储介质
US20230087081A1 (en) Method of detecting event, electronic device, and storage medium
CN114692247A (zh) 一种生成屋面板的方法、装置、存储介质及终端
CN116007500A (zh) 数据采集方法和装置、电子设备、可读存储介质和芯片
CN113986109A (zh) 一种对地图进行手势识别的方法和装置
CN104035628B (zh) 虚拟触控装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19857813

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019857813

Country of ref document: EP

Effective date: 20210406