TW561423B - Video-based image control system - Google Patents

Abstract

A method of using stereo vision to interface with a computer is provided. The method includes capturing a stereo image, and processing the stereo image to determine position information of an object in the stereo image. The object is controlled by a user. The method also includes communicating the position information to the computer to allow the user to interact with a computer application.

Description

561423 A7 B7 V. Description of the invention (1) ^ Related application comparison The patent application of this application is based on US Provisional Application No. 60 / 220,223 filed on July 24, 2000, and its title is VIDEO-BASED IMAGE CONTROL SYSTEM , And incorporated herein by reference. TECHNICAL FIELD The present invention relates to an image processing system, and more particularly relates to a video-based image control system for processing stereoscopic image data. -Background There are many operating systems that can be used to interact with or control a computer system. Most of these operating systems use standardized interfaces that use well-known graphical user interface (GUI) functions and control technologies. Since this function and control technology are common between GUIs, users who are not familiar with computer platforms and / or applications can still easily control different computer platforms and user applications. A common control technique is to use a mouse or trackball-type pointing device,-to move the cursor on the object of the camp screen. A gesture of clicking an object (once or twice) performs a GUI function. However, choosing a GUI function will prevent unfamiliar computer mouse operators from contacting the computer system. There are also situations where a computer mouse or trackball cannot be used, such as in front of a department store display window on a city street or the user is disabled. Summary In a general perspective, a method of using stereo vision to connect with a computer is revealed. The method includes capturing a stereo image and processing the stereo image to determine position information of an object in the stereo image. This object can be controlled by a user

k -4- The size of this paper applies to Chinese National Standard (CNS) A4 (210 X 297 mm) 561423 A7 B7 V. Description of Invention (2). This method also includes using this location information to allow the user to interact with a computer application. The step of capturing a stereo image includes capturing a stereo image using a stereo camera. The method also includes analyzing a change in position information of the object to identify a gesture related to the object, and controlling a computer application based on the recognized gesture. The method also includes determining an application state of a computer application and using the application state to recognize the gesture. This object can be a user. Name another-In the example, this object is part of the user. The method also includes providing feedback to a user about the computer application. In the above embodiment, the step of processing the stereo image to determine the position information of the object includes mapping the position information related to the position coordinates of the object to the screen coordinates of the computer application. The steps of processing the stereo image also include processing the stereo image to identify feature information and generate a scene description from the feature information. The step of processing the stereo image also includes analyzing the scene description to identify the location of an object and the mapping of the location change of the object. The step of processing the stereo image to generate a scene description also includes processing the stereo image to identify matching feature pairs in the stereo image, and calculating the inconsistency and position of each matching feature pair to generate a scene description. This method also includes analyzing the scene description in a scene analysis method to determine the position information of the object. Capturing the stereo image includes capturing a reference image of a reference camera and a comparison image of a two comparison camera, and processing the stereo image also includes processing the reference image and the comparison image to generate a feature pair. __ -5- This paper size applies to China National Standard (CNS) Α4 size (210 X 297 mm) 561423

Processing the stereo image to identify matching pairs in the stereo image also includes identifying the features in the reference image, and generating one for each feature in the reference image: comparing candidate matching features in the image, and from the candidate matching feature group. Each feature in the reference image selects a best-matching feature to generate a = sign pair. Processing this stereo image also includes the reference image and the comparison image. 0 Matching features to calculate a candidate matching feature to produce a matching score-processing this feature pair also includes for each candidate number and name "and select the student with the highest matching score. This feature pair. = Each feature in the reference image is generated—a group of candidate matching features includes selecting candidate matching features from the county decision area of the board image. Eliminating feature pairs based on the matching score of the candidate matching features. If the highest ranking :: match The matching score of the feature is lower than the pre-determined critical threshold, and it can also eliminate the feature pairs whose matching scores of the highest ranking candidate are in the matching score of the lower-ranking k-k matching features., 1 The step of calculating the number of PC matching knives includes identifying all Zheng Jin's feature pairs, adjusting the special score with different neighboring candidate matching feature matching score adjustments :: ', and selecting the candidate matching special desire with the highest adjusted matching score to generate Feature pairs. Images can be compared by using the comparison image as a reference image and a reference image. "^ Image to generate a second set of features, and 7 and brother have: and: intrinsic features of the original feature mapping of feature pairs in the group pair. Liwan ’includes the feature pairs in the description of this scene.

561423 A7

The real world of each stereo image is used to calculate the real world coordinates. Select the non-correspondence and position of the pair of images. Describes the lightness pattern of the reference pixels of the 4-volume image in the block of the image and comparison image. Stroke: A fixed-size pixel block included in a block divides the image into blocks. This pixel block is an 8x8 pixel area. Analyze the scene description to determine the object. The scene description excludes the desired area in the view and establishes the boundary of the desired area. The position information of the moon foot also includes trimming feature information outside this domain. Trimming includes_ Analyzing this scene description to determine the position information of this object includes, using a comparison with neighboring feature information within a predetermined range to compare areas of interest (feature information is grouped into groups with feature sets, and the Position. Analyzing this scene description also includes eliminating all groups with a threshold below the pre-determined threshold. Analyzing this image description also includes selecting the location of a group that matches a predetermined criterion, and remembering that the group that matches this predetermined criterion is green. Is the position of the object, and outputs the position of the object. This method also includes checking the characteristics of the detection area to determine the existence of users in this group. Calculating the position of each group can exclude the group outside the object detection area. Features. This method includes defining a dynamic object detection area based on the object position coordinates. In addition, the dynamic object detection area can be defined as a user's body. This method includes defining a body position detection area based on the object position coordinates. .Define this body position detection area also includes detecting this user's head and paper The scale applies to the Chinese National Standard (CNS) A4 specification (210 X 297 mm) A7 B7 561423 5. Description of the invention (5 position. This method also includes smoothing the movement of the position coordinates of the object to eliminate the jump between consecutive image frames. This The method includes calculating hand guidance information from the position coordinates of the object. Outputting the position coordinates of the object includes outputting the hand guidance information. Calculating the hand guidance information also includes smoothing the changes in the hand guidance information. Definitely the object detection area also includes Identify the position of the torso segmentation plane of the feature set and determine the position of the hand detection area on the torso segmentation plane perpendicular to the coordinate axis of the torso segmentation plane. Defining a dynamic object detection area includes identifying a central position of the body and a body boundary position from this feature set Use this feature to identify the intersection of the group and the torso segmentation plane from this feature set to identify a position indicating the user's arm part, and use the arm position related to this body position to identify this arm as the left or right arm. This method also includes Plane division by body center position, body boundary position, torso The winter left or right arm recognition establishes a shoulder position. Defining this dynamic detection area includes determining the position data of the hand detection area related to the shoulder position. This technology includes smoothing the position data of the hand detection area. In addition, this technology Including determining the position of the dynamic object detection area related to the torso segmentation plane perpendicular to the coordinate axis of the torso segmentation plane, determining the position of the dynamic object detection region at the horizontal coordinate axis related to the shoulder position, and using the body boundary position The vertical coordinate axis of the user ’s full scale determines the position of this dynamic object detection area. The steps to define this dynamic object detection area are as follows, unless the highest special -8-This paper size applies the Chinese National Standard (CNS) A4 specification (210X (297 mm)

Binding

561423 A7 B7 V. Description of the invention (6 If you want to be at the boundary, otherwise use the southernmost feature pair of the feature set to establish the user's head position and determine the position of the hand detection area related to the use of the head. This is disclosed in another point of view A method for communicating with a computer using stereo vision. The method includes capturing a stereo image using a stereo camera, and processing the stereo image to determine position information of an object in the stereo image, wherein the object is controlled by a user. This method It also includes processing the stereo image to identify the feature information, generate a scene description of this feature information, and identify the matching feature pairs in this three-body image. This method also includes calculating the inconsistencies and positions of each matching feature pair to generate this scene Description, and analysis of a scene description in a scene analysis method to determine the location information of the object. This method includes comparing feature information of a desired area into a group with a feature set by using a predetermined range of comparison with neighboring feature information. , Calculate the location of each group, and use this location information to allow The user interacts with a computer application. In addition, this technology includes mapping object positions from camera coordinates to screen coordinates and feature information related to this computer application, and using this mapped position to interact with this computer application. This method includes analyzing this scene description using Change the position information of the object in order to recognize the posture related to the object, and combine the position information and posture to interact with this computer application. The step of capturing the stereo image includes capturing the stereo image using a stereo camera. Another aspect discloses a A stereo video system for interacting with applications executed by the computer. The first and second video cameras are located near _ __9- This paper size applies to the Chinese National Standard (CNS) Α4 specification (210X297 public copy) 423 A7 B7 V. Description of the Invention (7) The structure can be operated to generate a series of stereo video images. A processor can be used to receive the stereo video images of this series and detect objects that are not within the range of the father of the camera field of view. The processor executes a program to define a three-dimensional relationship with the first and second video camera positions Target object detection area, select the control object shown in the object detection area, and when the control object moves into the object detection area, map the position coordinates of the control object to the position indicator related to this application This procedure · Sequential Selection-Select a control object as a detected object that appears closest to the video camera and within the object detection area. This control object is a human hand. A control related to the video camera The horizontal position of the object is mapped to the X-axis screen coordinate of this position indicator. A vertical position of the control object related to the video camera is mapped to the y-axis screen coordinate of this position indicator. Install this processor to map the video camera related Control the horizontal position of the object U The X-axis screen coordinate of this position indicator, map the vertical position of the control object related to this video camera to the y-axis screen coordinate of this position indicator and simulate a use provided to this application It has combined mouse functions of X Zhuoyou and y-axis screen coordinates. This processor is installed to simulate a mouse button using gestures derived from the position of an object. This processor is installed to simulate a mouse click for a predetermined time period based on the continuous position of the control object at any position within the object detection area. In another example, the processor is installed to simulate a mouse button for a predetermined period of time based on the position of a position indicator that holds a '' '' in a mutual display area. Install this processor to map -10- 561423 A7 ______ B7 V. Description of the invention (8) The z-axis depth position of the control object related to this video camera to the Z-axis screen coordinate of the indicator. Install this processor to map the X-axis position of the control object related to this video camera to the x-axis screen coordinates of this position indicator, and map the y-axis position of the control object related to this video camera to the y-axis of this position indicator Screen mount ", and map the z-axis depth position of the control object related to this video camera to the Z-axis screen coordinate of this position indicator. A position of the position indicator within the boundaries of the mutual display area triggers actions in the application. Movement of a control object along a z-axis depth position covering a predetermined / stomp distance within a predetermined time period triggers a selection action in this application. Any one of the object detection areas for a predetermined time period The continuous control of the position of the object triggers the selection action in this application. Another aspect discloses a stereo system for interacting with an application executed by a computer. The first and second video cameras are arranged in adjacent structures and Operable to generate a series of stereo video images. A processor is operable to receive this series of stereo video images And detecting an object shown in the X-cross range of the camera field of view. This process executes a program to define an object detection area at a three-dimensional coordinate relative to the first and second video camera positions, and selects a control object as an output Now it is closest to the video camera and the detected objects within the object detection and measurement area. Define the sub-area within this object detection area, identify the sub-area occupied by this control object, and when this control object occupies this area At this time, start an action related to this area, and apply this action to interact with a computer application. -11- This paper size applies the Chinese National Standard (CNS) A4 specification (210 X 297 mm) 5. Description of the invention (9 Redefine the action related to this area as the start simulation of the fish key. It is used for-predetermining the time period: ::: the relevant continuous control of the position of the object to trigger this action., Held by the long area = disclosed in another point of view -An interactive video system for interacting with applications executed by the computer. The first and second video cameras are equipped with near structure and can be operated to produce a series of stereo video images. A process can cry: It is used to receive the stereo video images of this series and detect the objects shown in this camera = foot fork and fan Ye. This process executes the program to identify the intersection and field of vision that are understood to appear in the camera's field of view. Located at-Predetermine the object of the depth range < the largest object, select this object as the target object, determine the position coordinate representing the desired position, and use this position coordinate as an object control point to control the application. This program also Causes the processor to determine and store a neutral control point position, maps the coordinates of the object control points related to the neutral control point position, and uses this mapping to control the coordinates of the body control point to control the application. This process also causes the processor An area having a position is defined according to the position of the neutral control point position, and an object control point related to its position is mapped within this area. The object control point coordinates of the map are used to control the application mode. This procedure also causes the processor to convert the mapped object control point to a velocity function, determine the perspective of the virtual environment related to the application, and use this velocity function to move this perspective within the virtual environment. This procedure causes the processor to shoot the coordinates of the control point of the object within the application to control a pointer position. The indicator in this implementation method is-avatar. -12-

561423 A7 B7

This procedure causes the processor to map the object within the application, controlling the coordinates of the control points to control the appearance of an indicator. The indicator in this implementation method is-avatar. This target is a human being in the field of vision. Another aspect discloses a stereo video system for interacting with applications executed by a computer. The first and second video cameras are disposed adjacent structures and are operable to generate a series of stereo video images. A processor is operable _ for receiving stereo video images of this series and detecting objects shown in the-'cross range of the camera's field of view. This process executes a program to identify an object that is understood to appear in the cross-field of view of the camera and the largest object in a predetermined depth range, selects this object as the target object, and defines the control area between the camera and the object of interest This control area is located at a predetermined position and has a predetermined size related to the size and position of the target object. Find the control area for the target object intelligent control area that is closest to this camera and within this control area. The point of this object of interest is within this control area_, select the point related to this object of interest as a control point, and move this control point into this control area, mapping the position coordinates of this control point to the relevant Position indicator in this application. The processor can be operated to map the horizontal position of the control object related to the video camera to the X-axis screen coordinate of the position indicator, and map the vertical position of the control object related to the video camera to the y-axis screen of the position indicator Coordinates, and simulate a mouse function that uses X- and y-axis screen coordinates. Another option is to operate this processor to map the photos related to this video. -13- This paper size applies to China National Standard (CNS) A4 (210 X 297 mm)

Binding

561423 A7 _____B7 V. Description of the invention (11) The X-axis position of the camera control object to the X-axis screen coordinate of this position indicator, mapping the y-axis position of the control object related to this video camera to the y-axis of this position indicator Screen coordinates, and map the z-axis depth position of the control object related to this video camera to the z-axis screen coordinates of this position indicator. The target in the carcass video system is a human appearing within a field of vision. In addition, this control point is related to the human hand that appears in this control area. > — Another aspect discloses a stereo video system for interacting with applications executed by a computer. The first and second video cameras are disposed adjacent structures and are operable to generate a series of stereo video images. A processor is operable to receive the stereo video images of the series and detect objects shown at the intersection of the camera's field of view. This process executes a program to define an object detection area at three-dimensional coordinates related to the positions of the first and second video cameras, selects a two-handed object that appears from a cross-field of view of the field of view within each object detection area, and When the hand object moves within the object detection area, the position coordinates of the hand object are mapped to the virtual hand position related to the avatar provided by this application. This program selects two-handed objects from objects that appear in the cross-field of view closest to the video camera and within the field of view of the object detection area. This incarnation takes on a format similar to the human body. In addition, this avatar is provided to form and interact with the virtual environment of this application. The processor executes a private sequence to compare the position of the virtual hand and the position of the virtual object related to the avatar within the virtual environment, so that the user can make this virtual within a virtual environment. -14- This paper size applies to China National Standard (CNS) A4 specification (210 X 297 mm) 561423 A7 B7 V. Description of invention (12) Intended object interaction. The processor also executes a procedure to detect the user's position coordinates within the intersection of the field of view and map the user's position coordinates to the virtual torso of the avatar provided by this application. If a mapped hand object is not selected, the program moves at least one virtual hand associated with the avatar to a neutral position. This processor also executes a program to detect the user's position and coordinates within the intersection of the field of view and coordinates ^ and map this user's position coordinate to the speed function applied to this avatar-enabling the avatar to roam the application Provided virtual environment. The speed function includes a neutral position representing the zero speed of the avatar. The processor also executes a program to map user position coordinates related to the neutral position to torso coordinates related to the avatar so that the avatar is tilted. The processor also executes a program to compare the position of the virtual hand related to the avatar with the position of the virtual object within the temporarily degraded virtual environment so that the user can interact with the virtual object while roaming in the ib virtual environment. As described in part of the interbody video system, this application obtains the virtual knee position associated with this avatar and uses it to improve the avatar scene. Another option is, 'This app gets the virtual elbow position associated with this avatar and uses it to improve the look of this avatar. The following figures and descriptions provide details of one or more methods of implementation. Other features and advantages are made easy by the description and drawings and the scope of patent application. Schematic description ___-_____ -1 5. This paper ruler Qi Cai @ ® 家 鲜) 61423 A7

Figure 1 depicts the hardware components and environment of one of the video control systems of the video horse king based on i-mesh Λ. Figure 2 is a summary picture! A flowchart of the processing technology used by the system. The diagram is related to the video-based image control system in Fig. I. ^ The field of view of each camera. Fig. 4 depicts a desired common point and upper polar line shown by a stereoscopic image of a stereoscopic camera r and a camera device generating a set of stereoscopic images. The current description in Fig. 5 is a stereo processing routine that generates scene description information from stereo images. The flowchart of Fig. 6 is a procedure for converting scene description information into position and orientation data. The graph of Fig. 7 describes the degree of reduction S with respect to a position such as the distance function D. Figure 8 describes the implementation of the image control system and system. "An object or hand detection area is set straight in front of the screen of a computer monitor. Figure 9 Flow: The process map describes the dynamic definition of hands related to a user's body Non-essential procedures for the detection area. Figure 1 0A-1 0C is an example depicting the procedure for dynamically defining the hand detection area related to the user's body. Figure 11A depicts a video-based image control system An example user interface and display area for contact. Figure 11B depicts a technique for mapping the position of a hand or pointer to a displayed TF area associated with the user interface of Figure 11A. Figure 12A depicts an example three-dimensional user in a virtual reality environment介 -1 6-This paper uses the Chinese National Standard (CNS) A4 specification (210X 297 ^ 17 561423) 5. The description of the invention (η) surface. Figure 12B depicts the view to eliminate the contents of the folder in the three-dimensional user interface of Figure 12A Figure 13A depicts an example representation of a three-dimensional user plane for manipulating a virtual three-dimensional space. 1 The graph in Figure 13B shows that the coordinate area in the image control system is dead. Domain without any necessary changes to the virtual position. Figure 14 depicts two examples of video game interface implementation methods, where actions and attitudes can be interpreted as joystick-type control functions for flying over a virtual three-dimensional cityscape. V Figure 15 The diagram of A describes an example head detection area divided into detection planes. Fig. 1 5B of the diagram describes an example head detection area divided into detection boxes. Figs. 15C and 45D describe one division into Two sets of directional detection box examples of head detection areas, and further describes a definition of the lack of α between adjacent directional detection boxes. 〇 The same reference symbols in different drawings indicate the same components. Detailed description Figure 1 describes a video-based Implementation method of the image control system 100. A person (or many characters) 101 places himself or puts his hand into a desired area 102. The desired area 102 is positioned in relation to the image detector 1 〇3 position, so as to be located within the overall visual field of view of the image sensor 104. The household area 102 includes a one-hand detection area 105, and if a dead line appears and is detected- 17 · This paper size applies the Chinese National Standard (CNS) A4 specification (210X 297 mm) ------ 561423 A7 B7 V. Description of the invention (15) For the body part, determine and measure the position and posture of the person The area, location, and measurement are described in a three-dimensional X, y, z coordinate or world coordinate, which does not need to be aligned with the image detector 103. The video generated by the image sensor ^ is used by the image series The metering device 107 for displaying video images on the video display 108 is processed, such as a personal computer. As described in more detail below, the computing device 107 processes the video image series so as to distinguish the position and attitude of an object, Such as the user ’s hand. The location and attitude information of the root cause is then projected to an application, such as a graphical user interface (GUI) or a video game. The position and posture of the user's hand is displayed on the video display 108 and allows execution and / or control of functions in the video game. An example function moves the cursor on the screen button and receives a light press on the gesture of the sister to select the screen button. The computing device 107 then performs the functions of the buttons. The video detector 103 is described in the following detailed description. The system 100 can be implemented in many configurations, such as a desktop computer configuration, and its image detector 103 is mounted on the surface of a video display 108 for viewing the desired area 102; or, for example, a two-head camera configuration Among them, the image detector 103 is erected on the supporting structure and positioned above the video display 10 8 of the desired area for viewing. FIG. 2 illustrates a video image analysis program 200, which can be implemented using computer software or computer hardware related to a typical implementation method of the system 100. The image detection ^ or the video camera 103 obtains the stereo image 201 and the surrounding scene of the desired area 102. Send this stereo image 201 to the computing device 107 (this device can be selectively referenced in the image detector 103), which is executed on the stereo image 201_ stereo analysis program 202 to generate a scene description 201 . Computing Device 戋 -18- This paper is suitable for Standard Paper ® ® Standard (CNS) A4 (21GX 297)

Binding

561423 A7 ________B7 V. Description of the invention (16) A different computing device uses this scene description 203 to calculate and output the hand / object position information of the person ’s hand 205 or other suitable pointing device and character in this scene description 203 Location or measurement of other features. The hand / object position information 205 is a set of three-dimensional coordinates and is provided to a position mapping program 207. This position mapping program 207 maps or converts the three-dimensional coordinates to a set of screen coordinates. The screen coordinates generated by the position mapping program 207 can be used as the screen coordinate position information used by the application program 208. The application program 208 is executed on the computing device 107 and provides user feedback 206. -It is also possible to detect specific gestures of the hand, that is to be detected as changes in the hand and / or other characteristic positions displayed by the hand / object position information 205, and interpreted by the posture analysis and detection program 209 as posture information or Posture 211. Then, the screen coordinate position information and attitude information 211 of the position mapping program 207 are transmitted together and used to control the application program 208. If the content of the gesture detection is sensitive, the gesture detection program 209 can use the application condition 210, and the criteria and meaning of the gesture are selected by the application program 208. Application · Condition 2 10 An example is to change the display of the cursor according to the position shown on the video screen 108. Therefore, if the user moves the cursor from one screen object to another screen object, the image representing the cursor will be changed from the pointer image to the hand image. Generally, the user will receive feedback 206 when the image shown in the video display 108 changes. Feedback 206-generally provided by the application 208 and related to the position and status of the application hand on the video display 108. In order to form a sub-set or a part of all or some objects of the scene, the image detector 103 and the computing device 107 generate scene description information 203, which includes a three-dimensional position or information contained in the three-dimensional position. If the position of the object is -19- This paper size applies to China National Standard (CNS) A4 (210X 297 mm) binding

Line 561423 A7

If the body has a shape that does not meet the expectations of the characters used in the system, or is detected by the stereo camera in the parallelizer 103, some limitations of the image detection environment are excluded. Also in this environment: add to the system operator. This is part of the image of the user that interacts with other search systems: the two systems are 'the image is still and / or imitated. In addition'-the user and the hand are slow: the A device is located ... the U device is also The limitation is increased because people and Γ〇〇: = three-dimensional form is used for cognition. User 1.1 operating system Temple even * has to wear gloves. Compared to other uses: Γ: Γ seeking the system 'The above description is also unique to the system ⑽ .... Because the appearance of the body and hands varies with different characters, the system ^ 00 is more dependent on the user and the hand The appearance method is more secure. However, please note that some implementation methods of the stereo analysis program 202 that can be used with the system 100 can also use the above expressions. Scene description information 203 is generally generated by a one-body camera. Image Detector 1 032: In this kind of system, two or more separate cameras are composed and used as the camera side. The camera can be a black and white video camera or a color video camera. Each independent camera requires a unique perspective image and produces a series of video images. Using the relevant position of a part of the scene of each camera image, the computing device 107 can calculate the distance between the object of the scene description 203 and the image detector 103. The following detailed description describes an implementation method of the stereo camera image detection rape 103 used by the system. There are other stereo camera systems and performances. -20- 561423 A7 B7 V. Description of the invention A different method can produce a scene description suitable for this system. It should be noted that the invention is not limited to the special stereo system used in this article. In FIG. 3, the cameras 30 1, 3 02 of the image detector or the stereo camera end 103 detect and generate scene images in the camera fields of view 304, 305 (respectively). The overall range of the field of view 104 is defined as the intersection of the respective ranges of all fields of view 304,305. An object 307 in the entire range of the field of view 104 may be detected in whole or in part by all the cameras 301 and 302. Because scene description 4: 203 allows f to include objects or object features outside the desired area 102, the object-body 307 need not be within the desired area 102. Regarding Fig. 3, it should be noted that the hand detection area 105 is a sub-collection area of the desired area 102. With regard to Fig. 4, images 401 and 402 of image pair 201 are detected by the camera pair ι03. The image 401 has a line group, and each line 403 has a mapping line 404 in the other image 402. In addition, any common point 405 located in the scene of line 403 can also be located on the mapping line 400 in the second camera image 402, so the point can be located in the entire range of the field of view 104 and can be seen in Cameras 301 and -302 (if not obstructed by other objects in the scene). The lines 403 and 404 are called upper polar lines. The difference in the position of the points of each upper polar line pair is called inconsistency. Inconsistency and distance are inverse constant, and provide the desired information to generate the scene description 203. The upper polar line pair depends on the camera image loss and geometric relationship between the cameras 301 and 302. These attributes are determined and selectively analyzed through prior processing of the classification. The system must account for the loss of radiation used by the lenses used by most cameras. A description of a technical feature of a camera for solving radiation loss

In Microsoft Research, titled A by Z. Zhang

In Flexible New Technique for Camera Calibration, -21- This paper size applies to Chinese National Standard (CNS) A4 (210X 297 mm) 561423 A7 B7 V. Description of the invention (19 http: " research.micros oft.com/~ zhang, which is incorporated herein by reference as the first step in classification. This technique does not find the epipolar line, but will result in a straight line, making it easier to find this line. The method of the sub-set is described in Z. Zhang Determining the Epipolar Geometry and its Uncertainty: A Review, The International Journal of

Computer Vision 1997 and Z. Zhang by Determining the Epipolar

Geometry and its Uncertainty: A Review, Technical Report 2: 2927, INRIA Sophia Antipolis, France, July 1996, all of which are incorporated herein by reference, and can be used to solve the upper polar line, this is the second step of classification . FIG. 5 illustrates an implementation method of a stereo analysis program 202 for generating a scene description 203. The image pair 201 includes a reference image 401 and a comparison image 402. The image filter 503 filters the individual images 401 and 402 and breaks into the features of the block 504. Each feature is an 8 × 8 image block. It should be noted, however, that features can be defined and processed as pixel blocks larger or smaller than 8x8. Matching process 505 finds a match for each feature in the reference image. To this end, the feature comparison process 506 compares the features in the reference image with all the features in a predefined range along the epipolar line mapped on the second or comparative image 402. In this special embodiment, the feature is defined as an 8 × 8 pixel block of the image 401 or 402, where it is expected that this block contains a part of the scene object, which represents the pixel intensity pattern within the block (because of the image filter 503 filter, so can not directly express π bright vision). The possibility of matching each feature pair is recorded by inconsistency and marked by two. If the best feature pair is less likely to be matched (compared to the critical threshold determined in advance) or many feature pairs are likely to have the best match (if there is a difference 561423 V. Description of the invention (20) The possibility is defined in advance Within a critical threshold, features are considered similar), the feature pair filter 507 eliminates blocks in the reference image 401. For the remaining reference features, the neighboring area support processing 508 adjusts the probability of all feature pairs by a proportional number to favor the possibility of neighboring reference features with the same inconsistent feature pair. For each reference feature, the feature pair selection program 509 can select the feature pair with the best possibility, providing inconsistency (and distance) for each reference feature. 'The reference feature T is generated by the program 504) The feature that is not shown in the second or comparison image 402 due to occlusion and the most likely matching feature shown will be wrong. Therefore, in the two camera systems, the features selected in the comparison image 40 will be checked by similar procedures (using the second parallel matching procedure 5 10, procedures 5 06, 5 07, 508, and 509) to generate a reference image. The best matching feature in 401 cancels the previous role of images 401 and 402. In a system with three cameras (ie using a third camera other than cameras 301 and 302), the second camera shadow volume replaces the comparison image 402, and the original reference image 401 continues to be used as the reference image. Through the same procedure (using the procedures 506, 507, 508, and 509 in the second parallel matching procedure 5 10), the best matching feature of the third video is determined. If there are more than three cameras available, this procedure can be repeated on other camera images. When the best matching pair of any reference feature has the same matching feature in the reference image, it is eliminated in the comparison process 5 丨 丨. Therefore, many erroneous matches eliminate the erroneous distance caused by occlusion. The result of the above procedure is the depth description map 512, which describes the positions and inconsistencies of the features related to the images 401 and 402. Coordinate system conversion program 5 i 3 Use the following Equation 1, Equation 2 and Equation 3 to change the position and inconsistency. ___ -23- This paper size applies to China National Standard (CNS) A4 (210 X 297 mm) 561423

Switch to the iterative three-dimensional world coordinate system (x, y, z coordinate system) (Figure κ 106). Since distance is not linear, it is difficult to work with inconsistencies. Therefore, the above three equations are generally used at this time, so that the coordinates of the scene description 203 can be described as a linear distance related to the world coordinate system 106. Reassigning the feature coordinates will affect the feature density in the area and make the clustering feature processing change. It is more difficult (implemented in a later step). ^ This usually maintains video-based and transformed coordinates. The conversion depth description mapping system generated by the conversion programmer 13 is the scene description 203 (in FIG. 2). The scene analysis program 204 makes this information meaningful and obtains valid information from it. In general, the scene analysis program 204 depends on the specific method used by the system. The flowchart of FIG. 6 summarizes the execution method of the scene analysis program 204. In the scene analysis program 204, the features in the scene description 203 are filtered by the feature exclusion module 601 to exclude features indicating that the features do not belong to the user or are located outside the desired area. Module 601 also eliminates background and other equivalent tractions (such as the user-there are others in the distance). The desired area 102 is generally defined as a bounding box aligned with the world coordinate system i 06. In this case, the module 601 can easily check whether each feature coordinate is located in the boundary E. Part of the background can be detected in the desired area 102, or the desired box-shaped area cannot distinguish the user from the background (especially when in a confined space). If there are no users in the desired area 102, the scene description two 203 may be randomly sampled and modified by the background sampling module 602 to generate a background reference 603. The background reference 603 is a description of the appearance of the scene, which does not change the appearance of the scene. -24- This paper size applies the Chinese National Standard (CNS) A4 specification (210 X 297 mm) 561423 A7 B7 V. Description of the invention (22 Changes (such as Changes in brightness). Therefore, the scene can be effectively sampled only when the system is established at 100. As long as the scene remains unchanged, the reference will remain valid. To ensure that the observed background remains within the shape defined by background reference 603 The background sampling module 602 can observe the scene description 203 for a short time and record the closest camera 103 for all positions. In addition, the distance defined by the feature is further extended by the predetermined distance (generally the distance meets the inconsistent pixels on the feature distance) k)). Once the sampling is completed, the background reference 603 can be compared with the scene description 203, and the street feature exclusion module 60 can remove the scene above or behind the background reference _ description any feature in 203. After eliminating the desire, the next step is to use the feature grouping program 604 to group the remaining features into one or more feature sets. Compare each feature in the previously defined range with Proximity features. Features are easier to distribute in their image coordinates than in other transformation coordinates. Therefore, image coordinates are usually used to measure close distances. The maximum receivable range is defined in advance and depends on the specific stereoscopic analysis process I 'used, such as Stereo analysis program 202. The above stereo analysis program 202 generates correlation density and average distribution features. This method is easier to group than other stereo processing technologies. Feature pairs that meet the standard can be considered as neighboring feature pairs. The range check depends on the inconsistent axial distance (the camera in the X axis is located before the desired area 'or the camera in the y axis is located in the desired area = if there is a path through the feature pair of the combined feature, the special on this path, If the standard is met, the group includes feature pairs that do not meet the standard. Continued Λ Real Λ method, using the group 遽 wave program 605 遽 wave group to ensure the group's :: be ... objects in the desired area 102, and Non-stereoscopic processing 歹 '仃 Character sequence at the wrongly defined position (or not) in the heart order. Part -25- 297 mm) This paper rule 561423 A7 _______B7_ 5 The group filtering program 605 of the invention description (23) also eliminates a group that contains too few features and provides confirmation measurements of size, appearance, or position. The area, boundary size, and number of features of the measurement group are compared with a previously defined description that describes the minimum quality of the measurement. The critical threshold is compared. Groups that do not meet the criteria and their characteristics will be deleted from other considerations. The representation detection module 606 in this implementation method determines whether a person appears. The indication detection module 606 is selective, because not all The system requires the component to provide this information. In its simplest way, the detection detection module two group 606 only needs to check the previously defined characteristics within the detection area 607 boundary. J (not previously eliminated) Whether it appears. It indicates that the detection area 607 is an area that may be occupied by any user 101. If there is no user, this area cannot be occupied by any object. It generally indicates that the detection area 607 corresponds to the desired area 102. However, in certain devices of this system, this means that the detection area 607 is defined to avoid moving objects in the scene. In the implementation method used in this component, if the user is found to be 101, it can be omitted and further processed. In the implementation method of the above-mentioned system 100, a hand detection area 105 is defined. Benefit _ The area 105 (through program 609) defined by this method depends on the structure used by the system described below. This sequence can selectively analyze the user's body and return other information including body position / measurement information 610, such as the position of the person's head. The hand detection area 105 need not have any content or need only include a human hand or a suitable pointer. Any group that has not been filtered out or has the characteristics of the hand detection area 105 can be regarded as or include a hand or a pointer. Two positions are calculated for each group (through program 611). If the position is within the hand detection area 105, it is recorded (in & memory) as the hand position coordinate 612. Measure this position as a weight. ____- 26- This paper size applies to China National Standard (CNS) Α4 (210 X 297 mm)

Binding

561423 A7 B7 Five invention description (24 positions. Identify the group feature (recognized by the example 1005 described in Figure 10) that is farthest from the entrance of the hand detection position area (1 002 in the example), and according to the top of the finger or pointer It is possible to assume that its position is given a weight of 1. The remaining group feature weights are based on the distance returned to its features, using the formula of Equation 4 below. If this application requires only one hand position and multiple groups have a hand detection area 105 Feature, providing the position farthest from the entrance 1002 as the hand position 612 and discarding other positions. Therefore, use the two that can reach the hand detection area 105 as far as: hand. Otherwise, if more than two have hand detection The group of features in the area 丨 〇5, the second longest distance and the second longest distance from the entrance 1 002 can be used as the hand position 6 12 and other locations are discarded. When the above rules cause the group to be included and used When replacing different groups, this included group is marked in the hand position data 6 丨 2. In the structure where the direction of the camera can detect the human arm, this direction is expressed as the direction of the hand of the arm or pointer The mark 613 "can be calculated by the 1t of the hand direction calculation module 614. If the height of the camera 103 is the same as the south of the hand detection area 105, including the structure of the camera 103 higher than the hand detection area 105. The standard coordinate axis of the group indicates this direction, which is calculated from the moment of the group. The following is a method that can also produce good results when the features are not evenly distributed. The arm can enter the position of the hand detection area 1 0, where The group is divided by the plane formed by the boundary of the hand detection area 105. The vector between this position and the hand position coordinate "^ provides the hand direction coordinate 613. Two dynamic smoothing programs 615 can be used for the hand position coordinate 612, and the hand direction coordinate 613 (if the (Solved) with any other body position or measurement 61. Smoothness is knot ____- 27- This paper size is suitable for the country's national standard (CNS) A4 specification (210X297 public love) 561423 A7 ____B7 V. Description of the invention (25) The procedure of combining the result and the previous solution to keep the attitude unchanged in the frame. One of the special coordinates 値 is specially smoothed, where each coordinate component X, y, Z is individually and dynamically smoothed. Calculated by Equation 5 below reduce The degree of S, which dynamically and automatically adjusts S according to the change of position. The distance critical 値 DA and Db shown in Figure 7 define three ranges of attitude. For position changes less than DA, the attitude will be in the area 701 by Sa. Ground reduction in order to reduce the tendency of backward and forward transitions between two adjacent borders (a marginal utility of image abstract sampling). Position changes greater than Ib are slightly trimmed-reduced by SB in area 702, or No reduction. This step reduces or eliminates the delays and ambiguities shown in other flattening procedures. For DA and DB actions, the degree of reduction is different. This area is 703, so it is less easy to notice slight and major Change between cuts. Equation 6 below is used to solve the constant a, which is used in Equation 7 (described below) to modify the coordinates. The result of the dynamic smoothing program 61 5 is the hand / object position information 205 in FIG. 2. When the program 611 mark and the previous position belong to different group power positions, since the current position is different from the previous position, the smoothing is not used. Step 609 uses the method to determine the hand detection area 105 according to the way the image control system 100 is used. Both approaches are described herein. The simplest hand measurement area 105 is a fixed area determined in advance, which does not contain any objects or only human hands or pointers. As shown in FIG. 8, the method used for this definition is the method of using the system 100 for controlling the user interface of a personal computer, in which the hand detection area 105 is the area before the computer display monitor and on the computer keyboard 802. In traditional computer use, the user's hand or other objects usually do not enter this area. Therefore, any detection in hand 2 8-This paper size applies to China National Standard (CNS) A4 (210 X 297 mm)-A7 B7

Show

Equipment 561423 V. Description of the invention (the moving foot object in the 26 area 105 can be interpreted as the result of the user's implementation of the use of hand gestures, and the pointer can be any object suitable for performing instructions, such as a person's posture. Pencil or other suitable indicator. 2. Should: The specific t-line method of B-body analysis program 2G2 can be used as a restriction of type or appearance. In addition, the above selection can be defined to indicate the detection area: for area 8G1 to Including the user's head in this way. The image detection cry 103 can be placed on the monitor 108. ° In some ways, the hand detection area 105 for one year can be dynamically defined in relation to the user — the body and does not contain any objects or Contains only people's hands or pointers. The use of dynamic areas removes the restriction of users at predetermined locations. The method of using this implementation method is illustrated. Figure 9 details the process of selecting the position of the dynamic hand detection area 609. Implementation method. In this program, the positions of the hand detection area and ι05 on each coordinate axis have been solved, and the size and direction of the hand detection area 105 are reset by a previously defined description. The example in Figure 10A_loc is used to describe this procedure. Using group data 901 (the output of the group filtering program 605 in Figure 6), the procedure described in block 902 includes finding the location of plane 1001 (this type of torso segmentation plane) (Depicted in the side field of view described in Figure 10c), its direction is parallel to the boundary 1002 of the detection area 105 and the user 101 can reach this position. If you want to distribute the features evenly on the original image ( As when implementing the method of the above-mentioned three-dimensional analysis program 202), most of the remaining features will belong to the user's torso rather than the user's hand. In this case, the platform 1001 can be positioned to divide the features into two equal numbers Group. If you don't want to distribute the features evenly (as when implementing the other methods of the three-dimensional analysis program 202 above), the above false-29 cannot be established. This paper size applies the Chinese National Standard (CNS) A4 specification (210X297 mm).

561423 V. Description of the invention (A7 B7 27), the characteristics of the outer boundary of the city group will still be expected to belong to the body. Under the k h-shaped positions, the platform 1 〇〇 丨 can be located The characteristics of ^! Are two groups with the same number. In another case, block 902 (the torso segmentation program locates platform 1001 to pass the user's torso. Sequence block 903 determines the hand detection on the coordinate axis. The position of the area 105 is generally set to the platform 1001 described above. The hand detection area 105 is defined as the second mouth 1001 and the predetermined distance 1004 is determined and positioned at the user's body. As shown by ^ in FIG. 1, The distance 1004 determines the 105-position of the hand detection area on the z-axis. The head of the right user is completely located in the desired area 102. The highest characteristic position of the expected group is the top of the user's head (and therefore also the user's height). Built in the program block 904 where the method is implemented. In program block 905, the position of the hand detection area 105 is located according to this head position. A predefined distance is located under the user's head. In the case of Figure 丨, the Determine the position of the hand detection area on the 4 axis. If the user's height cannot be measured or the group reaches the boundary of the desired area 102 (meaning that the character extends beyond the desired area 102), it will be positioned at a predefined height Hand detection area 丨 ... In many modes, it can be determined whether the user's left or right arm is connected to the hand detected by the position calculation block 611 in FIG. 6. In program block 906, it is determined that the arm is flat again. The location of the plane is defined before 1001. In general, the plane meets the boundary of the hand detection area indicated by 1002. If there are no features close to this plane, but some features are found in front of this plane, these features will block the plane The intersection point is assumed to be behind the blocking feature. Using the shortest proximity distance between block features, each intersection point can be contacted for hand binding

-30-

561423 A7

Point 0 can also find the position of the boundary between the middle of the user's body and the user's bone # in the program block 907. -In general, the position where features are uniformly distributed is = the main position with features. If you do not expect even-hook distribution, use another method of measuring the midpoint of the position between group boundaries. In program block 908, the position belonging to the arm found in program block 906 is compared with the body center position found in program block 907. If the position of the arm sufficiently offsets the left or right of the center position of the body, it indicates the hand; the source is the left or right shoulder of the user 101. If two hands are found but only one is identified as left or right, the other hand is also included. Therefore, the hand is marked as left- or right-handed according to the structure of the group to determine: proper markings in many forms where both hands can be found with the left-hand position to the right of the right-hand position.

If the hand is identified by program block 908, the position of hand detection area 105 can be located (from program block 109), so that the hand detection area 1 05 can be located in the desired range of action combined with the user's hand in. The position of the hand detection area 105 on the remaining coordinate axis can be biased toward the arm of the arm defined by Equation 8 (as described below). If the program block 908 cannot identify the arm, or if there is another method, the position of the hand detection area 105 on the remaining coordinate axis can be located at the center of the user's body found in the program block 907. In a form in which both hands must be tracked, the hand detection area 105 can be positioned at the center of the user's body. Program blocks 903, 906, and 909 each solve the position of the hand detection area 105 in an axis, and define this hand detection area} 〇5 position in three-dimensional space. A dynamic smoothing program 910 uses the same method used by component 615 __ _____- 31- paper & scale suitable financial SS home pure CNS Α4 specification (21GX 297 public Θ 561423 A7 B7 V. Description of the invention (29) method (utilization (Equation 5, Equation 6 and Equation 7) smooth the position. However, the higher layers that are subtracted are used in the program 91 0. The smooth position information output by the dynamic and smoothing programs 9 10, and the size and direction information defined in advance. 9 11 completely defines the boundary of the hand detection area 丨 05. In the process of solving the position of the hand detection area 105, program blocks 905, 907, and 908 find other methods for measuring the body position of various users 913 (Figure The procedure of 6 6 1 0) 〇 In short, the description of the characters in the scene is generated from the implementation method described in FIG. 6 using all the implementation methods containing the random components in FIG. 9 (as shown in the hand / object position information 205 in FIG. 2), where The information contained is as follows:-user presence / absence or quantity-for each user present: o left / right border of the body or torso o center point of the body or torso · if the head is in the desired area Inside) 〇 pair Appearing hand: Hand detection area mark left / right (if detectable) Fingertip position The direction of the hand or forearm is a known and improved method analyzed by scene description 203. The implementation method described in this article details this user (Such as identifying the position of the elbow). Hand / object position information 205 is a sub-collection or other information included in the above information, which allows users to interact with a variety of applications 208-32- (CNS) A4 size (210 X 297 public director_) 561423

And / or control application 208. The control methods of the three applications are explained in detail below. A variety of human poses can be detected by processing the above information. The above processing is not limited to the application 208 and the special control analysis described below. Examples of such gestures are untreated air orbits or rubbing hands to the other side. Generally, the type of posture detected by the posture analysis and detection program 209 uses hand / object position information 205.

Enables, detection technology to detect the state of the attitude. Detection, measurement procedures 209 maintain all hand- and body-position changes. One way to detect attitude is to check whether the position is clearly defined by rules. For example, if the following gesture detection rules are achieved, the gesture of wiping the other side can be identified: The horizontal position change of segment L less than a predefined time within a defined time is greater than a predefined distance. 2. The horizontal position changes consistently over time. 3 · # The change in the position of the vertex within day t is less than a predefined distance. 4. The position at the end of time is closer to (or just at) the boundary of the hand detection area than the position at the beginning of time. Some security states require that many rule qualifications be achieved in a clear order, so that this result can cause the system to change to a state qualified using different rules. This system cannot detect subtle gestures. In this case, the Hidden Markov module can be used, because this module still allows a series of specific actions to be detected, and also considers whether the operation of $ HAI fully meets the overall possibility of a gesture Sex. The implementation method of this system provides a method for user interaction. The user can use this method to cause the indication of an indicator to move within the image (user's anti-33- This paper size applies to the Chinese National Standard (CNS) A4 specification (210 X 297 mm) 561423 A7 _ —_ B7__ V. Description of the Invention (31) Feed 206), the above content is shown on the video display 108. This indicator moves in a way that reflects the posture of the user's hand. In a change in the form of a user interface, an indicator is displayed before other images, and its gesture is mapped to a two-dimensional space defined by the interface of the video display screen 108. This form of control is similar to the way a desktop computer typically uses a mouse. FIG. 11A depicts an example feedback image 206 using an application 208 of this type of control. ^ In the position mapping program 207, the hand position 205 detected by the scene analysis program 204 previously described 1 is mapped to the scene indicator or cursor 1101 through the method described below to cover the video display and position 108. The position on the screen image 206 is displayed. When a hand is detected and found to be within the hand detection area 105, the hand position 205 related to the hand detection area 105 is mapped to the information display 1 before the hand position 205 is transmitted to the application 208 〇8. One method of mapping coordinates is to obtain the X and y coordinates through the application of Equation 9 (described below). As shown in iJB, the entire display area 1102 is marked by a whole sub-area 1103 included in the hand detection area 1104 (approximately. In the hand detection area 105). The position in the sub-region 11 〇3 (such as the hand position 11 〇5) is linearly mapped to the position in the display region 1102 (such as 1106). A location (such as 1107) located outside the sub-region Π03 but still within the hand detection area 1104 is mapped to the closest location (such as 1108) on the boundary of the display area 1102. This reduces the possibility of the user accidentally removing his hand from the sub-region 1103 when he tries to move the cursor 110 1 to a position close to the border of the display. If both hands of the user are detected in the hand detection area 105, it is easy to select one hand in the position mapping program 207. In general, the hand that can reach as far as the hand detection area 105 will be selected. Because this hand tool -34- this paper is suitable for SS home standard (CNS) A4 specification (210 X 297 male I)-561423 A7 B7 V. Description of the invention (32) There are maximum or minimum X, y, z coordinates 値(Depending on the configuration of this system and the definition of the world coordinate system 106), it can be detected. Applications that use this interactive form often present data or control schemes (eg button 1 109). The user expects that the indicator 1 101 will be positioned on one of these objects. By comparing and remapping the pointer position 1 1 06 with the object pattern boundary (such as 1110), the above conditions can be detected. If the pointer position is within the object boundary, this condition is established. The user randomly receives feedback, which indicates that the ancestor is positioned on an object. Feedback can take many forms, including audio signals and / or graphical changes to groups and / or objects. Then use objects that can be activated, manipulated or moved under the group. The user is expected to indicate his purpose by activating, manipulating or moving the object through the performance of the gesture. In the system implementation method shown herein, the posture analysis program 209 recognizes the hand position or other position and measurement forms provided by the scene analysis program 204 and / or the position mapping program 207. Illustrate 'so that the shaker can indicate a purpose to activate an object located below the group, which will keep the group above the object for longer than a predefined period. This attitude detection needs the applied state 210, especially the boundary and / or state of the object to be fed back to the attitude analysis program 209. Since the existing technology can secretly monitor the application state 210 and use the coordinates provided by the position mapping program 207 to imitate other interface devices such as a computer mouse, there is no need to specifically create an application for the system. In some forms, this application status information 2 cannot be obtained and monitored. In this case, the attitude of the object below the activation group includes the attitude of keeping the hand. The posture is not changed or changed (beyond or extending the hand quickly forward or backward). Standards in this paper (Chinese Standard) (CNS) A Vision Grid X Tearing Love) Binding

k ^ 01423

Where = The detected excess method is the process of retaining the position of the hand, including the recording of all hand positions and the state of the latest definition at the end of the latest sampling. This time indicates the shortest time that the user must hold the same hand posture. Also: During this change, the smallest and largest positions within each three-dimensional coordinate (x, y, ζ) were found. If the hand appears in all the samples of the change process and the distance between the-and the position of the flaw is at the predefined threshold of each three-dimensional coordinate, 2, then the excess attitude is reported. The critical distance 値 indicates the hand is allowed to move, the number of flaws, and the maximum change (or the position where the itter expects to be induced by various components of the system). The general method of mimicking a mouse in return gesture will mimic the mouse connection. Simultaneously detect gestures that indicate the rest of the mouse's operations, double-click the boat, and mimic these operations. In addition, the position and position of the pointer position related to the object may be randomly detected ^ and the meaning given or related to the application state is given by the application. Applications that use this form of interaction generally do not explicitly use or display the user's hand or other location. !! Only in accordance with the position explanations made by this system, control these applications completely or in outline. Since the explanations made by this system can be used to imitate the gestures that traditional user output devices can perform, such as keyboards or joysticks, there is no need to create special applications for the system. Tera's useful explanation is directly relying on the absolute position of the hand within the 105 detection area. One way to generate an interpretation is to define a box, plane, or other shape. If the hand position is found in the first box and has not been placed in the immediate priority observation (because the hand position is within the hand detection area 105 or has not been detected), a state is activated. Keep this state until the hand position is not found in the second box (or far away from the boundary defined by the second plane), and it will be closed at this time -36- This paper size applies the Chinese National Standard (CNS) A4 specification (210 X 297 mm)

Binding

561423

This state. The second E must include the entire first £ and be larger in size. Using a larger E can reduce the situation of accidentally starting or closing the device when the hand position is close to the g boundary. In general, one of the interpretation methods used in this state depends on the intended purpose of the posture. In one method, the attitude directly reflects the state of the starting device on and off. When mimicking a keyboard key or joystick launch button, press the button when it is on, and release the button when it is off. In other general methods, only the transition from the closed state to the open state can be used to cause a posture. Although the time and the closed state are not reported to the application, the state is still maintained so that the posture will not be repeated until the state is closed. Therefore, each posture example requires a clearly defined purpose by the user. The third method generally used is to use attitude transitions from the closed to the open state to cause the attitude, and to re-initiate the attitude periodically at a pre-defined time interval of the same length as the state on time. Keeping keyboard keys down in this mimicry will cause features to repeat in some applications. A method of defining a cassette or plane for the above technique in the rabbit detection area 105 is described below. By defining the split hand detection area 105 as the first plane of the emission area 1053 and the neutral area 1504 (1501 in Figure 15) and the first plane 1502 (when the hand is in the area between the planes ι505) The gesture of return is dependent on the previous position of the above hand), so that the above technology can detect the hand's forward movement, which is to mimic the launch button on the joystick or cause the application to respond in a manner related to pressing the joystick (such as in the Fire weapons in video games). Another method of defining a cassette or plane for the above technique in the hand detection area 105 is described below. FIG. 15B depicts the first type of flat overlapped with corners and defined to divide the hand detection area 105 into left, right, top, and bottom 561423 A7 B7 V. Description of the invention (35) 15 06, 1507, 1508, 1509. The second type of plane is labeled 1510, 1 5 1 1, 1 5 1 2, 1 5 13. The first and second plane pairs are processed separately. This plane combination mimics the cursor keys in four directions, where the corner hand can activate two keys, and many applications are generally interpreted as four second 45 degree (diagonal) directions. Fig. 15C depicts another method for mimicking the abstract direction and application of an application ', which is expected to clearly represent the four 45 degree direction states. Define each of the four main (horizontal and vertical) boxes 1514, 1515, 1516, 1517 and define each second > 45-degree (diagonal) box 1518, 1519, 1520, 1 52 1. For clarity, only the first type of cassette is described. Locate a gap between g. FIG. 15D describes a method of defining a proximity bin. The gap between the first type II 1522 and 1 523 ensures that the user accidentally enters the box, and the gap 1524 is filled by the second type box 1 525 and 1526, so the system will report a gesture until the user clearly wants to To enter the adjacent box. This button combination can be used to simulate an eight-way joystick launch pad. Another type of stance depends on action rather than position, or on both action and position. This sad example is to rub your hand to the left. This gesture can be used to pass an application, which will return to the previous or previous state. By imitating the keyboard and mouse, this gesture generates the presentation software, especially pοwerp〇int, and moves to and from the sequence to show a slide. By imitating a keyboard and a mouse, this pose f generates a user interface for browsing the Web to perform security related to its aek press sister. With the same system, rubbing the hand to the right is a gesture that can be used to convey an application where the user must move to the next page or state seven times. For example, this pose causes the presentation software to move to the next slide of the presentation sequence and causes the browser software to move to the next page. " 11-3 8-This paper is suitable for g @ 家 料-561423 A7

" The following describes a method that is simpler than the previously described method for detecting the gesture of wiping the hand to the left, which uses the method of dividing the hand detection area 105 into areas divided by planes. The thin stripe on the leftmost part of the hand detection area i 〇5 is defined as the left area. The expression of the hand position is shown in the following three states: 1. The hand is present but the hand is not in the left area 2. The hand is present and the hand is in the left area 3. The hand does not appear in the hand detection area. The above transition from state 1 to state 2 results in The detection program 209 enters into a complaint 'and thereby starts a timer and waits for the next transition. If a state 3 transition is observed within a predetermined time, a gesture of wiping the hand to the left has already occurred. This technique generally repeats the right, upper, and lower edges, and because the hand position is found in 3D, it will also repeatedly detect the ulling the hand back. All the above gestures can be detected using the position of the hand or torso. In another variation of the system, the user causes one or two indicators to move within a representation (user feedback 206) of the three-dimensional virtual environment. Stereoscopic devices provide feedback and thereby allow each user to view the unique image of the illusion of depth created with the naked eye, even if this type of system cannot be practiced in many forms, so it can only be used randomly. However, it may also include the depth of objects that perform virtual environments using projection transformations. Figures 12A, 12B, and 13 A provide examples of how to perform this type of usage. Referring to FIG. 12A, the following describes a method that maps the hand position 205 in the position mapping program 207 detected by the scene analysis program 204 described above. -39- This paper size applies the Chinese National Standard (CNS) A4 specification (210X297). (Centi) 561423 A7 B7 V. Description of the invention (37 Shoot the position of the positioning indicator 1 2101 in the virtual environment. The hand position 205 related to the hand detection area 105 is converted by the position mapping program before being transferred to the application 208 207 is mapped to the coordinates related to the video display! 〇8. A method of mapping coordinates is obtained by applying Equation 9 to obtain the coordinates of the X coordinate and the y and z coordinates. This method is similar to the above method except that the third dimension is added. If the user has the ability to use the position of the pointer 1201 in all three dimensions, the user 101 can cause the pointer to contact an object (such as 1202) in a virtual environment like a real environment. This is a method for the user to interact with the virtual environment. Compare Represents the boundary of a cube or sphere pointer with an object (such as 1203 and 1204). The intersection of two types of boundaries indicates that the pointer is in contact with the object. If arranged The user may cause the pointer to move to a position where it contacts the object, where the pointer path avoids contact with any other object. Therefore, an ouch signal indicates the user's intention to activate, operate or move the object. Therefore, unlike the second The three-dimensional control of the “dimensional control” indicator 120 1 eliminates the need for a clear posture for one of the operations. Similarly, two-dimensional control of different bowls can arrange objects at different depths (such as the file box in Figure 12a) to provide An interface that is more similar to the familiar actions that users perform in the real world. In addition, the gestures that are not limited to pointers related to an object 丨 20j position can be randomly detected to indicate the intention to perform the action. Use Using this system, the user may navigate in a virtual environment. Compared to what once appeared in the user feedback 206, by allowing the user to cause the available selection of the objects or information shown, navigation allows the user to access many objects Or information. User 101 uses a random form of navigation to roam in a virtual environment 'and sub-collections of objects and a system of available things Depends on use in a virtual environment -40- This paper size applies to China National Standard (CNS) A4 (210X297 mm) 561423 A7

Πΐ touches Γ3, which means-example ', in which the user can roam in the virtual space to contact any set of objects that is not the storage space. Exam "ι4-A way for users to roam in _virtual environment = virtual environment to implement the virtual environment, by which: the camera's field of vision is within the scope of the camera, and is not represented by any virtual objects. -In a choice called impjn, the position of the camera represents the location of the user in the virtual environment. In another option, another indicator represents the location of the user in the virtual environment. This indicator can be regarded as a representative user The avatar of 010 (shown in video display 108). The position of the virtual camera that caused unscrupulous clasps, so that this indicator and all user objects related to the current user position can be located within the field of vision of the virtual camera. Use The position of the user ’s hand, body, or head all affects the virtual position of the user when roaming. One indicates that the position of the center or top of the user ’s torso was discovered by some implementation methods of the system, especially when it is fully implemented as shown in FIG. 9 In the implementation method of the random attitude analysis program 609. Using any of these positions allows the user 101 to perform the roaming gesture of the hand position independently, allowing use Touch virtual objects while roaming. Please note that these accessible objects can be fixed at the location related to the virtual environment or fixed to the position of the virtual camera, so that the user is available at any time. If there is no available location, use the user's hand position control Roaming. In this case, the system can automatically convert to contact content when the user has roamed close to an accessible virtual object or performed a predefined gesture. To provide an area that indicates no change to the virtual position, it is called Dead zone-41-This paper size applies to Chinese National Standard (CNS) A4 (210X297 mm)

Binding

561423 -42- A7-Description of the invention (domain 39, the virtual m γ π ~ spoon application of equation 1 0 (and similar equations passed to the y and z coordinates) is used to remap the position. This method leads to the problem described in FIG. 13B Relationship. Please note that the boundary and middle-earth position must conform to the hand detection area 105 and its center, or the area that has been dynamically adjusted to suit the user. When using the torso or head, the boundary and neutral position used in Equation 10 can be determined according to The following description adjusts to suit the user. First, the neutral position Xc'yc, zc used in Equation 10 i matches the neutral position of the user's body. With this system, all users will not be in absolutely the same position. After the user has given time to enter the desired area 102, sample the user's drive or head position and use it as a neutral position. Define the maximum range of action that the user expects to move freely (bite represents each centrifugal axis). To ensure that the user can still stay within the desired area 102 when moving to the end of the position, use the above posture (half of the maximum range's minimum range 'plus each x, y, and typical body groups in the z dimension The size of Ban Shao 'limits the boundary of the neutral position \ to the desired area 102: boundary-b, which is positioned at a position related to the neutral position, and the maximum range of half the action of the JL position in each distance. The above attitude is based on The position and / or action of the head or body torso. In this case T, use the area defined by these boundaries instead of the hand detection 105. The horizontal movement of the user (on the coordinate axis labeled χ in the example in Figure 丨) makes The visual field of the pseudo-environment moves to the left or right. The horizontal position of the transformation of Equation 10 is a function of the speed of the virtual vertical coordinate axis related rotation, causing the pointer or camera to deviate. It is also possible that the user's vertical posture makes the virtual view; private upward Or downward (on the coordinate axis labeled y in the example in Figure 1). Formula This paper size is suitable for towels® S Standard (CNS) A4 size (21GX297 male director) Binding line 561423 V. Description of invention A7 B7

10 The vertical position of the transformation can be directly interpreted as the tilt angle of rotation about the horizontal coordinate axis to locate the indicator and / or the camera. The displayed user 101 moves or results from the displayed user action (coordinates labeled z in the example in FIG. 1 fortunately from above) cause the virtual position to move forward or backward. One type of action is similar to walking, where indicators and / or cameras are held at a predefined height on the virtual floor and follow the contour of the floor (for example, going up a staircase). This translation position can be used as a velocity function of the vector, which is a speculation of the pointer and / or camera orientation to the plane defined by the floor. Another type of action is similar to a lie. If necessary, the transition position can be a function of the speed of a vector defined by the pointer and / or camera orientation. FIG. 14 illustrates an example of using the control false report method to navigate a virtual environment. This example uses the user's torso position found by the previously described method and using the mapping of Equation 10 and a suitable neutral position as previously described. Regardless of whether the method is used by the user to control or navigate the virtual environment, the pointer used in the virtual environment may take the form of an avatar. An avatar usually takes the form of a human body as shown in Figure 4 of 1401. The locations found by this system provide sufficient information to activate the virtual human form. When the user's hand is within the hand detection area 105, the system finds both hands of the user. Remap these locations: projected locations in front of the avatar's torso so that the hands of this avatar can reach the same location as the user touches. When the hand is not within the hand detection area 105, the user's hand will not be found or selected. In this case, the avatar's mapping virtual hand is neutral to the avatar's body. In the implementation of this method using roaming, it is found that it is related to neutral position -43- This paper size applies the Chinese National Standard (CNS) A4 specification (210X 297 mm)

Binding

k 561423 A7 B7 V. Control position of invention description (41). In these implementation methods, the feet of the avatar can be maintained in a fixed position and the relevant control position can be directly used to determine the position (posture) of the avatar torso on the fixed foot. FIG. 14 depicts an avatar controlled in this manner. In the practice method without using roaming, the position of the center of the user's torso or the position related to the top of the head can be used to directly determine the position of the torso of the avatar using the selection component 609. Details of other second bonding positions can be found through inverse mechanics techniques. In particular, the directional data 6 13 related to the arm can be used to limit the inverse mechanics interpretation to locate the elbow to the adjacent area, of which the forearm comes from the hand detection area 105. The orientation data 613 is limited to a flat elbow. The elbow position on the plane is defined as an arc cross with data segments representing the length of the avatar upper or lower arm, one focused on the position of the avatar hand (located in the virtual environment), and the other focused on the avatar torso related to the shoulder. The app also determines the knee position of the avatar. By positioning the feet of the avatar in a fixed position, it is urgent to ensure that the ankle of the avatar is not distorted. Secondly, fix the plane of the f-curved knee, and use the same cross calculation as the elbow to determine the knee position. In addition, using the fixed foot position, the avatar position can be calculated, where the avatar is tilted in the desired direction. Using the above calculations = the position of the avatar torso, hands, elbows, feet and knees sufficient to activate this avatar. Equation 1

D on X where I is the distance between the cameras and D is inconsistent

-44-

561423 A7 B7 V. Description of the invention (42) X is the image position X is the world coordinate position Equation 2 (d + dh -d0) 0 .. (sFI sin a) + Uycosa) i =-

D where I is the distance between the cameras D is inconsistent F is the average focal length s is the unit conversion factor applied to this focal length α is the tilt angle between the camera and the world coordinate z axis y is the image position Y is the world coordinate position Equation 3 7_ (sFIcosa) + (fysina) L ·-

D where I is the distance between the cameras < D is not consistent; F is the average focal length s is the unit conversion factor applied to this focal length; is the tilt angle between the camera and the world coordinate z axis; z is the image position; Z is the world Coordinate position equation 4 -dh) otherwise -45- This paper size applies to China National Standard (CNS) A4 (210X297 mm)

Binding

561423 A7 B7 V. Description of the invention (43) Where w is the weight and the quantity is from 0 to 1 d The distance from this feature to the hand detection area d is the distance from the furthest feature to the hand detection area dh is the expected size representing the hand The predetermined distance equation 5 S = ^ aSB + (1 -a) SA where a = d-da db —Da if (D < DA) if (DA < D < DB) if (D > DB) where D = | r (t) -s (tl) | s (t) is the smoothing at time t; r (t) is the unprocessed at time t; Da and Db are critical; SA and SB define the degree of reduction equation 6 α = | Where a is the boundary such that OSaSl where S is the subtraction found using Equation 8 e is the elapsed time since the previous sampling a is a constant Equation 7 where s (t) is a smoothing at time t 値 r (t) is at time t The untreated 値 a is a constant, where OSaS 1 This paper size applies to China National Standard (CNS) A4 (210 X 297 mm) 561423 A7

Equation 8 Λ: ^ βφι -bc) if ι6β _arm ^ β (br —bc) if right-arm ^ be if unknown where X is the position of the hand detection area bc is the position of the center of the body bl and br are the left and right of the body The position of the right border · 々 is a constant equation representing the number of positions of the hand detection area to the left or right. 9 夂 =

if xh < bt if where Xh is the hand position in this world coordinate system

Xc is the cursor position on this screen, mapping 0 to 1 b 1 and br are the equations for the position of the left and right borders of the sub-regions of this world coordinate system within the detection area of this hand 10-Xm χ / ι- (Xc -quad) b 丨 a) 〇

Xh ~ (Xc + ^ j) KX + Qua) Xm if ^ ^ bt 'z / 6 / < χΛ < (xc — ^-) lf (xc + ff) < xh < br if xh -47- This paper size applies the Chinese National Standard (CNS) A4 specification (210 x 297 mm) 561423 A7 B7 V. Description of the invention (45) where xv is the speed applied to this virtual coordinate system 乂 ^^ is applicable to this virtual coordinate The maximum speed of the system Xh is the position in this world coordinate system xc is the neutral position in this world coordinate system xd is the width of the "dead area π" in this world coordinate system b! And br are related to this world coordinate system here The positions of the left and right borders of the sub-areas within the hand detection area have already described some implementations. However, it will be understood that different modifications can be made. Therefore, other implementation cities are within the scope of the patent application below. -48- This paper size applies to China National Standard (CNS) A4 (210 X 297 mm)

Claims (1)

561423 A B c D 1. A method of communicating with a computer using stereo vision. The method includes: capturing a stereo image; processing the stereo image to determine position information of an object in the stereo image, the object being controlled by a user; and using the position information to allow the user to interact with a computer application. 2. The method according to item 1 of the patent application scope, wherein the step of capturing the stereoscopic image further comprises capturing the stereoscopic image using a stereo camera. 3. The method according to item 1 of the scope of patent application, further comprising analyzing a change in position information of the object to identify a posture related to the object, and controlling a computer application based on the identified posture. 4. The method of claim 3, further comprising: determining an application state of a computer application; and using the application state when identifying the gesture. 5. If the method of the first scope of the patent application, the object is the user. 6. If applying for the method of item 1 of the Lurie range, where the object is part of the user. -7 · The method according to item 1 of the patent application scope further includes providing feedback to users of the computer application. 8. The method according to item 1 of the scope of patent application, wherein processing the stereoscopic image to determine the position information of the object further includes mapping the position information related to the position coordinates of the object to the screen coordinates of the computer application. 9. The method according to item 1 of the patent application scope, wherein processing the stereoscopic image further includes processing the stereoscopic image to identify feature information and generating a scene description from the feature information. -49- This paper size applies to China National Standard (CNS) A4 (210 X 297 mm) 561423 Scope of applying for a patent 10. The method according to item 9 of the scope of applying for a patent, further comprising analyzing the scene description in the scene analysis process to determine the position information of the object. 11. The method according to item 9 of the scope of patent application, wherein processing the stereoscopic image further includes: analyzing the scene description to identify—the position change of the object; and mapping the position change of the object. 12. The method according to item 9 of the patent application, wherein processing the stereo image to generate the scene description further includes: processing the JL volume image to identify matching feature pairs in the stereo image; and calculating # for each matching feature pair. & y, li ,, female, J day 1 are inconsistent with the position to produce a scene description. 13. For example, the method of claim 12 of the patent scope, wherein: capturing the stereo image further includes capturing a reference image from a reference camera and a comparison image from a comparison camera; and processing the stereo image further includes processing This reference image is compared with the comparison image to generate a feature pair. 14. For example, the ancient, Tuguhebeiwan method of the scope of application for patent, wherein processing the stereo image to identify matching feature pairs in the stereo image further includes: identifying the features in the reference image; for each reference The features in the image generate a set of candidate matching features in the comparison image; and for each feature in the reference image, a feature pair is generated by selecting a best matching feature from the set of candidate matching features. -50 · 561423 A8 B8 C8 -------- _ D8 VI. Application scope of patent 15 · The method of item 13 of the scope of patent application, wherein processing the stereo image further includes filtering the reference image and comparing the image. 16. The method of claim 14 in the patent scope, wherein generating the feature pair further includes: calculating a matching score and ranking for each candidate matching feature; and selecting the candidate matching feature having the highest matching score to generate the feature pair. 17. The method according to item 14 of the scope of the patent application, wherein generating a set of candidate matching features for each feature in the reference image further includes selecting candidate matching features from a predetermined range in the comparison image. 18. The method according to item 6 of the patent application scope, wherein the feature pairs are eliminated according to the matching scores of the candidate matching features. 19. The method according to item 18 of the scope of patent application, wherein if the matching score of the highest ranking candidate matching feature is lower than the predetermined threshold, the feature pair is eliminated. A 20. The method as described in Application No. 18 of the Patent Scope, wherein the feature pair is eliminated if the matching score of the highest ranking candidate matching feature is within a predetermined threshold of the matching score of the lower ranking candidate matching feature. 21. The method of item 16 of the patent application, wherein the calculation of the matching score further includes: identifying those adjacent feature pairs; adjusting the feature pair matching score according to the proportion of matching scores of similar candidate features that are inconsistent nearby, and selecting The candidate matching feature with the highest adjusted matching score to create special -51- This paper size applies to China National Standard (CNS) A4 specifications (210 X 297 mm) 561423 A8 B8 Levy pair 22: If the scope of the patent application is No. 6 and the application comparison image is used as the reference image: and; generate a second set of feature pairs; and test ~ compare images like horses to eliminate those feature pairs of the original feature pairs. -、, and there is no phase in the competition pair 23. If the method of the scope of patent application No. 12, the further steps include ... Two each feature pair described in the t! Image, by: Each feature that runs through the world cliff is inconsistent with the inconsistency and position of the signature pair to calculate the true world coordinates. For example, the method of claim 14 in the scope of patent application, where the feature selection further includes the reference image of the interbody image and The comparison image is divided into blocks. 仏 For example, the method of applying for the scope of patent application No. 24, wherein this feature is described by the luminescence pattern of the pixels included in the block. The division further includes dividing this image into pixel blocks with a fixed size. 27. For the method of applying for item 26 of the patent scope, where the pixel block is an 8 × 8 pixel block. 28. If applying for a patent The method of scope item 10, wherein analyzing the scene description to determine the position information of the object further includes trimming the scene description to exclude feature information outside the area of interest in the horizon. 2 9. The method of the range of item 28, wherein trimming further includes establishing the boundary of the area of interest. 30. For the method of the range of application for the patent, the analysis of this scene description is based on -52- This paper scale applies Chinese national standards ( CNS) Α4 size (210 X 297 mm) A8 Βδ The determination of the position information of the object further includes the feature information in the region of interest within a predetermined range, and the position of each group is calculated. = The knife group is a group with a feature set by comparison with the near feature information; and 31. If the method of the 30th scope of the patent application, the group below a predetermined feature threshold 32. The method of item 30, selecting-selecting a group that matches a predetermined criterion further includes eliminating these positions that further include: records that match this predetermined target; and that the position of the group of quasi-groups is the object position to output the object position coordinate. It further includes checking the presence of the user. Among them, the position of each group can be calculated. 33. For example, the method in the 30th area of the patent application, the characteristics of the previous detection area determine the group. The characteristic of the group M application-the method of the patent scope item 32, further includes defining a dynamic object detection area according to the position coordinates of the object. 'Declares the method of patent scope item 35, wherein the dynamic object detection area can be defined as related to a user's body. 37. If the method according to item 32 of the scope of patent application is applied, the method further includes defining a body position detection area based on the position coordinates of the object. 38. The method of item 37 in the patent application of claim 4, wherein defining the body position detection area further includes detecting the head position of the user. 39. The method of claim 32 of the scope of patent application, further including smoothing the object 53- ^ 61423 The movement of the position coordinates to eliminate the jitter between consecutive image frames. 40. The method of claim 32, further comprising calculating hand-oriented information from the position coordinates of the object. 41 The method according to claim 40 of the patent scope, wherein outputting the position coordinates of the object further includes outputting the hand-guided information. 42. The method of claim 40, further comprising smoothing changes in the hand-guided information. 43. The method according to item 36 of the patent application, wherein defining the dynamic object detection area includes: identifying a position of a torso segmentation plane of a feature set; and determining a torso segmentation with respect to a coordinate axis perpendicular to the torso segmentation plane here Plane related hand detection area position. 44. The method according to item 43 of the patent application scope, further comprising: identifying a central position of the body and a boundary position of the body from the feature set;-using the feature to identify the intersection of the group and the torso split plane; identifying from the feature set-indicating the user The position of the arm portion; and using the position of the arm relative to the position of the body to identify whether the arm is a left arm or a right arm. 45. The method according to item 44 of the scope of patent application, further comprising identifying a shoulder position from the position of the center of the body, the position of the boundary of the body, the plane of the trunk division, and the left or right arm. 46. The method of claim 45, wherein defining the motion detection area includes determining position data of a hand detection area related to the shoulder position. -54- This paper size applies to China National Standard (CNS) A4 specification (21〇X 297 公 #) 561423 A8 B8 C8 The scope of patent application π is as described in the 46th item of the scope of application benefits, and further includes this The position information of the hand-monitored area is smooth. 48. The method according to item 45 of the scope of patent application, further comprising: determining the position of the dynamic object detection area related to the torso segmentation plane perpendicular to the coordinate axis of the torso segmentation plane; determining the dynamics on the horizontal coordinate axis related to the shoulder position The position of the object detection area; and the vertical coordinate axis with the overall height of the user who uses the body boundary position determines the position of the dynamic object detection area. 49. If the party in the 36th scope of the application for patents & defines & this dynamic object detection area includes the following steps: Unless the highest feature pair is at the boundary, the highest feature pair of the feature set is used to create the user Head position; and determine the position of the hand detection area associated with the use of the head. 50. — Seeds! Method for contacting stereo vision with a computer, the method includes: capturing a stereo image using two stereo cameras; processing the JL volume image to determine position information of an object in the stereo image, the object being controlled by a user; processing the interbody The image is used to identify feature information, to generate a scene description of the feature information, and to identify matching feature pairs in the stereo image; to calculate the inconsistencies and positions of each matching feature pair to create this scene description; to analyze the scene description in a scene analysis program to Determine the location information of this object; -55- This paper size applies Chinese National Standard (CNS) A4 specification (21〇X 297 public director) 561423 The feature information of the desired area is grouped into a group with a feature set by using a comparison with neighboring feature information in a predetermined range; and the position of each group is calculated; and the position information is used to allow the user to interact with a computer application. 51. The method as claimed in claim 50 of the patent scope, further comprising: mapping the position of the object from the camera coordinates to the screen coordinates of the feature information related to this computer application; and using this mapping: shooting position with this computer application Interaction ^ 5 2 · The method according to item 50 of the patent application scope, further comprising: analyzing the change of the position information of the object described in the scene to identify the posture related to the object; and interacting with the computer application by combining the position information and the posture . 53. The method of claim 50, wherein the step of capturing the stereoscopic image further comprises capturing the stereoscopic image using a stereo camera. 54 · —A stereo video system that interacts with an application program executed by a computer. The stereo video system includes: "" first and second video cameras configured in a neighboring structure and operable to generate a series of stereo video images; and A processor that can be used to receive a series of stereo video images and detect objects shown in the intersection of the camera's field of view. This process executes a program to define the three-dimensional mounts related to the first and second video camera positions > 椤Object detection area; v select the control object shown in the object detection area; and when this control object moves into this object detection area, map this 栌 Equipment -56- 561423 A8 B8 C8 —— ___D8____ VI. Patent application scope Control the position coordinates of the object to the position indicator related to this application. 5: > If the stereoscopic video system according to item 54 of the patent application scope, the program selects a control object as a detected object that appears closest to the video camera and within the object detection area. 56. The stereo video system according to item 54 of the patent application, wherein the control object is a human hand. 57. The stereo video system according to item 54 of the patent application, wherein the horizontal position of the control object related to the video camera is mapped to the X-axis screen coordinates of the position indicator. 58. A stereo video system as claimed in claim 54 in which the vertical position of the control object associated with the video camera is mapped to the y-axis screen coordinates of the position indicator. 59. The stereo video system according to item 54 of the patent application, wherein the processor is installed to:, map the horizontal position of the tantalum about the control object of the video camera to the X-axis screen coordinate of the position indicator; map related to this video The camera controls the vertical position of the object to the y-axis screen coordinate of this position indicator; and simulates a mouse function that uses the combined X-axis and y-axis screen coordinates provided to this application. 60. The stereo video system according to item 59 of the patent application scope, wherein the processor is further installed to simulate a mouse button using the posture obtained from the position of the object. 61. If you apply for a stereo video system in the 59th scope of the patent application, which is further installed -57- This paper size applies the Chinese National Standard (CNS) A4 specification (210 X 297 mm) 561423 A8 B8 C8 D8 6. Scope of Patent Application This processor is installed to simulate a mouse button that controls the continuous position of an object for a predetermined time period based on any position within the detection area of the object. 62. The stereo video system according to item 59 of the patent application, wherein the processor is further installed to simulate a mouse button for a predetermined time period based on the position of a position indicator that continues within the boundary of a mutual display area . 63. The stereo video system according to item 59 of the patent application, wherein the processor is further installed to map the z-axis depth position of the control object related to the video camera to the z-axis screen coordinate of the position indicator. 64. For example, the stereoscopic video system according to the scope of patent application No. 54, wherein the processor is further installed to: map the x-axis position of the control object related to the video camera to the x-axis screen coordinates of the position indicator,, mapping cabinet The y-axis position of the control object of this video camera to this position indicates the y-axis screen coordinate of II; and the z-axis depth position of the control object related to this video camera is mapped to the z-axis screen coordinate of this position indicator. 65. If the stereoscopic video system of the scope of patent application No. 64, the position of a position indicator within the boundary of the mutual display area triggers an action in the application. 66. If the stereoscopic video system of the scope of patent application No. 54, where A control object moving along a z-axis depth position covering a predetermined distance within a predetermined time period triggers a selection action in this application. -58- This paper size applies to Chinese National Standard (CNS) A4 (210 X 297 mm) 561423 A8 B8 C8 67. If the stereoscopic video of the 54th scope of the patent application, first determine any one of the object detection areas in the time period, and use it to pre-position the position of the object to trigger the selection control in this application. -An application program for computer execution, this stereo video system includes: a stereo video series of stereo video images arranged in adjacent structures and operable to produce a series of first and second video cameras; and Images and detections in this photograph of Riello, this process performs a one-to-one procedure to receive objects appearing in the cross-field of view of the stereo video camera series of this series: ^ x,…, the object detection area of the three-dimensional coordinates of the camera position; select-control the object as-the detected object that appears closest to the video camera and within the object detection area; what body detection is here Define a sub-area within the area; identify the sub-area occupied by this control object; when this control object occupies this area, a start action related to this area; And use this action to interact with a computer application. 69. For example, the stereoscopic video system with the scope of patent application No. 68, which further defines the action related to this area as the simulation of the activation of the keys of a computer keyboard. 70. Such as The stereoscopic video system with the scope of patent application No. 68, which is used to trigger the position of an object continuously in any sub-region in a predetermined time period. (Centi) 561423 AB c D 6. The scope of patent application. 71 ·-A stereo video system for interacting with applications running on a computer. This stereo video system includes: a nearby structure and operable to generate a series of First and second video cameras for stereo video images; and a processor that operates to receive this series of stereo video images and detect objects that appear in the cross-field of view of this camera field. This process executes a program to: Appears in the intersection of the camera's field of view and the largest object in a predetermined depth range An object; selecting this object as a target object; determining a position coordinate representing the desired position; and using this position coordinate as an object control point to control the application. 72_ If the system of the 71st aspect of the patent application, where This procedure results in this processor: — determining and storing a neutral control point position; mapping the coordinates of an object control point related to this neutral control point position; and using the mapped object control point coordinates to control the application. 73. If you apply The system of claim 72, wherein the program results in the processor: defining an area having a position based on the position of this neutral control point position; mapping an object control point related to its position within this area; and -60- This paper size applies to China National Standard (CNS) A4 specifications (210 X 297 mm) 561423 A8 B8 C8 ________ D8 VI. Patent application scope Use this mapped object control point coordinates to control this application. 74. The system of claim 72, wherein the program causes the processor to: convert the mapped object control point to a velocity function; determine the perspective of the virtual environment related to the application; and use the velocity function in Move this perspective within this virtual environment. 75. If the system of the scope of patent application No. 7 丨, where this program causes the processor to map the coordinates of the control point of the object within this application to control the position of a finger that is not consulted. 76. Such as the scope of patent application No. 75 The item system, wherein the indicator is an avatar. 77. The system of claim 71, which causes the processor to map the coordinates of the control point of the object within the application to control the appearance of a pointer. 78. If the system of item 77 of the scope of the present invention is applied, the indicator is a unitary 79. The system of item 71 of the scope of the patent application is applied, wherein the target object is a human appearing within the range of the field of vision. 80. A stereo video system for interacting with an application program executed on a computer, the stereo video system includes: a first and a second video camera configured in a neighboring structure and operable to generate a series of stereo video images; and A processor that operates to receive this series of stereoscopic video images and detect objects that appear in the intersection of the camera's field of view. This process executes a -61- This paper size applies to the Chinese National Standard (CNS) A4 specification (210 X 297 561423 AB c D 6. The scope of the patent application procedure is: Λ identifies the object that is understood to appear in the cross field of view of the camera and the largest object in a predetermined depth range; select this object as a target object; defined in The control area between the camera and the object of interest. The control area is located at a predetermined position and has a predetermined size related to the size and position of the target object. The control area of the target object within this control area; if the point related to this target object is controlled here Within the area, select the relevant point of the target object thereto a control point; and when this control within the control point is moved to this area, the mapping position coordinates to the control point to the position indicator associated thereto app. 81. The system of claim 80, wherein the processor is operable to:-map the horizontal position of the control object related to the video camera to the X-axis screen coordinates of the position indicator; map related to the video camera It controls the vertical position of the object to the y-axis screen coordinate of this position indicator; and simulates a mouse function that uses the X-axis and y-axis screen coordinates. 82. The system according to item 80 of the scope of patent application, wherein the process can be operated to: map the x-axis position of the control object related to the video camera to the x-axis screen coordinate of the position indicator; -62- Applicable to China National Standard (CNS) A4 specification (210X 297 mm) 561423 AB c D 6. The scope of patent application maps the y-axis position of the control object related to this video camera to the y-axis screen coordinate of this position indicator; and mapping Corresponds to the Z-axis screen coordinate of the Z-axis depth position of the control object of this video camera to this position indicator. 83. The system according to item 80 of the patent application scope, wherein the target object is a human appearing within a field of vision. 84. The system of claim 80, wherein the control point relates to a human hand appearing in the control area. 85 · —A stereo video system for interacting with an application program executed on a computer, the stereo video system includes: a first and a second video camera configured in a neighboring structure and operable to generate a series of stereo video images; and A processor that operates to receive this series of stereoscopic video images and detect objects that appear in the cross-field of view of this camera. A processor executes a program to define 3D coordinates in 2 relative to the first and second video camera positions. Object detection area; select up to two hand objects from objects that appear at the intersection of the field of view in this object detection area and appear in the range; and map this hand object when this hand object moves within this object detection area Position coordinates to the position of the virtual hand associated with the avatar provided by this application. 86 · If the system of the scope of patent application is No. 85, the program selects at most two from the closest to the video camera and looks in the object detection area -63- This paper size applies to China National Standard (CNS) A4 specifications (210X297 mm) " " '' 1 ~ 561423 The hand of the object that appears at the turn of the wild. 8 7. The system according to item 85 of the scope of the patent application, in which the avatar is obtained-similar to the format of a human body. 88. The system of claim 85, wherein the avatar is provided in and interacts with the virtual environment that forms part of the application. 89. The system of claim 88, wherein the processor further executes a program to compare the position of the virtual hand and the position of the virtual object related to the avatar within the virtual environment so that the user can Interact with this virtual object. 90. The system of claim 85, wherein the processor further executes a program to: detect the position coordinates of the user within the intersection of the field of view; and map the position coordinates of the user to the location of the application. The virtual torso of the avatar provided. 91. If the system of item 85 of the Quarry range is applied, if a phase-mapped hand object is not selected, this program moves at least one virtual hand associated with this avatar to at least one neutral hand. 92. The system according to item 85 of the patent application range, wherein the processor further executes a program to: detect the position coordinates of the user within the crossing range of the field of view; and map the position coordinates of the user to the speed applied to the avatar Function to enable this avatar to roam in the virtual environment provided by this application. 93. If the system of item 92 of the scope of patent application is applied, in which the speed function includes the following: -64- This paper size applies to China National Standard (CNS) Α4 specification (210X 297 mm) Binding 6 5 423 ABCD represents the neutral position of this avatar at zero speed. 94. The system of claim 93, wherein the processor further executes a program to map the user's position coordinates related to the neutral position to the torso coordinates related to the avatar so that the avatar appears tilted. 95. The system of claim 92, in which the processor further executes a program to compare the position of the virtual hand related to the avatar with the position of the virtual object temporarily in the virtual environment so that the user can roam in This virtual environment is constantly interacting with this virtual object. 96. The system of claim 85, in which the virtual knee position related to this avatar is obtained by this application and used to improve the avatar scene. · 97. The system according to item 85 of the patent application, wherein the virtual elbow position associated with this avatar is obtained by this application and used to improve the image of this avatar. 98. The system according to item 85 of the scope of patent application, further comprising a third camera disposed adjacent to the first and second video cameras and operable to generate a series of stereoscopic video images. / -65- This paper size applies to China National Standard (CNS) A4 (210 X 297 mm)