KR20100016240A - An apparatus system and method for human-machine-interface - Google Patents

Abstract

There is provided a 3D human machine interface (''3D HMI''), which 3D HMI may include (1) an image acquisition assembly, (2) an initializing module, (3) an image segmentation module, (4) a segmented data processing module, (5) a scoring module, (6) a projection module, (7) a fitting module,(8) a scoring and error detection module, (9) a recovery module, (10) a three dimensional correlation module, (11) a three dimensional skeleton prediction module, (12) an output module and a (13) depth extraction module.

Description

Device system and method for the human machine interface {AN APPARATUS SYSTEM AND METHOD FOR HUMAN-MACHINE-INTERFACE}

This application is filed under US 60 / 592,136, filed on July 30, 2004, PCT / IL2005 / 000813 filed on July 31, 2005, US 60 / 731,274, filed on October 31, 2005. Part of the PCT application of US 11 / 227,578, filed March 27 and PCT / IL2006 / 001254, filed October 31, 2006, whereby each of them is referred to in its entirety.

The present invention relates to a user interface, and more particularly, to a method and system of a 3D human machine interface.

One of the biggest patterns in the history of software is the shift from computation-intensive design to presentation-intensive design. As machines become more powerful, developers have been spending some of this steadily increasing power on presentations. The history of progress can simply be broken down into three eras: batch (1945-1968), command-line (1969-1983), graphical (graphical, 1984 and later). Of course, the history begins with the invention of the digital computer. The start dates of the next two eras are the year when vivid new interface technology has left the lab and changed the user's expectations of the interface in a significant direction. These technologies were interactive timesharing and graphical user interfaces.

In the batch era, computing power was extremely low and expensive. The largest computers of that time handled fewer logic cycles per second than today's typical toasters or microwave ovens, and fewer bits than today's cars, digital clocks or cell phones. Thus, the user interface was primitive. The user interface was considered overhead, and the software had to accept that it was designed to maintain the best use of the processor with the least load possible.

The input side of the user interface of batch machines was mainly media such as punched cards or paper tapes. The output side added a line printer to this media. Except for the limited exception of the console of the system operator, humans did not communicate at all with the batch machine in real time.

Delegating work to a batch machine involves first preparing a deck of punch cards describing the program and data set. The perforation of the program card was made not by the computer itself, but by a machine such as a typewriter, which is too stubborn and does not tolerate mistakes and is prone to mechanical mistakes. Similarly, mistakes are not tolerated by the very strict syntax, which means that the software interface is also analyzed by the smallest possible compiler and interpreter.

Once the card is punched, drop the card into the job queue and wait. Eventually, the operator presents the deck to a computer, perhaps mounting a magnetic tape to provide another data set or helper software. This operation results in a printout that includes a final result or an abort notification with an error log attached (often). A successful run can then write the result to a magnetic tape or generate any data card that can be used later in the operation.

The total processing time for a job often takes a whole day. Very lucky if any one took several hours; Real-time response is unprecedented. However, in a situation that is worse than a card queue, some computers actually require a more verbose and error-toggling process in binary code programs using console switches. Indeed, the earliest machines had to be rewired into partially merged program logic using a device known as plugboards.

Early batch systems provided current running jobs throughout the computer; Program decks and tapes had to include what we think of in the operating-system code and what housekeeping needed to communicate with the I / O device. In the middle of the batch period above, after 1957, various groups began experimenting with so-called "load-and-go" systems. These groups used a monitor program that always resided on a computer. Programs can be called monitors for services. Other functions of the monitor have been to perform error checking of the provided tasks well, catch errors faster and more intelligently, and produce more useful feedback to users. Thus, the monitors represent the first stage for both the operating system and the clearly designed user interface.

Command-line interfaces (CLIs) have evolved from batch monitors connected to the system console. Their conversation module is a series of request-response transactions and has a request expressed in technical terms as textual commands. Latency is several days or hours down to a few seconds lower than a batch system. Thus, the command-line system has changed the mindset of users for the next phase of the transaction in response to real-time or near real-time feedback in initial results. The software could be a two way worth exploring, a direction that was not possible before. However, these interfaces still forced the user to have good memory and required a lot of effort and learning time to be mastered.

The command-line interface is closely related to the emergence of timesharing computers. The concept of timesharing dates back to the 1950s; The most influential early experiment was the MULTICS operating system since 1965; And the most influential of the current command-line interface was the Unix of 1969, which had a developmental impact on most of the later.

Early command-line systems integrated teletypes and computers, making them effective in mediating the transfer of information transfers between people. Teletypes were initially developed as devices for automatic telegram transmission and reception; The teletype dates back to 1902 and has already settled in the editing room and elsewhere in 1920. While economics are certainly considered for the reuse of this teletype, psychological state and the Rule of Least Surprise are also important; Teletype provided interface points with the system to make it familiar to many engineers and users.

The widespread deployment of video-display terminals (VDTs) in the mid-1970s marked the second phase of the command-line system. In addition, because the characters could be phosphor dots on the screen that can move faster than the print head or carriage, the latency is reduced. And by eliminating consumables such as ink and paper to eliminate costly photographs, video-display terminals have helped to curb interfering with interactive programming and are more iconic than teletype, a pioneer of computer in the 1940s. iconic) and convenient, making it the first TV generation in the late 1950s and 60s.

As important as the presence of an accessible screen, a two-dimensional display of text that can be quickly and reversibly transformed has become economical for software designers to place interfaces that can be depicted more visually than textual. Pioneering applications of this kind were computer games and text editors; Sources of some of the early samples, such as rogue (6) and vi (1), still part of the Unix tradition.

As far back as 1961, the screen video displays that appeared in minicomputers, formerly PDP-1s, were not entirely new. However, until the move to VDTs was settled through serial cables, each very expensive computer could only support one addressable display on its console. In this situation any trajectory of the visual UI was difficult to grow; This interface was created once in a rare environment where the entire computer was at least temporarily used to satisfy only one user.

Back in 1962, attempts were made sporadically to be called graphical user interfaces, such as pioneering the SPACEWAR game on the PDP-1. The display of the machine was nothing more than a character terminal, but a modified oscilloscope designed to support vector graphics. The Space War interface also used toggle switches primarily, but was the first raw trackballs custom-made by the players themselves. Ten years later, in the early 1970s, these attempts gave birth to the video-game industry, and attempts to introduce an arcade version of Space War actually began.

The PDP-1 console display is derived from radar display tubes of World War II, and some 20 years earlier, several important pioneerings of minicomputing at MIT's Lincoln Labs were radar technology. It suggests that you are ahead of it. In the same year in 1962, on the continent, other radar engineers of earlier periods were beginning to pioneer other areas at the Stanford Research Institute. His name was Doug Engelbart. He was deeply inspired by his personal experience with the early graphic displays and the original essay by Vannevar Bush's, As We May Think, represented in the 1945 version of what is now called hypertext. .

In December 1968, Jensenbart and his team SRI demonstrated the first hypertext system, NLS / Augment ^9, for 90 minutes. In this demonstration, a three-button mouse (the invention of Angelbart), a graphical display with a multiple-window interface, a conference by hyperlinks and on-screen video were first introduced. This demonstration was a significant milestone throughout the 25 years of computer history and, in 1991, included the invention of the World Wide Web.

Thus, in the early 1960's, it was already accepted that graphical presentation would be helpful for an inevitable user experience. Pointing devices such as mice have already been developed, and many of the mainframes of the late 1960s were superior to those of the PDP-1. One of the programmers, in 1968, had a vivid memory of playing another fairly early video game on the console of the Univac 1108 mainframe, which could cost $ 45 million if it could be bought in 2004. However, since it was close to $ 45 million, the realistic demand for interactive graphics was minimal. Although custom hardware in NLS / Augment systems is a bit less expensive, it's still expensive for the average user. Even with a $ 100,000 PDP1, it's very expensive for a machine to make trajectories for graphical programming.

Video games have become popular before computers because hardware programs can run on extremely cheap and simple processors. However, after the advent of general purpose computers, oscilloscope displays have reached the end of their development. The concept of using a graphical and visual interface for normal interaction with a computer had to have been around for years, and in fact led to an improved graphics-enabled version of the serial-line character VDT in the late 1970s. .

Since the earliest PARC system in the 1970s, the design of graphical user interfaces (GUIs) was dominated by what was called the WIMP (Windows, Icons, Mice, Pointer) model pioneered by Alto. Over the next decade, there have been major changes in computing and display hardware that seemed very difficult to think beyond WIMP.

There have been several attempts. Perhaps the most obvious of these is the virtual reality (VR) interface, which allows users to move and gesture in an immersive graphical 3-D environment. VR has fascinated large research groups since the mid-1980s. Although the computing power to support these is not expensive, the physical display device was still the price of VR before it became universal in 2004. The more fundamental problem familiar to designers of flight simulators for many years is that VR can confuse human proprioceptive systems; VR motion can cause brain fatigue, dizziness, or nausea, because the inner ear reports from real-world movements to match the visual simulation of movements.

Jeff Raskin's The Humane Environment project is studying the zoom world model of GUIs that spatialize without going in 3D. In THI, screens become windows in a 2-D virtual world where data and programs are organized into spatial addresses. In this virtual world, objects can be represented at various levels depending on one height in the reference plane, and the best selection action is to zoom in and reach over it.

Yale's lifestreams project went in the opposite direction, and it's not really spatializing the GUI. The user's document is represented by several kinds of world-line or temporary streams that can be organized by change date and filtered in various ways.

All three approaches give up traditional filesystems on the part of the context of trying to avoid naming and using names as the main form of reference. This makes it difficult to match the file system with the hierarchical namespace of Unix structures, which is seen as one of the most permanent and valid features. However, one of these early attempts still appears to be as important as the 1968 demonstration by NLS / Augment Angel Bart.

There is a need in the field of user interfaces for improved systems and methods of human-machine-interfaces.

According to some embodiments of the invention, a 3D human machine interface (“3D HMI”) is provided, wherein the 3D HDI comprises (1) image acquisition assembly, (2) initialization module, (3) image partition module, (4) compartment Data processing module, (5) scoring module, (6) projection module, (7) fitting module, (8) scoring and error detection module, (9) recovery module, (10) three-dimensional correlation Module, (11) three-dimensional skeletal prediction module, and (12) output module.

According to some embodiments of the invention, the image acquisition assembly may acquire a set of images, in particular each image being substantially related to a different point in time. According to another embodiment of the invention, the images may be a single user or a plurality of users.

According to some embodiments of the invention, the initialization module may detect and define (1) color, (2) organization's parameters, environment, and other parameters related to the user.

According to some embodiments of the invention, the user can be any person and / or animal and / or a moving object entering the frame.

According to some embodiments of the invention, the image partition module may extract partition data from the image. According to another embodiment of the present invention, the partition data may also include the following.

Color

Movement

Edge detection

Texture

According to some embodiments of the invention, the partition data processing module may process the partition data. According to another embodiment of the invention, the partition data can be processed in the following way:

Color-using known color parameters to detect composition and / or light changes, for example using skin color to detect palms and faces

Motion-detects moving configurations in the frame

Background removal

Edge detection-detects edges in images

Texture-uses known texture parameters to detect configuration

According to some embodiments of the present invention, the partition data processing module may detect a deviation in tissue distance from the image acquisition assembly according to the deviation of the relative size of the tissue.

According to some embodiments of the invention, the scoring module (1) examines the processed compartment data, (2) evaluates the quality of the processed compartment data, and (3) the It is possible to determine whether parts of the partition data are reliable enough to be used by the HMI system.

According to some embodiments of the invention, the 3D skeletal prediction module may predict the location of a 3D skeleton that may have the best match or correlation to the processed image.

According to some embodiments of the invention, the three-dimensional prediction module may use a set of dynamic and motion processes to predict the position of a fraudulent three-dimensional skeleton.

According to some embodiments of the invention, the projection module may project a skeleton on the image. According to some embodiments of the invention, this projection may be in a two-dimensional plane.

According to some embodiments of the invention, the fitting module may fit compartment data into the projected skeleton. According to some embodiments of the invention, the fitting module may fit portions of the partition data to portions of the projected skeleton.

According to some embodiments of the invention, the scoring and error detection module (1) examines the processed skeleton after the partition data is associated, (2) calculates the fitting quality of the skeleton, (3) It is possible to determine whether an error has occurred during the skeletal prediction process or during association of partition data.

According to some embodiments of the invention, the recovery module may recover the detected error. According to another embodiment of the present invention, this recovery comprises a multiple processing layer that repartitions the image, uses a 3D skeletal motion history to re-predict the correct position, and performs reprojection and refitting of the 3D skeleton. can be performed by a process of multiple processing layers).

According to some embodiments of the invention, the three-dimensional correlation module may update the position of the three-dimensional skeleton according to the position of the fitted skeleton.

According to another embodiment of the present invention, the update processor couples the 3D skeleton on the fit skeleton, fits between the 3D skeleton and the fit skeleton, and updates the 3D skeleton to the correct position.

This embodiment is directed to the invention specifically pointed out and explicitly claimed at the end of the description. However, the description of the invention with respect to the configuration and method of operation and their purpose, features and beneficial effects can be understood with reference to what is disclosed in the accompanying drawings and the description.

1 shows a block diagram illustrating a system according to some embodiments of the invention.

2 shows a flowchart illustrating the steps of an HMI system according to some embodiments of the invention.

3 shows a block diagram illustrating a system according to some embodiments of the invention.

4 shows a flow chart illustrating the steps of an HMI system according to some embodiments of the invention.

5 shows a block diagram illustrating a system according to some embodiments of the invention.

6 shows a flow chart describing the steps of an HMI system according to some embodiments of the invention.

For the sake of clarity, the components illustrated in the drawings are not necessarily drawn to scale. For example, any dimension of a component may be exaggerated for clarity from other components. And where considered reasonable, reference numerals may be repeated among the figures that represent equivalent or similar components.

In the following detailed description, various detailed descriptions are made to help a thorough understanding of the present invention. However, even without a detailed description, those skilled in the art can understand the matter. In other instances, well-known methods, procedures, components, and circuits have been omitted from the scope of the invention so as not to obscure the present invention.

On the other hand, if not described in detail, as will be apparent from the discussion below, the registers and / or registers of the computing system are in memory, registers, or other data that are simply represented in physical quantities in such other information storage media, transmissions, or display devices. Or " processing ", " computing ", relating to the operation and / or process of a computer or computing system or similar electronic computing device that deals with and / or transforms physically represented data, such as an electron with a quantity having memory. It is understood throughout the detailed description using the term "calculation", "determination" or similar terms.

Embodiments of the present invention may include an apparatus for performing an operation in consideration of this. The device may be specially made to suit the needs or may comprise a general purpose computer which is selectively activated or reconfigured by a computer program stored in the computer. Computer programs may be stored in readable storage media, which may include floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memories (ROMs), read-only memories (RAMs), and EPROMs (random). access memories (RAMs) electrically programmable read-only memories (EEPROMs), electrically erasable and programmable read only memories (EEPROMs), magnetic or optical cards, or other types of media suitable for storing electrical instructions and being connectable to a computer system bus. May be, but is not necessarily limited thereto.

According to some embodiments of the present invention, a 3D human machine interface (“3D HMI”) is provided, wherein the 3D HMI comprises (1) an image acquisition assembly, (2) an initialization module, (3) an image partition module, (4) Compartment data processing module, (5) scoring module, (6) projection module, (7) fitting module, (8) scoring and error detection module, (9) recovery module, (10) three-dimensional correlation Relationship module, (11) three-dimensional skeletal prediction module, and (12) output module.

According to some embodiments of the invention, the image acquisition assembly acquires a set of images, where each image is generally associated with a different point in time. According to another embodiment of the present invention, the image may be a single user or multiple users.

According to some embodiments of the invention, the initialization module may detect and define (1) color of the user, (2) organization's parameters, environment, and other parameters related to the user. Thresholds, scores for all image segments, etc.) can determine the best method for image segmentation.

According to some embodiments of the invention, the image partition module may extract partition data from an image. According to another embodiment of the present invention, the partition data includes the following.

Color

Movement

Edge detection

Texture

According to some embodiments of the present invention, the partition data processing module may process the partition data. According to another embodiment of the present invention, the partition data may be processed by the following method.

Color-using known color parameters to detect composition and / or light changes, for example using skin color to detect palms and faces

Motion-detects moving configurations in the frame

Edge detection-detects edges in images

Texture-uses known texture parameters to detect configuration

According to some embodiments of the present invention, the scoring module (1) examines the processed partition data, (2) measures the quality of the processed partition data, and according to this quality (3) It can be determined whether the parts are reliable enough to be used by the HMI system.

According to some embodiments of the present invention, the 3D skeletal prediction module may predict the position of the 3D skeleton that may have the best match or correlation with the processed image.

According to another embodiment of the present invention, the three-dimensional prediction module is limited in that it is derived from the type of skeleton used above, for example if the skeleton's head cannot be rotated 360 degrees when the skeleton is a human shape. Can be used.

According to some embodiments of the invention, the three-dimensional prediction module may predict the position of the three-dimensional skeleton using a set of dynamic and motion processes.

According to some embodiments of the invention, the projection module may project a skeleton on the image. According to some embodiments of the invention, the projection may be in a two-dimensional plane.

According to some embodiments of the invention, the fitting module may fit compartment data into the projected skeleton. According to some embodiments of the invention, the fitting module may fit portions of the partition data to portions of the projected skeleton.

According to some embodiments of the invention, the scoring and error detection module (1) examines the processed skeleton after the partition data has been combined, (2) calculates the fitting quality of the skeleton and (3) It may be determined whether an error occurred during the skeletal prediction process or during association of the partition data.

According to some embodiments of the invention, the recovery module can recover the detected error. According to another embodiment of the present invention, this recovery comprises a multiple processing layer that repartitions the image, uses a 3D skeletal motion history to re-predict the correct position, and performs reprojection and refitting of the 3D skeleton. can be performed by a process of multiple processing layers).

According to some embodiments of the invention, the three-dimensional correlation module may update the position of the three-dimensional skeleton according to the position of the fitted skeleton.

According to another embodiment of the present invention, the update processor couples the 3D skeleton on the fit skeleton, fits between the 3D skeleton and the fit skeleton, and updates the 3D skeleton to the correct position.

Referring to FIG. 1, an exemplary HMI system is shown in accordance with some embodiments of the invention, and a flow chart describing the steps of such an HMI system can be better described with FIG. 2.

According to some embodiments of the invention, FIG. 1 discloses a 3D human machine interface (referred to as 3D HMI), which comprises (1) an image acquisition assembly (1000), (2) an initialization module ( initializing module (1100), (3) image segmentation module (1200), (4) segmented data processing module (1300), (5) scoring module (1400), (6 ) Projection module (1450), (7) fitting module (1500), (8) scoring and error detection module (1550), (9) recovery module (1600). ), (10) three dimensional correlation module (1700), (11) three dimensional skeleton prediction module (1800), and (12) output module (1900). Include.

According to some embodiments of the invention, the image acquisition assembly acquires a set of images, as shown in step 2000, wherein substantially each image is associated with a different point in time. According to some other embodiments of the invention, the image may be a single user or multiple users.

According to some other embodiments of the invention, the image acquisition assembly may comprise a digital camera, a web camera, a film camera, a video camera, a web camera, a digital video camera, an analog video camera, a stereo camera and / or another type of camera known today or It can include any kind of camera that will come in the future.

According to some embodiments of the invention, after the system acquires one or more images, the system enters an initialization step (step 2100), which is performed by the initialization module 1100. The initialization module may detect and define (1) color, (2) organization's parameters, (3) environment, and other parameters related to the user.

According to some embodiments of the invention, the system may extract partition data, as shown in step 2200, wherein the partition data includes the following.

Color

Movement

Edge detection

Texture

According to another embodiment of the present invention, the image partition module 1200 may extract the partition data from an image.

According to some embodiments of the present invention, the system may process the partition data, as shown in step 2300. According to another embodiment of the present invention, the partition data may be processed by the following method.

Color-using known color parameters to detect composition and / or light changes, for example using skin color to detect palms and faces

Motion-detects moving configurations in the frame

Background removal

Edge detection-detects edges in images

Texture-uses known texture parameters to detect configuration

According to another embodiment of the present invention, the partition data processing module 1300 may process the partition data.

According to some embodiments of the present invention, the system may assess the quality of the partition data, as shown in step 2400, which assessment may comprise (1) examining the processed partition data and (2) By estimating the quality of the processed partition data, and determining (3) whether the section data portion is reliable enough for use by the HMI system according to the estimated quality.

According to another embodiment of the present invention, the scoring module 1400 may evaluate the quality of the partition information.

According to another embodiment of the present invention, the system can predict the location of the three-dimensional skeleton, as shown in step 2800, which location will have the best match or correlation with the processed image. According to another embodiment of the present invention, this prediction is a limitation derived from the type of the skeleton, for example when the skeleton is of human shape, the head of the skeleton cannot be rotated 360 degrees without the movement of the shoulder. To make it more accurate.

According to some embodiments of the present invention, the prediction result may use dynamic and motion processes and the like.

According to some embodiments of the present invention, the 3D skeleton prediction module 1800 may predict the position of the 3D skeleton.

According to some embodiments of the present invention, the system may project the skeleton onto the image, as shown in step 2450. According to some embodiments of the invention, this prediction can be applied to a two-dimensional plane.

According to some embodiments of the invention, the projection module 2450 may project the skeleton on the image.

According to some embodiments of the invention, the system is capable of fitting the partition data to the projected skeleton as shown in step 2500. According to some embodiments of the invention, this fitting process includes fitting a portion of the partition data to a portion of the projected skeleton.

According to some embodiments of the present invention, fitting the partition data includes associating a portion of the extracted partition data with a current skeleton parameter, wherein the current skeleton parameter provides relevant portions of the extracted partition data. can do.

According to another embodiment of the present invention, the result of this process is a "pitted backbone".

According to some embodiments of the invention, the fitting module 2500 may couple the partition data to the projected skeleton.

According to some embodiments of the present invention, the system may score the fitted skeleton and detect errors, as shown in step 2550. According to some embodiments of the present invention, scoring and detecting errors include (1) examining the fitted skeleton, (2) assessing the fitting quality of the skeleton, and (3) the skeletal prediction process. Determining whether an error has occurred during or while connecting partition data.

According to some embodiments of the invention, the scoring and error detection module 1550 may assign scores and detect errors.

According to some embodiments of the invention, if an error is detected in step 2550, the system may enter a recovery step, as shown in step 2600. This recovery process can be a process of multiple processing layers.

According to some embodiments of the invention, said restoring step comprises re-partitioning the image, re-predicting the 3D skeletal position, re-projecting, and re-fitting the skeleton again. Include. Further, according to another embodiment of the present invention, in case of an error in the image information, the system may determine to skip one frame or more.

According to some embodiments of the invention, while recovery is performed, the system may detect an object whose tracking is not within the frame. According to another embodiment of the present invention, the system may skip one or more frames until the object returns to the frame.

According to another embodiment of the present invention, the recovering step may return the system to the initializing step.

According to some embodiments of the invention, the recovery module 2600 may perform the recovery process.

According to some embodiments of the invention, if no error is detected during step 2550, the system may update a portion of the three-dimensional skeleton with a portion of the fitted skeleton, as shown in step 2700. . According to another embodiment of the invention, the update process comprises (1) projecting a 3D skeleton onto the fitted skeleton, (2) coupling the 3D skeleton to the fitted skeleton, and (3) It may include updating the position of the 3D skeleton.

According to some embodiments of the invention, the three-dimensional correlation module 1700 may update the position of the three-dimensional skeleton.

According to some embodiments of the invention, the three-dimensional correlation module 1700 and skeletal prediction module 1800, PCT application of PCT / IL2005 / 000813 filed on July 31, 2005 as the same applicant Some or all of the algorithms and processes described herein may be used.

Referring to FIG. 3, an exemplary HMI system in accordance with certain embodiments of the present invention is shown and a flow chart describing the steps of such a system may be better described with FIG. 4.

According to some embodiments of the invention, FIG. 3 discloses a 3D human machine interface (called a 3D HMI), wherein the 3D HMI comprises (1) a Z-lens image acquisition assembly 3000, (2) initialization. Module 3100, (3) image partition module 3200, (4) partition data processing module 3300, (5) fitting module 3500, (6) scoring module 3550, (7) three-dimensional correlation Relationship module 3700, and (8) output module 3900.

According to some embodiments of the invention, the Z-lens acquisition assembly can acquire a set of images, as shown in step 4000, in particular, each image being substantially connected to a different location in time. According to some embodiments of the invention, the image may be a single user or multiple users.

According to some embodiments of the invention, the Z-lens acquisition assembly may be mounted in an image acquisition assembly other than the component shown at 1000 in FIG. 1.

According to another embodiment of the present invention, the Z-lens acquisition assembly 3000 is the same applicant as PCT / IL2006 / 001254 filed on October 31, 2006, and the same applicant as the applicant. This may be better explained in connection with the US patent application of US 60 / 731,274 filed October 31, 1989.

According to some embodiments of the invention, the system may enter an initialization step performed by the initialization module 3100, as shown in step 4100. The initialization module may detect and define the user's (1) color, (2) organization's parameters, environment, and (3) other parameters related to the user.

According to some embodiments of the invention, the system may extract partition data, as shown in step 4200, wherein the partition data includes the following.

Color

Movement

Edge detection

Texture

According to another embodiment of the present invention, the image partition module 3200 may extract the partition data from an image.

According to some embodiments of the invention, the partition data may be processed, as shown in step 4300. According to another embodiment of the present invention, the partition data may be processed by the following method.

Color-using known color parameters to detect composition and / or light changes, for example using skin color to detect palms and faces.

Motion-detects moving configurations in the frame

Background removal

Edge detection-detects edges in images

Texture-uses known texture parameters to detect configuration

According to another embodiment of the present invention, the partition data processing module 3300 may process the partition data.

According to some embodiments of the present invention, the system may fit a portion of the extracted partition data into the obtained image, as shown in step 4500. According to some embodiments of the invention, this fitting process may comprise combining the portions of said extracted partition data to relevant areas of said obtained image.

According to another embodiment of the present invention, the relevant area may be stored in the system or determined during the initialization step. According to another embodiment of the present invention, the relevant area may be a special tissue (hand, head, leg) of the user or another configuration obtained in step 3000.

According to some embodiments of the invention, this fitting process may include testing whether the extracted partition data defines a parameter associated with the relevant region.

According to some embodiments of the present invention, the fitting module 3500 may couple the partition data to the obtained image.

According to another embodiment of the invention, the result of this process is a "fitted image".

According to some embodiments of the present invention, the system may evaluate the quality of the fitted segmented data, as shown in step 4550. According to some embodiments of the present invention, evaluating the quality of the fitted partitioned data includes (1) examining the processed partitioned data, (2) estimating the quality of the processed partitioned data, According to the estimated quality, (3) determine whether the parts of the partition data are reliable enough for use by the HMI system, (4) examine the fitted image, and (5) fit the image above. Assessing quality and (6) determining whether air has occurred during the combining of the compartment data.

According to some embodiments of the invention, the scoring module 3550 may evaluate the quality of the fit partition data.

According to another embodiment of the present invention, the system may include an error detection mechanism and a rounding mechanism as mentioned above.

According to some embodiments of the present invention, the system, as shown in step 4700, locates a three-dimensional body in accordance with the fitted image and a depth map using the Z-lens image acquisition assembly. You can update the extrapolation of the depth map. According to some embodiments of the invention, this update process may comprise combining the extracted depth map and the extracted partition data and (3) updating the position of the three-dimensional body of the output model.

According to some embodiments of the invention, the three-dimensional correlation module 3700 may update the position of the three-dimensional body.

According to some embodiments of the invention, for extrapolation of depth from an image obtained using the three-dimensional correlation module 3700 and the Z-lens image acquisition assembly 3000 and in particular the Z-lens device, Better explanation with respect to PCT / IL2006 / 001254, filed October 31, 2006 as the same applicant, and US 60 / 731,274, filed October 31, 2005 as the same applicant as the applicant. Can be.

Referring to FIG. 5, an exemplary HMI system is shown in accordance with certain embodiments of the present invention, and a flow chart describing the steps of such an HMI system can be better described with FIG. 6 shown.

According to some embodiments of the invention, FIG. 5 discloses a 3D human machine interface (called 3D HMI), wherein the 3D HMI comprises (1) Z-lens acquisition assembly 5000, (2) initialization module 5100, (3) image compartment module 5200, (4) compartment data processing module 5300, (5) scoring module 5400, (6) projection module 5450, (7) fitting module 5500, (8 ) Scoring and error detection module (5550), (9) recovery module (5600), (10) three-dimensional correlation module (5700), (11) three-dimensional skeletal prediction module (5800), (12) output module (5900) And an optical (13) depth extraction module (5050).

According to some embodiments of the invention, the Z-lens acquisition assembly acquires a set of images, as shown in step 6000, and substantially each image is related to a different point in time. According to another embodiment of the invention, the image may be a single user or multiple users.

According to some embodiments of the invention, the Z-lens acquisition assembly may be mounted in an image acquisition assembly other than the component shown at 1000 in FIG. 1.

According to another embodiment of the present invention, the Z-lens acquisition assembly 5000 is PCT / IL2006 / 001254 filed on October 31, 2006 and is the same applicant as the applicant. This may be better explained in connection with the US patent application of US 60 / 731,274 filed October 31, 1989.

According to some embodiments of the present invention, after the system acquires one or more images, the system may enter an initialization step performed by the initialization module 5100, as shown in step 6100. have. The initialization module can detect and define (1) color of the user, (2) organization's parameters, (3) environment, and other parameters related to the user.

According to some embodiments of the invention, the system may extract partition data, as shown in step 6200, wherein the partition data includes the following.

Color

Movement

Edge detection

Texture

According to another embodiment of the present invention, the image partition module 5200 may extract the partition data from an image.

According to some embodiments of the present invention, the system may process the partition data, as shown in step 6300. According to another embodiment of the present invention, the partition data may be processed by the following method.

Color-using known color parameters to detect composition and / or light changes, for example using skin color to detect palms and faces.

Motion-detects moving configurations in the frame

Background removal

Edge detection-detects edges in images

Texture-uses known texture parameters to detect configuration

According to another embodiment of the present invention, the partition data processing module 5300 may process the partition data.

According to some embodiments of the present invention, the system may assess the quality of the partition data, as shown in step 6400, which assessment may include (1) examining the processed partition data and (2) By estimating the quality of the processed partition data, and determining, according to the estimated quality, that parts of the partition data are reliable enough for use by the HMI system.

According to another embodiment of the present invention, the scoring module 5400 may evaluate the quality of the partition information.

According to some embodiments of the present invention, the system can predict the location of the three-dimensional skeleton, as shown in step 6800, which location will have the best match or correlation with the processed image. According to some embodiments of the invention, this prediction is a limitation derived from the type of skeleton, such as when the skeleton is of a human shape such that the head of the skeleton cannot be rotated 360 degrees without shoulder movement. To be more accurate.

According to some embodiments of the present invention, the prediction result may use dynamic and motion processes and the like.

According to some embodiments of the present invention, the 3D skeleton prediction module 5800 may predict the position of the 3D skeleton.

According to some embodiments of the invention, the system is capable of extracting the depth using the Z-lens acquisition assembly, as shown in step 6050, and extraction of the depth using the Z-lens acquisition assembly is (1). ) PCT / IL2006 / 001254, filed October 31, 2006 as the same applicant, and (2) US 60 / 731,274, filed October 31, 2005, as the same applicant as the applicant. Are described.

According to some embodiments of the present invention, the depth extraction module 5050 may extract the depth from the obtained image.

According to some embodiments of the invention, the system is also capable of projecting a skeleton on the image, as shown in step 6450. According to some embodiments of the invention, this projection is also applicable in a two-dimensional plane.

According to some embodiments of the present invention, the projection of the skeleton is also possible in a three-dimensional plane if the module 5050 is used.

According to some embodiments of the invention, the projection may be on a three-dimensional image, which is a three-dimensional image and / or a point.

According to some embodiments of the invention, the projection module 6250 may project the skeleton onto the image.

According to some embodiments of the invention, the system is able to fit the partition data into the projected skeleton, as shown in step 6500. According to some embodiments of the present invention, this fitting process includes combining portions of the partition data to portions of the projected skeleton.

According to some embodiments of the invention, fitting the partition data comprises combining the portions of the extracted partition data to a current skeleton parameter, wherein the current skeleton parameter provides relevant portions of the extracted partition data. can do.

According to another embodiment of the present invention, the result of this process is a "fitted backbone".

According to some embodiments of the invention, the fitting module 5500 may couple the partition data to the projected skeleton.

According to some embodiments of the present invention, the system may score and detect errors in the fitted skeleton as shown in step 6550. According to some embodiments of the present invention, scoring and detecting errors include (1) examining the fitted skeleton, (2) assessing the fitting quality of the skeleton, and (3) performing the skeletal prediction process. Determining whether an error has occurred during or during joining the partition data.

According to some embodiments of the invention, the scoring and error detection module 5550 may assign scores and detect errors.

According to some embodiments of the invention, if an error is detected in step 6550, the system may enter a recovery phase, as shown in step 6600. This recovery process can be a process of multiple processing layers.

According to some embodiments of the invention, said restoring step comprises re-partitioning the image, re-predicting the 3D skeletal position, re-projecting, and re-fitting the skeleton again. Include. Further, according to another embodiment of the present invention, when there is an error in the image information, the recovery module may determine to skip one frame or more.

According to some embodiments of the invention, while recovery is performed, the system may detect an object whose tracking is not within the frame. According to another embodiment of the present invention, the system may skip one or more frames until the object returns to the frame.

According to another embodiment of the present invention, the recovering step may return the system to the initializing step.

According to some embodiments of the invention, the recovery module 5600 may perform the recovery process.

According to some embodiments of the present invention, if no error is detected during step 6550, the system may update the location of the three-dimensional skeleton to the location of the fitted skeleton, as shown in step 6700. have. According to another embodiment of the invention, the update process comprises (1) projecting a 3D skeleton onto the fitted skeleton, (2) combining the three-dimensional skeleton to the fitted skeleton, and (3) Extracting a depth using the Z-lens assembly, (4) combining the three-dimensional skeleton with a depth parameter, and (5) updating the position of the 3D skeleton.

According to some embodiments of the invention, the three-dimensional correlation module 5700 may update the position of the three-dimensional skeleton.

According to some embodiments of the invention, the three-dimensional correlation module 5700 and skeletal prediction module 5800 are the same as the applicant of the PCT application of PCT / IL2005 / 000813 filed on July 31, 2005. Some or all of the algorithms and processes described herein may be used.

According to some embodiments of the invention, (1) a three-dimensional correlation module 5700, (2) a Z-lens image acquisition assembly 5000, a depth extraction module 5050, and in particular the Z-lens device For extrapolation of the depth from an image obtained using, PCT / IL2006 / 001254 filed on October 31, 2006 to the same applicant as the applicant, and filed October 31, 2005 to the same applicant as the applicant This may be better explained in connection with the US patent application of US 60 / 731,274.

According to some embodiments of the invention, the system described above may receive from an external source depth image and / or a three-dimensional image. According to some embodiments of the present invention, when a depth image and / or a three-dimensional image is received, the system may extract their parameters from the module involved.

The processes and displays presented herein are not fundamentally related to any particular computer or other device. Various general purpose systems may be used in the programs consistent with the disclosure herein, or it may be easy to make more specialized devices for carrying out the preferred methods. Preferred structures for these types of systems will appear from the detailed description. And embodiments of the present invention are not described with respect to any particular programming language. Various types of programming languages may be used to meet the spirit of the described invention. Those skilled in the art will naturally appreciate that the described invention can be used in any kind of wireless or wire-line system.

While certain features of the invention have been shown and described, various modifications, substitutions, changes, and substitutions with equivalents may occur in the art. Accordingly, the appended claims are intended to cover such modifications and variations as fall within the spirit of the invention.

Claims (20)

An image acquisition assembly for acquiring a set of two-dimensional images of a user with each image relating to a different point in time; And A processing unit that induces one or more estimated user bodies by associating a three-dimensional data model of a template body with an extrapolated data set from one or more images acquired by the image acquisition assembly. 3D human machine interface. The method of claim 1, Initialization module, image compartment module, compartment data processing module, scoring module, depth extraction module, projection module, fitting module, scoring and error detection module, recovery module, 3D skeletal prediction module 3D human machine interface further comprising one or more modules selected from the group consisting of: a three-dimensional correlation module and an output module. The method of claim 2, And the image partition module extracts partition data from the image. The method of claim 3, wherein The partition data is a 3D human machine interface consisting of parameters selected from color, movement, size, edge detection and texture. The method of claim 3, wherein And the compartment data processing module processes the compartment data. The method of claim 5, The scoring module is a 3D human machine interface for evaluating the quality of the processed compartment data. The method of claim 2, The 3D skeletal prediction module predicts the position of the 3D skeleton. The method of claim 2, And the projection module projects a three-dimensional skeleton on the obtained image. The method of claim 8, The 3D human machine interface of the three-dimensional skeleton is a projection on a two-dimensional plane. The method of claim 2, The fitting module is a 3D human machine interface for coupling portions of the extracted partition data to the projected skeleton. The method of claim 2, And the three-dimensional correlation module updates the position of the three-dimensional skeleton. The method of claim 1, And the image acquisition assembly is selected from the group comprising a film camera, a web camera, a digital video camera, an analog video camera and a stereo camera. A processing unit for deriving one or more estimated user bodies to be equivalent from the set of two-dimensional images; And An image acquisition assembly for acquiring one or more sets of two-dimensional image data using two or more optical paths; Each of the optical paths comprises a lens, a mirror and a diaphragm structure, each optical path collecting two-dimensional optical image information from a common projection surface. The method of claim 13, At least one module selected from the group consisting of an initialization module and an image partition module, 3D human machine interface further comprising a compartment data processing module, a fitting module, a scoring and error detection module, a depth extraction module, a recovery module, a three-dimensional correlation module and an output module. The method of claim 14, And the image partition module extracts partition data from the image. The method of claim 15, And the partition data comprises a parameter selected from the group consisting of color, motion, size, edge detection and texture. The method of claim 15, The compartment data processing module is configured to process the compartment data. The method of claim 14, The fitting module couples portions of the extracted partition data to the obtained image. The method of claim 14, And the three-dimensional correlation module updates the position of the three-dimensional body. The method of claim 13, And the image acquisition assembly is a Z-lens assembly.

Images