JP2013047961A - Human/machine interface device system and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide the improved system and method of a human/machine interface in the field of a user interface.SOLUTION: There is disclosed a three-dimensional human/machine interface (3D HMI) in which the 3D HMI includes: (1) image acquisition assembly; (2) start module; (3) image division module; (4) division data processing module; (5) scoring module; (6) projection module; (7) compatible module; (8) scoring and error detection module; (9) recovery module; (10) three-dimensional correlation module; (11) three-dimensional skeleton prediction module; (12) output module; and (13) depth extraction module.

Description

  This application includes US Provisional Patent Application No. 60 / 592,136 filed July 30, 2004, PCT / IL2005 / 000813 filed July 31, 2005, and US Provisional Patent Application filed October 31, 2005. No. 60 / 731,274, U.S. Patent Application No. 11 / 227,578, filed March 27, 2006, and PCT / IL2006 / 001254, filed Oct. 31, 2006. . Each application is incorporated by reference in its entirety.

  The present invention relates generally to user interfaces, and more particularly to 3D human-machine interface methods and systems.

  One of the biggest patterns in software history is the shift from computation-intensive design to presentation-intense design. As machines have become more and more powerful, the inventor has been working hard to steadily increase some of this power in presentations. The history of this progress can be divided into three eras for convenience. Batch (1945-1968), command-line (1969-1983), and graphical (1984 and later). Of course, this story begins with the invention of the digital computer. The start dates of the latter two eras are the years when vibrant new interface technologies have moved out of the laboratory and have begun to seriously change user expectations regarding interfaces. These technologies were interactive time-sharing technology and graphic user interface technology.

  In the era of batches, computing power was very low and expensive. The largest computer of this era commanded fewer logic cycles per second than today's normal toaster or microwave oven, and much less than today's cars, digital watches and cell phones. Therefore, the user interface has not been developed. Conversely, the user had to apply to the computer, the user interface was considered expensive, and the software was designed to maximize the processor at the lowest possible cost.

  The input side of the user interface for batch machines was mainly punch cards or equivalent media such as paper tape. The output side added a line connecting the printer to these media. With limited exceptions in the system operator console, people were not interacting with the batch machine in real time.

  Providing jobs to the batch machine involved first preparing a punch card deck describing the program and data set. The punching of program cards was not done by the computer itself, but by a machine like a specialized typewriter that was very bulky, unacceptable, and had many mechanical failures. The software interface was similarly unacceptable in that the meaning of the syntax to parse the wording was very limited by the smallest possible compiler and translator.

  When you punch a card, put it in the job queue and wait. Eventually, the operator pushes the deck into the computer and probably wears a magnetic tape that supplies another data set or helper software. This job prints out a final result or a (not very often) abort notice accompanied by an error log. If it worked well, it could write the results to magnetic tape, or it could make several data cards for later calculations.

  The time required for one job was often several days. If you were very lucky, it could take hours, but the real-time response was unheard of. But there was something worse than a card queue. Some computers required a process that was error-prone and more error-prone than using a console switch to toggle a binary code program.

  Early batch systems gave the entire computer a job that was still running. The program deck and tape had to have what we now consider operating system code to talk to I / O devices, and did whatever else needed maintenance. Since 1957, the middle of the batch era, various groups started experimenting with the so-called “load and run” system. These systems always used a monitor program that was in the computer. The program was able to call the monitor as a service. Another function of the monitor was to perform better error checking on the provided job, catch errors faster and sensible, and generate user-friendly feedback. Thus, the monitor represented the first step towards both the operating system and a well-designed user interface.

  Command line interfaces (CLIs) developed from a batch monitor connected to the system console. The interaction model is a transaction that requires a series of responses, and the request was expressed as a text command in a particular vocabulary. Latency was much larger than batch system latency, dropping from days or hours to seconds. Thus, the command line system allowed the user to change their mind about the post-transaction stage in response to feedback on previous results in real time or near real time. The software was preliminary and interacted in ways that were not possible before. However, these interfaces impose a relatively heavy mnemonic load on the user and require a series of effort and learning time to master.

  The command line interface was very closely related to the rise of time-sharing computers. The concept of time-sharing data reverted back to the 1950s, and the most promising early experiment was the MULTICS operating system since 1965, which most affected today's command line interface was Unix itself after 1969 This has affected most systems that have since occurred.

  Early command line systems were a combination of teletypes and computers, and adapted to mature technology that was successful in mediating information transfer between people. Teletype was originally developed as a device for automatic telegraphic transmission and reception. The history of teletypes dates back to 1902, and in 1920 it was already established in newsrooms. In reusing teletypes, economic matters should have been considered, but so is psychology and the least law of surprise. Teletype provided an interface point with systems familiar to many engineers and users.

  With the wide adoption of video display terminals (VDTs) in the mid 1970s, the command line system entered the second phase. Since the characters could be projected onto dot dots on the screen faster than the printer head or carriage moved, the waiting time was further shortened. VDTs reduced conservative resistance to interactive programming by reducing consumables such as ink and paper due to cost considerations, and was a computer pioneer in the 1940s versus the first TV generation in the late 1950s and 60s. It was more symbolic and comfortable than the teletype.

  As important as the presence of accessible screens, software designers are not costly to develop a 2D display of text that can be quickly and reversibly transformed into a visual interface instead of text. I did it. Applications that are pioneers of this kind are computer games and text editors. Older versions close to some of the early examples such as logue (6) and vi (1) are still unix style live parts.

  Screen video displays are not entirely new, and have appeared in minicomputers as early as the 1961 PDP-1. However, until the transition to VDTs attached via a serial cable, the console could support only one display that could be addressed by an expensive computer. Under these conditions, it has been difficult to develop a visual UI style. Such an interface was an accidental event that occurred only in rare circumstances where all computers could at least temporarily run exclusively for a single user.

  Since 1962 there have been sporadic experiments with what we now call a graphical user interface and the pioneer PDP-1 SPACEWAR game. The machine display was not just a character terminal, but a modified oscilloscope designed to support vector graphics. The SPACEWAR interface, which mainly uses a toggle switch, also characterizes the early prototype trackball and was custom made by the player himself. Ten years later, in the early 1970s, these experiments spawned the video game industry, which began with an attempt to actually create an arcade version of SPACEWAR.

  The PDP-1 console display originated from the WWII radar display tube early 20 years, and several key pioneers of minicomputers at the Lincoln Laboratory at the Massachusetts Institute of Technology (MIT) Reflects that he was an engineer. In the same 1962, another former radar engineer began to become another pioneer in the Stanford Research Institute across the continent. His name is Doug Engelbart. He was inspired by both his personal experience with these very early graphical displays and As We May Think, Vannevar Bush's seminary essay which gave him a vision now called hypertext in 1945.

  In December 1968, Engelbart and his team at SRI gave a 90-minute public demonstration of NLS / August, the first hypertext system. The demonstration included a three-button mouse (Engelbart's invention), a multi-window interface graphical display, hyperlinks, and the first video conference on screen. This demonstration was a sensation that resonated in the computer science world for a quarter of a century up to and including the invention of World Wide Web in 1991.

  Thus, in the early 1960s, it was already well understood that graphical presentations attracted users. A pointing device equivalent to a mouse has already been invented, and many large general-purpose computers in the late 1960s had a display capacity comparable to PDP-1. One of the authors has a vivid memory about playing another video game very early in 1968 on the Univac 1108 console that would cost nearly $ 45 million if purchased as of 2004 ing. But at $ 45 million, there were only a few actual customers for interactive images. The custom hardware for the NLS / Augment system was not so expensive, but it was still too expensive for general use. Even the $ 10,000 PDP-1 was a machine that was too expensive to establish a graphics processing programming style.

  Video games have become huge market devices faster than computers. This is because the game is a very inexpensive and simple processor and runs a program embedded in the hard wire. However, on general purpose computers, oscilloscope displays have come to an end. The concept of using a graphical, visual interface for normal interaction with a computer has to wait a few years and in fact comes with the latest graphics capable version of the serial line character VDT of late 1970 did.

  Since the earliest PARC systems of the 1970s, the design of GUIs was almost completely dominated by what came to be called the WIMP (Windows, icons, Mice, Pointer) model, which Alto led. Given the significant changes in computing and display hardware over the next decades, it is surprisingly difficult to think beyond WIMP.

  Several attempts have been made. Perhaps the most prominent attempt is the VR (Virtual Reality) interface. With this interface, the user navigates through an immersive graphical 3-D environment and gestures within it. VR has attracted a large research community since mid-1980s. The computational power to support these is no longer expensive, but physical display devices have priced VR beyond the common use in 2004. A more fundamental problem, familiar to flight simulator designers, is that VR is a way to disrupt human-specific sensory systems. Rather, VR movement at normal speed may cause dizziness and nausea as the brain attempts to match the visual simulation of movement with the informed report of the body's real world movement.

  JEF Raskin's THE (The Human Environment) project investigates the GUIs zoom world model and states that it will be spatialized without going to 3D. In THI, the screen becomes the window of a 2D virtual world where data and programs are organized by spatial location. The objects in this world can be represented in several different levels depending on the height above the reference plane, and the most basic selection action is zooming in and landing there.

  The Yale University Lifestreams project is actually moving in the opposite direction, de-spatching the GUI. The user's document is represented as a kind of world line or temporary stream organized by modification date and can be filtered in various ways.

  All three of these approaches truncate traditional file systems in favor of a context that avoids naming things and using names as the primary form of reference. This makes it difficult to match these approaches to the Unix architecture file system and hierarchical namespace, which seems to be one of the longest-lasting and useful features. Nevertheless, it can be seen that one of these early experiments is still the same seminar as NLS / Augment's Engelbart 1968 demo.

  There is a need in the user interface field for improved systems and methods for man-machine interfaces.

  In accordance with an embodiment of the present invention, a 3D human machine interface (“3D HMI”) is provided, which includes (1) an image acquisition assembly, (2) an activation module, (3) an image segmentation module, ( 4) split data processing module, (5) scoring module, (6) projection module, (7) fitting module, (8) scoring and error detection module, (9) recovery module, (10) three-dimensional correlation module, (11) a three-dimensional skeleton prediction module; and (12) an output module.

  According to an embodiment of the present invention, the image acquisition assembly can be configured to acquire an image set, wherein substantially each image is used at a different time. In a further embodiment of the invention, this image relates to a single user or to multiple users.

  In accordance with an embodiment of the present invention, the activation module can be configured to detect and define the user's (1) color, (2) organ parameters, environment, and other parameters associated with the user.

  According to embodiments of the present invention, the user may be a person and / or animal and / or a moving object that enters its frame.

According to an embodiment of the present invention, the image segmentation module can be configured to extract segmented data from an image. According to a further embodiment of the invention, this split data is also:
-Color / movement / edge detection / texture may be provided.

According to an embodiment of the present invention, the divided data processing module can be configured to process the divided data. According to a further embodiment of the invention, the split data can be processed in the following way:
Color—Detects element and / or light changes using known color parameters. For example, palms and faces are detected using skin color.
• Movement-detects moving elements in the frame.
-Background removal.
Edge detection—detects the edge of the image.
Texture—Detects elements using known texture parameters.

  According to an embodiment of the present invention, the split data processing module is configured to detect a deviation in organ distance from the image acquisition assembly in response to a deviation in relative organ size.

  According to an embodiment of the invention, the scoring module (1) examines the processed divided data, (2) predicts the quality of the processed divided data, and depending on this quality, which part of the divided data Can be configured to determine if it is reliable enough for use by the HMI system.

  According to an embodiment of the present invention, the 3D skeleton prediction module can be configured to predict the position of the 3D skeleton having the best match or relationship of the processed images.

  According to an embodiment of the present invention, the 3D prediction module has constraints from the type of skeleton used, such as whether the skeleton is a human finger or whether the skeleton head can rotate 360 degrees. Can be used.

  According to a further embodiment of the present invention, the 3D prediction module can also predict the position of the 3D skeleton using dynamic and motion process sets.

  According to an embodiment of the present invention, the projection module can be configured to project a skeleton on an image. According to a further embodiment of the invention, this projection can be applied to a two-dimensional plane.

  According to the embodiment of the present invention, the division data can be adapted to the skeleton on which the adaptation module is projected. According to a further embodiment of the present invention, the adaptation module may be configured to associate the segmented data portion with the projected skeleton portion.

  According to an embodiment of the present invention, the scoring and error detection module (1) examines the post-processed skeleton associated with the segmented data, (2) tests the quality of the skeleton, and (3) skeleton prediction. It can be configured to determine whether an error has occurred during the association of the program or split data.

  According to an embodiment of the present invention, the recovery module can be configured to recover from detected errors. According to a further embodiment of the invention, the recovery is a multi-stage processing layer that re-segments the image, uses the 3D skeleton movement history, re-predicts the correct position, re-projects the 3D skeleton and adapts again. It may be a process. The recovery module may also decide to skip the frame if the image information is corrupted.

  According to an embodiment of the present invention, the three-dimensional correlation module can be configured to update the position of the three-dimensional skeleton according to the position of the matching skeleton.

  According to a further embodiment of the present invention, the update process relates to a three-dimensional 3D skeleton on a compatible skeleton, matches between the 3D skeleton and the compatible skeleton, and updates the 3D skeleton to the proper position.

The subject matter relating to the invention is particularly pointed out and distinctly claimed in the final part of the specification. However, the present invention relates to both tissue and method of operation, and together with its objects, features and advantages, will be best understood by reading the following detailed description in conjunction with the accompanying drawings. .
FIG. 1 is a block diagram illustrating a system according to an embodiment of the present invention. FIG. 2 is a flowchart describing the steps of an HMI system according to an embodiment of the present invention. FIG. 3 is a block diagram illustrating a system according to an embodiment of the present invention. FIG. 4 is a flowchart describing the steps of an HMI system according to an embodiment of the present invention. FIG. 5 is a block diagram illustrating a system according to an embodiment of the present invention. FIG. 6 is a flowchart describing the steps of an HMI system according to an embodiment of the present invention.

  The elements shown in the drawings need not be to scale in order to make the figures simple and clear. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. In addition, symbols may be repeatedly used between the drawings to display corresponding or similar elements.

  In the following detailed description, specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be understood without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail but have been described so as not to obscure the present invention.

  Unless otherwise stated, as will be apparent from the following description, throughout this specification, descriptions using terms such as “process”, “calculate by computer”, “calculate”, and “determine” Means the operation and / or process of a system, or similar electronic computer device, which represents data represented as physical quantities, such as electronics in a computer system's registers and / or memory, in a computer system's memory, registers Alternatively, other data is manipulated and / or transformed into other data similarly represented as physical quantities in the device to store, transfer or display information.

  Embodiments of the present invention include devices that perform the operations described herein. The apparatus may be specially configured for the desired purpose, or may comprise a general purpose computer that is selectively achieved or reconstructed by a computer program stored on the computer. . Such computer programs include but are not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read only memory (ROM), random access memory (RAM), electronically programmable read only memory. Computer readable, such as any type of disc, including (EPROM), electronically erasable and programmable read-only memory (EEPROM), magnetic or optical cards, or other media suitable for storing electronic instructions Can be stored on a storage medium connectable to the computer system bus.

  According to an embodiment of the present invention, a 3D human-machine interface (3D HMI) is provided, which includes (1) an image acquisition assembly, (2) an activation module, (3) an image segmentation module, and (4) split data processing module; (5) scoring module; (6) projection module; (7) matching module; (8) scoring and error detection module; and (9) recovery module; (10) a three-dimensional correlation module; (11) a three-dimensional skeleton prediction module; and (12) an output module.

  According to an embodiment of the invention, the image acquisition assembly is configured to acquire a set of images, where each image is used at a substantially different point in time. In further embodiments of the invention, the image may relate to a single user or may relate to multiple users.

  According to an embodiment of the present invention, the activation module detects and defines the user's (1) color, (2) tissue parameters, environment, and other parameters associated with the user, and in the next step It can be configured to determine the best method (threshold, score for each image segmentation, etc.).

According to an embodiment of the present invention, the image segmentation module is configured to extract segmented data from the image. According to a further embodiment of the invention, this split data is:
-Color / movement / edge detection / texture may be provided.

According to an embodiment of the present invention, the divided image processing module is configured to process the divided data. According to a further embodiment of the invention, this split data is:
Color—Detects element and / or light changes using known color parameters. For example, palms and faces are detected using skin color.
• Movement-detects moving elements in a frame.
-Background removal.
Edge detection—detects the edge of the image.
Texture—Detects elements using known texture parameters.
It is possible to perform processing by such a method.

  According to an embodiment of the present invention, the scoring module (1) verifies the processed divided data, (2) predicts the quality of the processed divided data, and (3) the divided data according to the quality. Determine which part of the is reliable enough to be used by the HMI system.

  According to an embodiment of the present invention, the 3D skeleton prediction module can be configured to predict the position of the 3D skeleton that best matches or is most relevant to the processed image.

  According to a further embodiment of the invention, the 3D prediction module can use the restrictions derived from the type of skeleton used. For example, if the skeleton is a human figure, the skeleton head cannot rotate 360 degrees.

  According to a further embodiment of the present invention, the 3D prediction module can predict the position of the 3D skeleton using a dynamic and moving process set.

  According to an embodiment of the present invention, the projection module can be configured to project a skeleton on an image. According to a further embodiment of the invention, this projection can be applied to a two-dimensional plane.

  According to an embodiment of the present invention, the adaptation module can be configured to adapt the segmented data to the projected skeleton. According to a further embodiment of the present invention, the adaptation module can be configured to associate the segmented data portion with the projected skeleton portion.

  According to an embodiment of the present invention, the scoring and error detection module (1) associates the processed skeleton with the divided data, then tests the skeleton, and (2) evaluates the conformity quality of the skeleton. , (3) determining whether an error has occurred during the skeletal prediction process or the association of the divided data.

  According to an embodiment of the present invention, the recovery module can be configured to recover from detected errors. According to a further embodiment of the invention, this recovery is a multiple process such as layer processing, image subdivision, 3D skeleton movement history to predict correct location, 3D skeleton refit. The recovery module may also decide to skip the frame if the image information is corrupted.

  According to an embodiment of the present invention, the three-dimensional correlation module can be configured to update the position of the three-dimensional skeleton according to the position of the adapted skeleton.

  According to a further embodiment of the present invention, the update process associates the 3D skeleton with the matched skeleton, matches between the 3D skeletons and matches the 3D skeleton to the correct position.

  Referring back to FIG. 1, an exemplary HMI system according to an embodiment of the present invention is shown and is best described in FIG. 2, which is a flowchart describing the steps of such an HMI system. ing.

  In accordance with an embodiment of the present invention, FIG. 1 is a diagram illustrating a 3D human machine interface (3D HMI), which includes (1) an image acquisition assembly 1000, (2) an activation module 1100, ( 3) Image segmentation module 1200, (4) Segmented data processing module 1300, (5) Scoring module 1400, (6) Projection module 1450, (7) Adaptation module 1500, (8) Scoring and errors A detection module 1550; (9) a recovery module 1600; (10) a three-dimensional correlation module 1700; (11) a three-dimensional skeleton prediction module 1800; and (12) an output module 1900.

  According to an embodiment of the present invention, the image acquisition assembly is configured to acquire an image set, as described in step 2000, and each image of the image set is used at a substantially different time. . According to a further embodiment of the invention, the image is a single user or multi-user image.

  According to a further embodiment of the invention, the image acquisition assembly comprises a digital camera, a webcam, a film camera, a video camera, a webcam, a digital video camera, an analog video camera, a stereo-camera and And / or other cameras that are known today or that will come out in the future.

  According to an embodiment of the invention, after the system has acquired one or more images, the system enters step 2100, which is an activation phase that is executed by the activation module 1100. The activation module is configured to detect and define the user's (1) color, (2) organization parameters, (3) environment, and other parameters associated with the user.

According to an embodiment of the present invention, as shown in step 2200, the system is configured to extract split data. This split data is
-Color / movement / edge detection / texture may be provided.

  According to a further embodiment of the invention, the image segmentation module 1200 is configured to extract segmented data from this image.

According to an embodiment of the present invention, as shown in step 2300, the system is configured to process split data. According to a further embodiment of the invention, this split data is:
Color—Detects element and / or light changes using known color parameters. For example, palms and faces are detected using skin color.
• Movement-detects moving elements in a frame.
-Background removal.
Edge detection—detects the edge of the image.
Texture—Detects elements using known texture parameters.
It is possible to perform processing by such a method.

  According to a further embodiment of the invention, the split data processing module 1300 is configured to process split data.

  According to an embodiment of the present invention, the system is configured to evaluate the quality of the split data, as shown in step 2400, which comprises (1) examining the processed split data; (2) predicting the quality of the processed split data, and (3) determining which part of the split data is sufficiently reliable for use by the HMI system with the predicted quality; And done.

  According to a further embodiment of the invention, scoring module 1400 is configured to evaluate the quality of the segmented information.

  According to a further embodiment of the present invention, the system is configured to predict the position of the three-dimensional skeleton as shown in step 2800, which position best matches the processed image or is processed. Correlate with the captured image. According to a further embodiment of the invention, this prediction is made more accurately with the restrictions taken from the type of skeleton used. For example, if the skeleton is in the form of a person, the head of the skeleton cannot rotate 360 degrees without shoulder movement.

  According to an embodiment of the present invention, the prediction sequence may use a dynamic motion process or other set.

  According to an embodiment of the present invention, the 3D skeleton prediction module 1800 is configured to predict the position of the 3D skeleton.

  In accordance with an embodiment of the present invention, the system can be further configured to project a skeleton onto the image, as shown in step 2450. According to a further embodiment of the invention, this projection can be applied to a two-dimensional plane.

  According to an embodiment of the present invention, the projection module 1450 can be configured to project a skeleton onto an image.

  In accordance with an embodiment of the present invention, the system can be further configured to adapt the split data using the projected skeleton, as shown in step 2500. According to a further embodiment of the invention, the fitting process may comprise associating the projected skeleton part with the segmented data part.

  According to an embodiment of the present invention, the step of adapting the split data comprises the step of associating the extracted split data with the current skeleton parameter, the current skeleton parameter supporting the relevant part of the extracted split data. To do.

  According to a further embodiment of the invention, the result of this process is a “fit skeleton”.

  According to a further embodiment of the invention, the adaptation module 1500 is configured to associate the segmented data with the projected skeleton.

  In accordance with an embodiment of the present invention, the system is further configured to score the matched skeleton and detect errors, as shown in step 2550. According to an embodiment of the present invention, the step of scoring and detecting errors comprises: (1) examining a matched skeleton; (2) evaluating the fit quality of the skeleton; and (3) a skeleton. Determining whether an error has occurred during the prediction process or associating the split data.

  According to an embodiment of the invention, the scoring and error detection module 1550 is configured to score and detect errors.

  According to an embodiment of the present invention, if an error is detected at step 2550, the system enters the recovery phase as shown at step 2600. This recovery process is a process including a plurality of processing layers.

  According to an embodiment of the present invention, the recovery phase comprises an image re-segmentation step, a 3D skeleton position re-prediction, a skeleton re-projection with extended effort and a re-fit step. According to a further embodiment of the present invention, the extension module may decide to skip one or more frames if the image information is corrupted.

  In accordance with an embodiment of the present invention, during recovery, the system is configured to detect that there is no target tracking in that frame. According to a further embodiment of the invention, the system can be configured to skip one or more frames until the subject returns to that frame.

  According to a further embodiment of the invention, the recovery phase can return the system to the startup step.

  According to an embodiment of the present invention, recovery module 1600 is configured to perform a recovery process.

  According to an embodiment of the present invention, if no error is detected during step 2550, the system is configured to update the position of the three-dimensional skeleton according to the position of the matching skeleton, as shown in step 2700. be able to. According to a further embodiment of the present invention, the update process comprises (1) projecting a 3D skeleton onto a conforming skeleton, (2) associating the 3D skeleton with the conforming skeleton, and (3) the 3D skeleton. Updating the position.

  According to an embodiment of the present invention, the three-dimensional correlation module 1700 can be configured to update the position of the three-dimensional skeleton.

  According to an embodiment of the present invention, the 3D correlation module 1700 and the skeleton prediction module 1800 are assigned to the same assignee as the present application, PCT application filed July 31, 2005, PCT / IL2005 / 000813. Some or all of the algorithms and processes described in the issue can be used.

  Referring to FIG. 3, an exemplary HMI system according to an embodiment of the present invention is shown. This system is best described in connection with FIG. 4, where a flowchart describing the steps of the HMI system is shown.

  In accordance with an embodiment of the present invention, FIG. 3 illustrates a 3D human machine interface (3D HMI) that includes (1) a Zlens image acquisition assembly 3000, (2) an activation module 3100, (3) Image segmentation module 3200, (4) Segmented data processing module 3300, (5) Adaptation module 3500, (6) Scoring module 3550, (7) Three-dimensional correlation module 3700, (8) Output A module 3900.

  According to an embodiment of the present invention, the Zlens acquisition assembly is configured to acquire a set of images, as seen in step 4000, with each image being used at a substantially different time. According to further embodiments of the invention, the image may be of a single user or multiple users.

  According to an embodiment of the present invention, the Zlens acquisition assembly may be mounted on another image acquisition assembly, for example, element 1000 of FIG.

  According to a further embodiment of the present invention, the Zlens acquisition assembly (3000) is assigned to PCT / IL2006 / 001254, filed October 31, 2006, assigned to the same assignee as the present application, It is best described in connection with US patent application 60 / 731,274, filed October 31, 2005, assigned to the same assignee as the application.

  According to an embodiment of the present invention, the system is further configured to enter an activation phase that is executed by the activation module 3100 as shown in step 4100. The activation module can be configured to detect and define (1) color, (2) organ parameters, environment, and (3) other parameters associated with the user.

According to an embodiment of the present invention, the system can be configured to extract split data, as shown in step 4200, which split data is:
・ Includes color, movement, edge detection, and texture.

  According to a further embodiment of the invention, the image segmentation module 3200 is configured to extract segmented data from the image.

According to an embodiment of the present invention, the system is configured to process the split data as shown in step 4300. According to a further embodiment of the invention, the split data is processed as follows:
Color—Detects changes in elements and / or light using known color parameters. For example, palms and faces are detected using skin color.
• Movement-detects moving elements in the frame.
-Background removal.
Edge detection—detects the contour of each organ.
Texture—Detects elements using known texture parameters.

  According to a further embodiment of the invention, the split data processing module 3300 is configured to process split data.

  According to an embodiment of the present invention, the system can be further configured to fit the extracted segmented data portion to the acquired image, as shown in step 4500. According to a further embodiment of the invention, the adaptation process may comprise associating the extracted segmented data portion with a dedicated region of the acquired image.

  According to a further embodiment of the invention, this dedicated area can be stored in the system or determined during the startup phase. According to a further embodiment of the present invention, this dedicated region may be a user's specific organ (hand, head, foot) or other elements that can be obtained at 3000.

  According to a further embodiment of the invention, the adaptation process may comprise the step of testing whether the extracted segmented data has determined the parameters associated with the dedicated area.

  In a further embodiment of the present invention, the adaptation module 3500 can be configured to associate a portion of the segmented data with the acquired image.

  According to a further embodiment of the invention, the output of this process is a “fit image”.

  According to an embodiment of the present invention, the system can be further configured to evaluate the quality of the adaptive split data, as shown in step 4550. According to the embodiment of the present invention, the step of evaluating the quality of the adaptive divided data includes (1) inspecting the processed divided data, (2) predicting the quality of the processed divided data, and the prediction. (3) determining which part of the segmented data is sufficiently reliable for use by the HMI system, (4) examining the conforming image, and (5) conforming this image. And (6) determining whether an error has occurred during the association of the divided data.

  According to an embodiment of the present invention, the scoring module 3550 is configured to evaluate the quality of the adaptive split data.

  According to a further embodiment of the invention, the system comprises an error detection mechanism and a recovery mechanism as described above.

  According to an embodiment of the present invention, the system is configured to update the position of the three-dimensional body by extrapolating the fit image and the depth map using a Zlens image acquisition assembly, as shown in step 4700. Can do. According to a further embodiment of the invention, the updating process comprises associating the extracted depth map with the extracted segmented data and updating the 3D body position of the output model.

  According to an embodiment of the present invention, the three-dimensional correlation module 3700 is configured to update the position of the three-dimensional body.

  According to an embodiment of the present invention, the functionality of the 3D correlation module 3700 and the Zlens image acquisition assembly 3000, in particular the images acquired using the Zlens device, was assigned to the same assignee as the present application, 2006. Related to PCT / IL2006 / 001254, filed Oct. 31, and US patent application 60 / 731,274, filed Oct. 31, 2005, assigned to the same assignee as the present application. And is best described.

  Referring to FIG. 5, an exemplary HMI system according to an embodiment of the present invention is shown and is best described in connection with FIG. 6, which is a flowchart describing the steps of the HMI system.

  FIG. 5 is a diagram illustrating a 3D human machine interface (3D HMI) according to an embodiment of the present invention, which includes (1) a Zlens acquisition assembly 5000, (2) an activation module 5100, ( 3) Image segmentation module 5200, (4) Segmented data processing module 5300, (5) Scoring module 5400, (6) Projection module 5450, (7) Matching module 5500, (8) Scoring and errors Detection module 5550, (9) recovery module 5600, (10) three-dimensional correlation module 5700, (11) three-dimensional skeleton prediction module 5800, (12) output module 5900, and (13) depth extraction selectively A module 5050.

  In accordance with an embodiment of the present invention, the Zlens acquisition assembly is configured to acquire a set of images, as seen in step 6000, with each image being used at a substantially different time. According to further embodiments of the invention, the image may be of a single user or multiple users.

  According to embodiments of the present invention, the Zlens acquisition assembly may be placed on another image acquisition assembly, for example, element 1000 of FIG.

  According to a further embodiment of the present invention, the Zlens acquisition assembly (5000) is a PCT / IL2006 / 001254 filed October 31, 2006, assigned to the same assignee as the present application, It is best described in connection with US patent application 60 / 731,274, filed October 31, 2005, assigned to the same assignee as the application.

  According to an embodiment of the present invention, after the system has acquired one or more images, the system is further configured to enter an activation phase step 6100 that is executed by the activation module 5100. The activation module can be configured to detect and define (1) color, (2) organ parameters, (3) environment, and other parameters associated with the user.

According to an embodiment of the present invention, the system can be configured to extract split data, as shown in step 6200, which is:
・ Includes color, movement, edge detection, and texture.

  According to a further embodiment of the present invention, the image segmentation module 5200 is configured to extract segmented data from the image.

According to an embodiment of the present invention, the system is configured to process the split data as shown in step 6300. According to a further embodiment of the invention, the split data is processed as follows:
Color—Detects changes in elements and / or light using known color parameters. For example, palms and faces are detected using skin color.
• Movement-detects moving elements in the frame.
-Background removal.
Edge detection—detects the contour of each organ.
Texture—Detects elements using known texture parameters.

  According to a further embodiment of the present invention, the split data processing module 5300 is configured to process split data.

  According to an embodiment of the present invention, the system can be configured to evaluate the quality of the split data, as shown in step 6400. This evaluation consists of (1) inspecting the processed divided data, (2) predicting the quality of the processed divided data, and (3) which part of the divided data is used by the HMI system according to the predicted quality. Determining whether it is sufficiently reliable to do.

  According to an embodiment of the present invention, the scoring module 5400 is configured to evaluate the quality of the split information.

  According to a further embodiment of the present invention, the system is configured to predict the position of the three-dimensional skeleton that best matches or correlates with the processed image, as shown in step 6800. . According to a further embodiment of the invention, this prediction is made more accurately with the restrictions taken from the type of skeleton used. For example, if the skeleton is in the form of a person, the head of the skeleton cannot rotate 360 degrees without shoulder movement.

  According to an embodiment of the present invention, the prediction sequence can use a set of dynamic motion processes.

  According to an embodiment of the present invention, the 3D skeleton prediction module 5800 is configured to predict the position of the 3D skeleton.

  According to a further embodiment of the invention, the system can be further configured to extract depth using a Zlens acquisition assembly, as shown in step 6050. This Zlens acquisition assembly was assigned to PCT / IL2006 / 001254, filed October 31, 2006, assigned to the same assignee as the present application, and assigned to the same assignee as the present application. U.S. Patent Application No. 60 / 731,274 filed on Jan. 31.

  According to an embodiment of the present invention, the depth extraction module 5050 is configured to extract the depth from the acquired image.

  In accordance with an embodiment of the present invention, the system can be further configured to project a skeleton onto the image as shown in step 6450. According to a further embodiment of the invention, this projection can be applied to a two-dimensional plane.

  According to a further embodiment of the invention, when using module 5050, this projection can be applied to a three-dimensional plane.

  According to an embodiment of the invention, this projection may be on a 3D plane and / or a 3D point cloud.

  According to an embodiment of the present invention, the projection module 5450 can be configured to project a skeleton onto an image.

  According to an embodiment of the present invention, the system can be further configured to fit the segmented data to the projected skeleton, as shown in step 6500. According to a further embodiment of the invention, this adaptation process may comprise associating a portion of the segmented data with a portion of the projected skeleton.

  According to an embodiment of the present invention, the step of adapting the divided data comprises associating the extracted divided data with the current skeleton parameter, wherein the current skeleton parameter supports the relevant part of the extracted divided data. It is a parameter.

  According to a further embodiment of the invention, the result of this process is a “compatible skeleton”.

  According to a further embodiment of the invention, the adaptation module 5500 is configured to associate the segmented data with the projected skeleton.

  According to an embodiment of the invention, the system is further configured to score the matched skeleton and detect errors, as shown in step 6550. According to an embodiment of the present invention, the step of scoring and detecting errors comprises: (1) examining a matched skeleton; (2) evaluating the fit quality of the skeleton; and (3) a skeleton. Determining whether an error has occurred during the prediction process or associating the split data.

  According to an embodiment of the present invention, this scoring and error detection module 5550 is configured to score and detect errors.

  According to an embodiment of the present invention, if an error is detected at step 6550, the system enters the recovery phase as shown at step 6600. This recovery process is a process comprising a plurality of processing layers.

  According to an embodiment of the present invention, the recovery phase comprises an image re-segmentation step, a 3D skeleton position re-prediction, a skeleton re-projection with extended effort and a re-fit step. According to a further embodiment of the present invention, the recovery module may decide to skip one or more frames if the image information is corrupted.

  In accordance with an embodiment of the present invention, during recovery, the system is configured to detect that there is no target tracking in that frame. According to a further embodiment of the invention, the system can be configured to skip one or more frames until the subject returns to that frame.

  According to an embodiment of the present invention, the recovery phrase can return the system to the startup step.

  According to an embodiment of the present invention, the recovery module 5600 is configured to perform a recovery process.

  According to an embodiment of the present invention, if no error is detected during step 6550, the system is configured to update the position of the three-dimensional skeleton according to the position of the matching skeleton, as shown in step 6700. be able to. According to a further embodiment of the present invention, the update process includes (1) projecting a 3D skeleton onto a conforming skeleton, (2) associating a three-dimensional skeleton with the conforming skeleton, and (3) Zlens. Extracting the depth using the assembly; (4) associating the three-dimensional skeleton with a depth parameter; and (5) updating the position of the 3D skeleton.

  According to an embodiment of the present invention, the three-dimensional correlation module 5700 can be configured to update the position of the three-dimensional skeleton.

  In accordance with an embodiment of the present invention, the three-dimensional correlation module 5700 and the skeleton prediction module 5800 are assigned to the same assignee as the present application, PCT application filed July 31, 2005, PCT / IL2005 / 000813. Some or all of the algorithms and processes described in the issue can be used.

  According to embodiments of the present invention, (1) the functionality of the three-dimensional correlation module 5700, (2) the Zlens image acquisition assembly 5000, and (3) the depth extraction module 5050, (4) in particular the Zlens device PCT / IL2006 / 001254, filed on October 31, 2006, to which the extrapolation of depth from images acquired using the same assignee as the present application was filed, and the same assignee as the present application. It is best described in connection with assigned US patent application 60 / 731,274 filed October 31, 2005.

  According to an embodiment of the present invention, the system described above can be configured to receive depth images and / or 3D images from external sources. According to a further embodiment of the invention, when receiving a depth image and / or a three-dimensional image from the system, the system is configured to extract its parameters in the relevant module.

  The processes and displays presented here are not inherently related to special computers or other devices. It will be appreciated that various general purpose systems can be used with the program in accordance with the teachings herein, and that more specialized devices can be conveniently constructed to perform the desired method. The desired structure for a variety of these systems will appear from the description below. Furthermore, embodiments of the present invention are not specifically described with respect to programming languages. Various programming languages can be used to implement the teachings of the invention described herein. Those skilled in the art will appreciate that the invention described herein can be used in any type of wireless or wired system.

  While certain features of the invention have been illustrated and described, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. Accordingly, the claims are intended to cover these modifications and changes as long as they fall within the true spirit of this invention.

Claims (1)

In the 3D human machine interface:
A processing unit configured to retrieve one or more predicted user body coordinates from the two-dimensional image set;
An image acquisition assembly configured to acquire one or more two-dimensional image data sets using two or more optical paths, each optical path comprising a lens, a mirror, and a diaphragm structure; An image acquisition assembly, wherein each optical path is configured to collect two-dimensional optical image information from a common projection plane;
A 3D human-machine interface characterized by comprising:

Images