DE102019107103A1 - METHOD AND SYSTEM FOR OBJECT SEGMENTATION IN A MIXED REALITY ENVIRONMENT - Google Patents

Abstract

Methods and systems for facilitating mixed reality (MR) interaction are provided. In one example, the method includes: capturing a first image of a user's limb and a real peripheral input device, wherein the limb of the user at least partially obscures the real peripheral input device; Segmenting the first image into a first pixel region corresponding to the limb of the user, wherein a second pixel region corresponds to the real peripheral input device; Creating a composite image from the first image based on rendering pixels representing a virtual peripheral input device instead of the second pixel region such that the virtual peripheral input device spatially appears in the composite image at the same location as the real peripheral input device and at least the user's limb partially obscured; and displaying the composite image to the user.

Description

REFERENCE TO RELATED APPLICATIONS

This non-provisional application takes the rights from the provisional filed on March 20, 2018 U.S. Application No. 62 / 645,760 entitled "ACCURATE HAND TRACKING ON A PERIPHERAL DEVICE IN A MIXED REALITY ENVIRONMENT" and the priority thereof, the application of which is hereby incorporated by reference in its entirety for all purposes.

• • Anmeldung Nr. 16/267,200 mit dem Titel „METHOD AND SYSTEM FOR MIXED REALITY INTERACTION WITH PERIPHERAL DEVICE“, eingereicht am 4. Februar 2019.

The following regular US patent application (including this) has been filed concurrently, the entire disclosure of the other application being incorporated herein by reference for all purposes:

• • Application No. 16 / 267,200 entitled "METHOD AND SYSTEM FOR MIXED REALITY INTERACTION WITH PERIPHERAL DEVICE" filed on 4 February 2019.

BACKGROUND

Peripherals can be used as an interface between humans and computers. Keyboards, computer mice, image scanners, speakers, microphones, web cameras, a head-mounted display (HMD), styli, pens, pointing devices and the like are some of the common peripherals. Some of these peripherals may interact to enhance usability. For example, an HMD may be operated with other peripherals (eg, a mouse, a keyboard, etc.) to facilitate a mixed reality (MR) experience or an augmented reality (AR) experience , For example, an HMD set up by a user may include cameras to capture real-time images of an interaction of the user with other peripheral devices (eg, a keyboard) in front of the user. By combining a real element (eg, a real hand or other body parts, a real keyboard, etc.) from the images with a virtual element (eg, a virtual keyboard, a virtual hand, etc.), a composite image may be created be generated. A composite image may be generated for each of the real-time images, where the composite image may be displayed to the user at the HMD instead of the real-time images.

As the HMD moves due to movement of the user, the composite image displayed in the HMD may be updated to track the change in the viewing angle of the user relative to the peripheral device. To optimize consistency and minimize nausea, the rendering of the virtual element and the real elements in the composite image should be synchronized with the movement of the HMD.

SUMMARY

Certain embodiments provide a method for providing mixed-reality interaction (MR interaction). The method includes capturing a first image of a user's real interaction with a real object at a first time and using a head-mounted display (HMD) camera of a user, measuring movement of the HMD relative to the real object between the user first and second timings, processing the first image based on measuring the movement of the HMD to generate a second image, generating a composite image of a virtual interaction involving the user based on the second image, and displaying the virtual interaction instead of real interaction for the user at the HMD based on the composite image at the second time.

In some embodiments, the first image is taken at a first time. The method further comprises measuring the movement of the HMD relative to the real object between the first time and the second time to obtain one or more measurement results, and processing the first image based on the one or more measurement results to obtain the second image to create.

In some embodiments, processing the first image based on the one or more measurement results to produce the second image comprises storing the first image in a frame buffer, shifting a constituent of the first image in the frame buffer based on the one or more a plurality of measurement results and generating the second image based on the shifted component of the first image.

In some embodiments, the frame buffer includes a first buffer area and a second buffer area. The data of the first image is divided between the first buffer memory area and the second buffer memory area. The second image is generated based on a portion of the data of the first image stored in the first buffer memory area. Moving the component of the first image in the frame buffer includes changing a location of the first buffer area within the frame buffer.

In some embodiments, the frame buffer includes a first buffer area and a second buffer area. The data of the first image may be divided between the first buffer memory area and the second buffer memory area. The second image may be generated based on a portion of the data of the first image stored in the first buffer memory area. The shifting of the constituent of the first image in the frame buffer may include changing locations of the data of the first image relative to a location of the first buffer area within the frame buffer.

In some embodiments, the one or more measurements include an angular velocity of the HMD and a direction of rotation of the HMD.

In some embodiments, shifting a component of the first image in the frame buffer based on the one or more measurement results includes determining a rotation angle based on the angular velocity, the direction of rotation, and a time difference between the first time and the second time, determining a displacement distance, and Displacement direction based on the angle of rotation, a first position of the HMD and a second position of the real object and a shifting of the component of the first image in the image buffer memory on the basis of the displacement distance and the displacement direction.

In some embodiments, the method further comprises determining the time difference based on an estimate of a time delay between the first time and the second time.

In some embodiments, the first image includes an image of a display device that displays a time-varying pattern. The method further comprises identifying the pattern in the first image and determining the first time based on the identified pattern.

In some embodiments, the first image includes a time varying light pattern emitted by the real object. The method further comprises identifying the light pattern in the first image and determining the first time based on the identified light pattern.

In some embodiments, the first time is determined based on a determination of the time at which a shutter of the camera of the HMD opens.

In some embodiments, the second time is determined based on a determination of the time at which the first image is stored in the frame buffer.

In some embodiments, the second position of the real object is determined based on information received from an optical sensor attached to the real object.

In some embodiments, the second position of the real object is determined based on information received from an optical sensor that receives light from a light emitter attached to the real object.

In some embodiments, the first position of the HMD is determined based on detecting directional light rays of multiple light emitters and predetermined positions of the plurality of light emitters.

In some embodiments, generating the composite image is based on determining an image position of the real object in the second image based on a position of the HMD, a position of the real object and the one or more measurement results, and a replacement of pixels corresponding to the real one Object at the image position in the second image correspond, through pixels of a virtual object, to produce the composite image.

In some embodiments, generating the composite image is based on determining a first image position of the real object in the second image based on a position of the HMD, a position of the real object, and the one or more measurement results, determining a second image position of a body part of the user based on the first image position and the second image, extracting pixels of the body part of the user at a second image position in the second image, and inserting the pixels of the body part of the user into the composite object involved in real interaction with the real object Image.

In some embodiments, the body part of the user includes at least a portion of a user's hand.

Certain embodiments specify a device. The apparatus includes a display device, a camera configured to capture a first image of a real user interaction with a real object at a first time, and an image processor coupled to the camera and the display device that is relatively capable of measuring movement of the device to the real object between the first time and a second time, for Processing the first image based on the measurement of the movement of the second image forming device, generating a composite image of a virtual interaction involving the user based on the second image and outputting the composite image to the display device to show the user the virtual interaction instead of the real interaction at the second time.

Certain embodiments specify a non-transient, computer-readable medium. The non-transitory computer-readable medium stores instructions that, when executed by a hardware processor, cause the hardware processor to receive from a camera of a head-mounted display (HMD) a first image of a real user interaction with a real object, the first image being from the user Camera was taken at a first time to measure a movement of the HMD relative to the real object between the first time and a second time to process the first image based on the measurement of the movement of the HMD to produce a second image a composite image of a virtual interaction involving the user based on the second image and providing the composite image to a display device to display the virtual interaction to the user at the second time instead of the real interaction.

Certain embodiments provide a method for providing mixed reality (MR) interaction. The method includes capturing a first image of a limb of a user and a real peripheral input device, wherein the limb of the user obscures at least a portion of the real peripheral input device, segmenting the first image into a first pixel region corresponding to the limb of the user second pixel region of the physical peripheral input device corresponds to generating a composite image from the first image based on rendering pixels representing a virtual peripheral input device instead of the second pixel region so that the virtual peripheral input device in the composite image spatially occupies the same location as the real peripheral input device appears and is at least partially obscured by the limb of the user, and displaying the composite image to the user.

In some embodiments, the composite image is displayed to the user at virtually the same time that the limb of the user obscures at least a portion of the physical peripheral input device.

In some embodiments, segmentation of the first image is based on a distribution of activities to multiple locations of the physical peripheral input device.

In some embodiments, segmenting the first image includes determining an historical distribution of the activities to the multiple locations of the physical peripheral input device, wherein the historical distribution of the activities is based on previous interactions between the user or a group of users with the physical peripheral input device one or more thresholds based on the historical distribution of the activities, comparing the pixels of the first image with the one or more thresholds, and determining the first pixel region corresponding to the limb of the user and the second pixel region corresponding to the physical peripheral input device based on the comparison results ,

In some embodiments, segmenting the first image includes determining a distribution of the probabilities that the user's limb is located at a respective one of the plurality of locations of the physical peripheral input device based on a statistical analysis of the historical distribution of the activities, identifying pixels, which are likely to be part of the user's limb, based on the distribution of the probabilities and determining the one or more thresholds based on one or more attributes of the identified pixels.

In some embodiments, the method further comprises determining a first threshold of the one or more thresholds corresponding to a first portion of the limb of the user, determining a second threshold of the one or more thresholds corresponding to a second portion of the limb of the user, and comparing the pixels of the first image having the first threshold and the second threshold to determine the first pixel region corresponding to the limb of the user.

In some embodiments, the method further comprises determining the first and / or second threshold based on ambient brightness and / or user identity.

In some embodiments, the distribution of activities includes a current distribution of the activities to the multiple locations of the real peripheral input device.

In some embodiments, the method further comprises receiving a sensed activity message from a sensor located at a first location of the physical peripheral input device, capturing a third image from the physical peripheral input device based on the message, selecting a third corresponding to the first location A pixel region and a fourth pixel region surrounding the first location, determining a first color component histogram for the third pixel region, determining a second color component histogram for the fourth pixel region, determining one or more thresholds based on the first histogram and the second Histogram and comparing pixels of the first image with the one or more thresholds to determine the first pixel region of the first image that corresponds to the limb of the user.

In some embodiments, the method further comprises identifying a second location of the real peripheral input device on which less than a predetermined number of previous activities occurred based on a historical activity distribution, identifying a fifth pixel area corresponding to the second location in the third image, determining a third color component histogram for the fifth pixel region, determining a second threshold value of the one or more threshold values based on the third histogram, and comparing the pixels of the first image with the second threshold value to determine the second pixel region of the first image that is the second pixel real keyboard corresponds.

In some embodiments, the method further comprises determining a change in the distribution of the activities over time, determining a region of interest based on the change, identifying pixels in the region of interest of the first image, and determining the limb of the user first pixel area from the identified pixels.

In some embodiments, the method further comprises capturing a second image of the real peripheral input device not obscured by a limb of the user, determining one or more thresholds based on the second image, transmitting a message to the user to set the limb of the user to a predetermined one Places the real peripheral input device, capturing a third image of the limb of the user at the predetermined locations of the real peripheral input device, adjusting the one or more thresholds based on pixels of the third image corresponding to the predetermined locations of the real keyboard, and comparing the pixels of the first image with the adjusted one or more thresholds to determine the first pixel region corresponding to the limb of the user.

In some embodiments, the distribution of the activities from multiple activity distributions is determined based on a model of the real peripheral input device and / or an application for which the user interacts with the real peripheral input device.

In some embodiments, the composite image is created based on replacement of pixels corresponding to the real environment in which the real peripheral input device is located by pixels representing a virtual environment in which the virtual peripheral input device resides.

In some embodiments, the composite image is created based on a luminance match of the first pixel region based on a luminance of the virtual environment.

In some embodiments, the composite image is created based on adding a virtual shadow that the user's limb casts in the virtual environment.

In some embodiments, the first image is captured by a camera of a head-mounted display (HMD). The composite image is displayed on the HMD.

In some embodiments, segmenting the first image into the first pixel region corresponding to the limb of the user comprises identifying two or more thresholds for the pixel values representing the limb of the user, where pixels of the first image are not within the two or more thresholds lie, be discarded.

Certain embodiments specify a device. The apparatus may include a display device, a camera adapted to capture a first image of a limb of a user and a real peripheral input device, wherein the limb of the user obscures at least a portion of the real peripheral input device, and an image processor coupled to the camera and the display device is formed, for obtaining the first image from the camera, for performing a segmentation of the first image into a first pixel region corresponding to the limb of the user, wherein a second pixel region of the real A peripheral input device for generating a composite image from the first image based on rendering pixels representing a virtual peripheral input device instead of the second pixel region, so that the virtual peripheral input device spatially appears in the composite image at the same position as the real peripheral input device the limb of the user is at least partially hidden, and outputting the composite image on the display device for display to the user.

Some embodiments specify a non-transitory computer-readable medium. The non-transitory computer readable medium stores instructions that, when executed by a hardware processor, cause the hardware processor to receive from a camera a first image of a limb of a user and a real peripheral input device, the limb of the user obscuring at least a portion of the physical peripheral input device, a segmentation the first image into a first pixel region corresponding to the limb of the user, wherein a second pixel region corresponds to the real peripheral input device to create a composite image from the first image based on rendering pixels representing a virtual peripheral input device instead of the second pixel region so that the peripheral virtual input device in the composite image spatially appears at the same position as the real peripheral input device and through the limb of the user s is at least partially obscured, and to provide the composite image to a display device for display to the user.

list of figures

• 1A und 1B zeigen eine typische Implementierung eines Systems, bei dem die offenbarten Techniken eingesetzt werden.
• 2 zeigt ein System zum Betreiben eines Peripheriegerätes.
• 3A - 3F veranschaulichen Beispiele für ein Erzeugen eines immersiven Mixed-Reality-Erlebnisses (MR-Erlebnisses) mit einem System nach 1A und 1B, gemäß einiger Ausführungsformen.
• 4A - 4D veranschaulichen Beispiele für ein Bildverzögerungskompensationskonzept gemäß einiger Ausführungsformen.
• 5A - 5F veranschaulichen Beispiele für Handsegmentierungskonzepte gemäß einiger Ausführungsformen.
• 6 veranschaulicht ein Beispiel für eine Bildverarbeitungsengine gemäß einiger Ausführungsformen.
• 7 zeigt ein Flussdiagramm zur Veranschaulichung eines Beispiels für ein Verfahren zum Bereitstellen eines Mixed-Reality-Erlebnisses (MR-Erlebnisses) gemäß einiger Ausführungsformen.
• 8 zeigt ein Flussdiagramm zur Veranschaulichung eines Beispiels eines Verfahrens zum Bereitstellen eines MR-Erlebnisses gemäß einiger Ausführungsformen.

In the detailed description, reference is made to the accompanying figures.

• 1A and 1B show a typical implementation of a system employing the disclosed techniques.
• 2 shows a system for operating a peripheral device.
• 3A - 3F illustrate examples of creating an immersive mixed reality (MR) experience with a system 1A and 1B , according to some embodiments.
• 4A - 4D illustrate examples of an image delay compensation concept according to some embodiments.
• 5A - 5F illustrate examples of hand segmentation concepts according to some embodiments.
• 6 illustrates an example of an image processing engine according to some embodiments.
• 7 FIG. 12 is a flowchart illustrating an example of a method for providing a mixed-reality (MR) experience in accordance with some embodiments. FIG.
• 8th FIG. 12 is a flowchart illustrating an example of a method of providing an MR experience according to some embodiments. FIG.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to peripherals in general, and more particularly to peripherals that enable interactive mixed reality (MR) or augmented reality (AR) experiences according to particular examples. In this document, Augmented Reality (AR) is understood to mean an interactive experience of a real environment. The real environment can be viewed directly by the user with his / her eyes (eg with the aid of AR glasses) or displayed with the aid of cameras which are aligned in the direction of the user's gaze. In AR, real-world objects are augmented with computer-generated perceptual information that sometimes appeals to multiple senses that include seeing, hearing, touching, feeling, and smelling. Mixed reality (MR), sometimes referred to as hybrid reality, can refer to the merging of real and virtual worlds. In a hybrid reality, a completely computer-generated environment can be represented in which images or representations of real objects can be integrated. A hybrid reality can bring about new environments and visualizations in which real and digital objects coexist and interact in real time, as well as generate new user interactions. Mixed Reality offers a mix of reality and virtual reality, and can incorporate augmented reality through immersive technology.

In the following description, various examples of peripheral devices that enable interactive MR and / or AR experiences will be discussed. The specific embodiments and details set forth herein are illustrative, which provides a thorough understanding of the embodiments. However, one of ordinary skill in the art will readily recognize that certain embodiments may be practiced or practiced without that every detail is disclosed. Furthermore, well-known features may be omitted or simplified to more clearly illustrate the novel features set forth in this specification.

Peripherals can be used as an interface between humans and computers. Some of the popular peripherals include keyboards, computer mice, image scanners, speakers, microphones, web cameras, a head-mounted display (HMD), styli, pens, pointing devices, and the like. Some of these peripherals can detect motion as user input and translate the motion into a computer input. For example, a portable peripheral device, such as a computer mouse, may sense its motion relative to a surface and translate the motion relative to the surface into corresponding movement of a cursor displayed on a graphical user interface (GUI). Further, the keyboard may detect a movement of a key and / or a jogging movement of a finger on the key and send a signal to a computer for allowing certain information represented by the key (eg, a letter, a symbol, a cursor movement, etc .) is shown.

Some of these peripherals can work together to optimize the user experience. For example, an HMD may be operated in conjunction with other peripherals (eg, a mouse, a keyboard, etc.) to form a Mixed Reality (MR) or Augmented Reality (AR) system. For example, a user-worn HMD may have cameras for capturing real-time images that show a real environment from the perspective of a user. To enable an immersive MR / AR experience, real-time images of a user's interaction with another peripheral device (eg, typing on a real keyboard) can be captured. To create a composite image, a first real element (eg, a real hand or another body part / limb of a user) shown in the images may be combined with a virtual element that has a second real element in it represents the real environment (eg a virtual keyboard). For example, the relative image positions of the first real element and the virtual element in the composite image may be determined based on the real positions of the first real element and the second real element relative to the HMD.

When the HMD moves due to a user's movement (eg, turning the head, nodding, the user's full body movement, etc.), the composite image displayed in the HMD may be updated to reflect the change in the user's view of the real world Mimic environment as the user moves to optimize the immersive interactive experience. For example, if a composite image is created by inserting a virtual keyboard (or other virtual peripherals) into the hand-held image, the image position of the virtual keyboard relative to the image may be shifted to accommodate a new position / orientation of the HMD. Further, if a composite image is created by removing hand pixels from the image and pasting them into a virtual image containing the virtual keyboard, the location from which the hand pixels are extracted may also be shifted to the new position / orientation HMD.

In both ways of creating a composite image, the shift of the virtual element's image position or the displacement of the real element's extraction location may be based on the assumption that the image reflects the current position / orientation of the HMD. However, such an assumption is not necessarily valid. For example, if an image is captured at the beginning of a movement of the HMD, a significant amount of time (eg, 0.1 second) may have elapsed by that time due to various factors such as video camera latency, frame and processing delay, data transfer delay, and so forth where the image is available to extract the real element or for combination with the virtual element to create the composite image. Therefore, the image can only reflect an older position / orientation of the HMD. The latency in building the image, which reflects the final position / orientation of the HMD, is recognizable to the user and may affect the user experience. For example, extracting pixels from the image at an image position that has been updated based on the last position / orientation of the HMD may not yield the expected real element (eg, the hand). Additionally, inserting the virtual element into the video image at the updated image position in the composite image may result in misplacement of the real and virtual elements relative to each other. For example, at some point in time, the HMD may display and interact with a hand on a virtual keyboard. When the HMD moves, the HMD may display the hand in a position lagging relative to the keyboard due to the aforementioned erroneous location, so that the hand is aligned with the virtual keyboard only in stationary HMD, but not in a moving HMD. This can lead to an inconsistent interactive experience or even nausea.

In addition, the aforementioned AR / MR system may generate a composite image as indicated above by extracting hand pixels from the image and placing the hand pixels in a virtual image containing the virtual keyboard (or other virtual peripheral device). To provide an interactive AR / MR experience, the composite image may be displayed to the user at the HMD when the user interacts with a real keyboard represented by the virtual keyboard. The quality of the interactive experience may depend on the precise segmentation of the hand in an image that depicts the interaction of the hand with the real keyboard. Segmentation of a hand may be based on comparing the image pixels with one or more thresholds and classifying the pixels based on the comparison results as belonging to the hand or peripheral device (s). The accuracy of the segmentation of the hand may be due to various factors such. As skin tones, illumination changes, etc. are affected. A segmentation concept that uses one or more fixed thresholds when comparing the pixels can only work well for a small range of skin tones and a small range of environments. To enable the use of the AR / MR system in a wider range of environments, which can improve the utility of the system and enhance the user experience, a more robust approach is needed where the handpixels cover a wider range of skin tones and a wider range environments can be more accurately distinguished from the background pixels.

Conceptual overview of certain embodiments

Examples of the present disclosure relate to methods and systems for providing immersive AR / MR experiences, and can solve some of the objects described above. Some embodiments provide a method for providing mixed reality (MR) interaction. The method includes: capturing a first image of a user's real interaction with a peripheral device at a first time using a user's head-mounted display (HMD) camera, measuring movement of the HMD relative to the peripheral device between the first time and a second second timing, processing the first image based on measuring the movement of the HMD to produce a second image; Creating a composite image based on the second image to represent a virtual interaction involving the user and displaying the virtual interaction instead of the real interaction at the HMD and based on the composite image for the user at the second time. The virtual interaction may, for example, be a virtual interaction between a body part of the user and a virtual object representing the peripheral device (eg a virtual keyboard), a virtual interaction between a virtual object representing the body part of the user (eg a virtual hand ) and the peripheral device, etc.

Specifically, the first image of the user's real interaction (eg, with the hands or other body parts) may be captured with the peripheral device at the first time before the movement of the HMD begins, due to the processing delay the first image is only at the second time is available for generating a composite image when the HMD has already moved a certain distance or has been rotated by a certain angle in a certain direction. Therefore, the first image does not reflect the user's current view of the interaction with the peripheral device at the second time. Combining a real element of the real-time, initially-recorded interaction with a virtual element at a location of the real interaction being considered by the user at the second time may result in a placement error between the real element and the virtual element. The HMD, to give an illustrative example, may capture an image of a real interaction between a user's fingers and a keyboard and create a composite image representing a virtual interaction between the user's fingers and a virtual keyboard. When the HMD is moving, it may happen that the HMD, due to the above-mentioned placement error, lifts or offsets the user's fingers from the virtual keyboard so that the fingers are correctly aligned with the virtual keyboard only at steady state HMD. This can lead to an inconsistent interactive experience (eg reduced efficiency when typing on a virtual keyboard) or even nausea.

To reduce misplacement, the first image may be further processed based on a measurement of the movement of the HMD to produce a second image, where the second image may be used to construct the composite image. Processing to generate the second image may include shifting a component of the first image. The second image may represent a view of the interaction taken by the camera of the HMD at the second time. In some examples, the shift of the component of the first image may occur on a frame buffer in which the first image is stored. The frame buffer may be in a display area and a Non-display area be split. Image data stored in the display area is maintained for generating a composite image and / or for display, while image data stored in the non-display area is not held for generation of a composite image or for display. The storage location of the display area in the frame buffer may be shifted to select a different portion of the image data to be displayed, whereby an effect can be achieved in which the component of the image data is shifted within or relative to the composite image. The second image may be formed by the component of the first image data located within the display area. In some examples, the display area and the non-display area may also be static, and the image data for generating the displacement effect within the image buffer memory may be shifted relative to the static display area and non-display area.

In some examples, the value of the displacement of the component of the first image may be determined based on a measurement of the movement of the HMD as well as the positions of the HMD and peripheral device at the second time. The measurement of the movement can z. B. include a rotation angle of the HMD. To determine the angle of rotation, an angular velocity and direction of rotation and a time interval between the first time and the second time can be determined. The angular velocity and direction can z. B. be determined using an inertial measurement unit (IMU) of the HMD. The elapsed time between the first time and the second time may also be determined on the basis of a delay by the further processing of the first image on the HMD. There are various ways of determining the delay by further processing. For example, as part of a calibration process, the camera of the HMD may be used to capture an image of an object that provides a time indication. The object may include, for example, a clock, a display device displaying a time-varying image (eg, a bar code), a time-varying pattern emitted by a peripheral device, and so forth. The image may be analyzed to identify a time-indicating pattern (e.g., numbers indicated by the watch, positions of clock hands, bar code patterns, etc.) and time information may be derived from the identified pattern. The peripheral device may include, for example, a keyboard, a mouse, etc. The time indicated by the object can be synchronized with the timing of the HMD and used as the recording time of the image. The display time can be recorded on the HMD when the image is displayed. The difference between the display time and the recording time may represent the processing delay. Since the delay through further processing in a known system is generally quite stable, the aforementioned calibration method can be used to determine the processing delay on other subsequent images on the HMD.

In addition, the positions of the HMD and the peripheral device in real space can be determined at the second time. Between a position of the peripheral device and a position of the HMD, a first position vector can be set. In addition, a second location vector of the peripheral device related to the HMD can be set by rotating the first location vector by the rotation angle. The second location vector may be used to represent an uncorrected position of the image of the peripheral device in the first image. The second location vector may also represent the reciprocal of the shift required to effect the correction. The amount of displacement can be determined by determining a displacement vector between the uncorrected position of the peripheral device image and the peripheral device's position and scaling the displacement vector back to the display coordinates. For example, scaling may be based on a matrix expression that projects world coordinates onto display coordinates.

There are several ways to determine the position of the HMD and the peripheral device. In one example, one or more sensor capsules may be attached to the peripheral device to detect directional light rays from light emitters (eg, lighthouses) located at different predetermined positions in space. The sensor capsules can also take distance measurements based, for example, on the transit time of the directional light. Based on the direction and transit time information as well as the known positions of the lighthouses, the position of the peripheral device can be determined. In another example, the peripheral device may include light emitters for emitting directed light rays that may be detected by a plurality of light sensors at different predetermined positions in space. The light sensors can also make distance measurements. The position of the peripheral device can also be determined on the basis of the distance measurements and the directions of the light beams detected by the light sensors as well as the known positions of the light sensors. To determine the position of the peripheral device, similar techniques may be used.

The second image generated by processing the first image based on the measurement of the motion / rotation of the HMD may be used in various ways to produce the composite image. In one example, a virtual element (eg, the virtual keyboard) may be placed in the second image at the image position predicted for the second time of interaction. In another example, from the second image, a real element (eg, the user's hand / finger or another body part of the user) may be extracted at the position predicted for the second time of interaction, where the real element is in a virtual Image that also has a virtual element (eg, the virtual keyboard) at the position predicted for the second time of interaction. In both cases, an interaction between the hand and the virtual keyboard can be presented in the composite image. The real element of the interaction (eg, the user's hand) is more aligned with the virtual element of the interaction (eg, the virtual keyboard), thereby improving the consistency and quality of the interactive experience (especially in interactions involving coordination between visual and physical stimuli, such as typing) and the likelihood of a nausea attack are reduced.

In addition, some embodiments provide a method for providing interaction with a mixed reality (MR). The method may include taking a first image of a real interaction between a body part of a user and a real object using a user's head-mounted display (HMD) camera, segmenting the first image into a body part of the user using the HMD corresponding first pixel area and a second pixel area corresponding to the real object, wherein the segmenting is based on a distribution of the real interactions on a plurality of locations of the real object, generating a composite image of a virtual interaction involving the user from the first pixel area or the second pixel area and displaying the composite image of the virtual interaction instead of the real interaction on the HMD to the user.

Concretely, the first image may be segmented to be in an image of a real interaction between a user and a real object between the body part of the user (eg, a part of a hand, the fingers, etc.) and the real part operated with the body part Object (eg, a peripheral such as a keyboard, a mouse, etc.). The segmentation method can be used to extract pixels from the image that are assumed to be part of a body part. Based on the distribution of interactions, one or more thresholds may be configured to include different pixel attributes (eg, the colors of the pixels, the luminance of the pixels, etc.). Based on the comparison results, it can be determined whether the pixels are likely to belong to a part of the user's body or to a part of the real object, and a subset of the pixels can be extracted based on the determination. The extracted pixels may, for example, comprise a body part of the user or pixels belonging to a real object. In some examples, the extracted pixels may be arranged in a composite image that represents a virtual environment for representing the virtual interaction. In some examples, the composite image may be derived from the first image by replacing the extracted pixels with pixels of a virtual object (eg, virtual hand / finger). In both cases, to allow the user an MR experience, the virtual interaction should be displayed instead of the real interaction.

As stated above, the segmentation of the first image may be based on a distribution of the interactions. The distribution can be determined based on historical information and / or current information. Concretely, the historical information may be in the form of a heat map or a probability map, based on a statistical analysis of past interaction activities. The prior interaction activities may be those between the user and the peripheral device, between other users and other instances of the peripheral device and / or a similar peripheral device, and so on. The heat map may specify a frequency distribution of interaction activities across multiple locations of the peripheral device. The statistical analysis may include, for example, determining a likelihood that an interaction activity will occur at a location of the peripheral device based on a number of occurrences and the amount of time over which the number is determined. Locations of the peripheral device in the thermal map may be mapped to pixels of an image from user interaction with the peripheral device. On the basis of the heat map or the probability map, a probability can be determined with which corresponding pixels can be found at a respective one of the multiple locations of the body part of the user (eg hand, fingers, etc.). The heat map probability information may be used to specify one or more thresholds related to particular attributes of a pixel (eg, luminance, color, etc.). For example, values of attributes (eg, luminance, color, etc.) of pixels in regions having a high probability of finding hand pixels may be thresholds for a determination of whether one pixel is a hand pixel, be adapted in other images.

The threshold values can also be adapted to different operating conditions. In some examples, the thresholds for comparing pixels may be increased for the locations where the likelihood of finding hand / finger pixels is low, while the thresholds for comparing pixels may be lowered for the locations where the likelihood is is high for finding hand / finger pixels. The thresholds may also be adjusted based on the ambient light of the operating environment. For example, the luminance threshold for hand / finger pixels may be corrected upward at high ambient light levels and the luminance threshold at low levels of ambient light intensity. In some examples, multiple thresholds may be adjusted, e.g. For example, different thresholds may be adjusted to extract pixels from fingernails, finger segments, palm, wrist, etc., each of which may have different colors. In addition, the thresholds can also be adapted to different users. For example, some users may have painted fingernails, and the thresholds for extracting pixels of the fingernails of these users may be adjusted to account for the colors of the painted fingernails. In another example, some users may have tattoos and / or garments that completely or partially cover their hands, etc., and the thresholds for extracting hand pixels for these users may be adjusted to accommodate the covered portions of their hands. For a user, a hand model may be defined that includes a distribution of thresholds for different areas of a hand of that user.

In some examples, the historical information may be used to calculate, for example, the temporal variation of the same pixel that may be used to improve computational efficiency when performing a multi-level pixel comparison process. For example, a continuous calculation of the change in the overall intensity (or color change) from one image to another may allow for a visual map of the image changing parts that can be used to identify one or more boundaries of the pixels that are moving Objects such as a user's hand or other body parts that interact with the peripheral device represent. For example, the bounds may define pixel areas that are likely to represent a user's hand or other body parts, with only the pixels associated with those areas being compared to the threshold (s) to determine if the pixels belong to the body part of the user. With such arrangements, the total number of pixels to be compared in the segmentation procedure with the threshold (s) can be reduced, which can improve the efficiency of the calculation.

In some examples, the action distribution of the interactions may also be determined based on current information. For example, the peripheral device may have multiple motion sensors (eg, keyboard keys) at multiple locations. A camera (which may be part of the HMD or a stand-alone device) may capture an image of the multiple locations of the peripheral device. A sensor card can map different pixels of the image to the multiple locations where the multiple motion sensors are located. When the motion sensors detect the interaction activities at one or more locations of the plurality of locations (eg, keystrokes, keystrokes, touching / non-touching gestures, etc.), the motion sensors may cause the camera to take a picture of the multiple locations of the peripheral device, where the image also contains interaction activities. From the image and on the basis of the sensor card, a first region of the peripheral device, which contains the motion sensors detecting the interaction activities, and a second region surrounding the first region can be determined. The first area may correspond to a portion of a peripheral device (eg, keyboard keys, buttons, touch pads, etc.) where the interaction activities take place, and may include pixels of the user's hand / fingers (or other body parts) participating in the interaction activities are. The second area may include pixels of the peripheral device or other background object. For the first region, a first histogram of different color components (eg relative intensities of the respective color components) can be determined. For the second region, a second histogram of such components may be further determined. The thresholds for determining whether a pixel belongs to the user's hands / fingers can be determined based on the first and second histograms. For example, to determine whether a pixel belongs to a user, a threshold value (eg, a borderline probability that the pixel belongs to the hand) may be determined for each color component, the threshold being based on a weighted average of the intensities of that color component in FIG first and second histogram is determined. For other pictures For example, to determine whether the pixel belongs to the user's hand, each color component of a pixel may be compared to the corresponding threshold.

In some examples, a combination of historical and current information may be used to set a second background detection threshold. For example, images of peripheral device locations where there are very few or no interaction activities can be periodically captured and a third color component histogram determined for those locations. Based on the third histogram, a second threshold for the background detection can be determined. The second background detection threshold (eg, a borderline probability that a pixel belongs to the background) may be at the first threshold for hand / finger detection (eg, a borderline probability that a pixel belongs to the hand / fingers) be combined in different ways. In one example, the second background detection threshold may be used in the first step of the verification. If the pixel is very close to the second threshold and the result is ambiguous, the pixel may be compared to the first threshold for hand / finger detection. In another example, a combined threshold may be determined by averaging the first and second thresholds, and the pixels of the image may be compared to the combined threshold to identify the hand pixels.

The determination of the thresholds may be part of a calibration process based on image substitution. For example, an image of a peripheral device (eg, a keyboard) not covered by the user's hand or fingers can be captured. The pixels of the image may be used as a reference / starting point in determining the thresholds for the distinction between the hand / finger pixels and the pixels of the peripheral device. As part of the calibration process, a user may place his / her fingers on the peripheral device as described, whereupon a second image of the finger-covered peripheral device is captured. Based on the known pixel positions of the hand / fingers as well as the known pixel positions of the peripheral device that the hand / fingers do not have, the thresholds can be adjusted from the starting point to, for example, minimize the likelihood of correctly distinguishing between the pixels of the peripheral device and the pixels To maximize the user's hand / finger. The acquisition of images from the peripheral device may also be periodic to obtain an updated image for the determination of the reference thresholds, taking into account a change in operating conditions (eg, change in ambient brightness, inactivity of the peripheral device, etc.).

After identifying the pixels (eg, hand, fingers, etc.) associated with the body part of the user, a composite image can be created based on the identification of the pixels. For example, the extracted pixels (eg, a hand, finger, etc.) may be inserted into a virtual environment rendered by the composite image to depict the virtual interaction. The extracted pixels and / or the virtual environment may also be modified to be matched. For example, the luminance of the extracted pixels can be adapted to the lighting conditions of the virtual environment. In another example, virtual shadows may also be added to the composite image that are thrown by hand (represented by the extracted pixels) onto a virtual peripheral device. In some examples, additional pixels of the virtual environment may also be replaced by hand / finger pixels to close gaps in the extraction of the hand / finger pixels.

By adapting hand / background detection based on a distribution of actions on the peripheral, the hand segmentation scheme may be less sensitive to other factors that may affect the accuracy of hand detection, such as skin tone, light intensity, etc. For example, suppose a user has a darker skin tone and / or uses a keyboard in a relatively dark environment, and an image is taken of the use of the keyboard by the user. Due to the darker skin tone of the user and / or the low ambient light of the environment, the hand pixels and the keyboard pixels in the image may have less differences than a user with a lighter skin tone using a dark keyboard in bright ambient light, thereby degrading the accuracy of hand segmentation can. In addition, the hand segmentation may be disturbed in low ambient light. By adapting the detection thresholds based on a historical or instantaneous distribution of actions on the keyboard, the hand detection can be based on an additional set of information, which can mitigate the effects of skin tone and light intensity. This can provide a more robust segmentation scheme that can accurately identify hand pixels for a wider range of skin tones and in a wider range of environments.

Typical system environment for specific examples

1A and 1B show a typical implementation of a system 100 in which the disclosed techniques can be used. The system 100 can the computer 110 , the ad 120 and a plurality of peripheral devices, such as the peripheral device 130 (eg "Computer mouse 130 "), The peripheral device 140 (eg "keyboard 140 "), The peripheral device 150 (eg "Microphone 150 ") And the peripheral device 160 (eg "Head-Mounted Display (HMD) 160 "). As one of ordinary skill in the art can see, the computer mouse 130 , the keyboard 140 and the microphone 150 of the system 100 to control aspects of the computer 110 , the display 120 and the HMD 160 be educated. computer 110 may be referred to as a "host computer" or as a "host computing device". Each of the peripherals 130 - 160 may include one or more sensors for detecting an input action (or input signals, such as audio signals) of a user and corresponding sensor data (eg, sensor data representing a travel distance of the computer mouse 130 represent, sensor data, pressing a key on the keyboard 140 show audio data to a microphone 150 to be detected, etc.). The peripherals 130 - 150 can transfer the sensor data to the computer 110 direct one or more operations on the computer 110 to control. For example, the computer 110 based on one of the computer mouse 130 captured motion from the user to one at the display 120 recognize selected audio file selection and play the selected audio file. In addition, the HMD 160 as in 1B displayed images (eg the camera image 170 ) an interaction of the user with a mouse 130 and / or a keyboard 140 capture and display the images of the user's interaction with the user. As described in more detail below, the HMD 160 real elements from the camera image 170 (eg the hand 180 the user, the keyboard 140 , the mouse 130 , etc.) combine with virtual elements to enable an immersive augmented reality (AR) / mixed reality (MR) experience.

Both the computer 110 as well as the HMD 160 may comprise a machine-readable medium (not shown) for storing computer code such. B. keyboard driver software and the like is formed, wherein the computer code from a processor of the computer 110 can be run to using the peripherals 130 . 140 and or 150 on the control of the computer 110 act. The various examples described herein generally refer to a peripheral device 120 in the form of a display, a peripheral device 130 in the form of a computer mouse, a peripheral device 140 in the form of a keyboard and a peripheral device 150 in the form of a microphone; However, it should be clear that the peripherals 130 - 150 may be in the form of any input / output device (I / O), user interface device, control device, input device, or the like. At the peripherals 130 - 150 It may also be, for example, a remote control device, a wearable device (eg, smartwatch, wristband, glasses), a smartphone, or the like, which inputs and outputs via a peripheral device 120 enable the displayed graphical user interface.

At the computer 110 it is usually a desktop or laptop computer device. However, it should be understood that any suitable computing device that further includes a tablet computer, a smartphone, etc. may constitute the host computing device. The computer 110 can also be part of the HMD 160 be. The HMD 160 may include a virtual or augmented reality interface (eg, with 2D or 3D displays), a holographic interface, or the like. One of ordinary skill in the art will recognize many variations, modifications and alternative embodiments thereof.

2 shows a system for operating a peripheral device (eg, one or more of the peripheral devices 130 - 160 ) according to certain embodiments. The system 200 has (a) processor (s) 210 , a storage arrangement 220 , an energy management system 230 , a communication system 240 and an input acquisition module 250 on. Each of the system blocks 220 - 250 can work with the processor (s) 210 be electrically connected (eg via a bus system). The system 200 may also include other systems that are not shown or explained in terms of a clear presentation of the novel features described herein in this document. The system blocks 220 - 250 may be implemented in the form of separate modules, alternatively, more than one system block may be implemented within a single device. In some examples, the system blocks may be 220 - 250 be implemented at a peripheral device. In some examples, the system blocks may be 220 - 250 implemented as a multi-device system comprising a peripheral device and a dongle. The system blocks 220 - 250 may be configured to implement the offset estimation technique to be described below.

In some examples, the processor (s) is / are pointing 210 One or more microprocessors (μCs) and may be used to control the operation of the system 200 be educated. Alternatively, the processor (s) may 210 as for one One of ordinary skill in the art will recognize one or more microcontrollers (MCUs), digital signal processors (DSPs), or the like having supporting hardware and / or firmware (eg, memory, programmable I / Os, etc.). The processor (s) 210 may have some or all aspects of operating the peripherals 130 - 160 control (eg system block 220 - 250 ). Alternatively or additionally, some of the system blocks may be 220 - 250 an additional specialized processor associated with the processor 210 can work together. One of ordinary skill in the art will recognize many variations, modifications and alternative embodiments.

The memory arrangement 220 may be configured to store information representing one or more configurations for operation of the peripherals 130 - 150 affect. As further explained below, one or more of the configurations for operation of the peripherals 130 - 150 adjusting features of the peripherals 130 - 150 include, but are not limited to, a computer mouse scrolling speed, a computer mouse motion sensor sensitivity, keyboard key-mapping assignment, microphone sensitivity, and the like. The memory arrangement 220 As described in more detail below, it may also store other configuration information necessary for communication with the peripherals 130 - 150 be used.

In addition, the memory arrangement 220 store one or more software programs for execution by processors (eg, the processor (s)) 210 ). It should be noted that "software" may refer to instruction sequences that the system 200 when executed, cause the processing unit (s) (eg, processors, processing devices, etc.) to perform certain function of software programs. The instructions may be stored as firmware residing in a read-only memory (ROM) and / or as applications stored in a media memory which may be read into memory for processing by processing devices. Software may be implemented as a single program or as a bundle of individual programs, and may be stored in nonvolatile memory and, during program execution, copied in whole or in part into volatile memory.

The energy management system 230 can handle power distribution, charging, energy efficiency and the like for the peripherals 130 - 150 be educated. In some embodiments, the energy management system 230 a battery (not shown), a USB based charging system for the battery (not shown), and energy management devices (eg, low dropout voltage regulator - not shown). In certain embodiments, those may be performed by the energy management system 230 Available Features in the Processor (s) 210 be integrated. The energy source may be a replaceable battery, a rechargeable energy storage device (eg, a supercapacitor, a lithium polymer accumulator, NiMH, NiCd), or a wired power supply. The charging system can be an extra cable (especially for charging) or use a USB port to charge the battery.

The communication system 240 may be to allow for wireless or wired communication between, for example, the processors 210 and one or more of the peripherals 130 - 150 be educated. The communication system 240 may be Radio Frequency (RF), Bluetooth®, BLE, WiFi, Infrared (IR), ZigBee®, Logitech Unifying®, or any other suitable communications technology for communicating with other computing devices and / or peripherals. The communication system 240 can also have a hardwired connection with the processors 210 provide. The hardwired connection can be a serial connection, such as a serial connection. Universal Serial Bus (USB), FireWire®, DisplayPort®, High-Definition Multimedia Interface (HDMI), etc.

The input capture module 250 may involve capturing an interaction of a user with the peripherals 130 - 150 control input elements. The input capture module 250 For example, user input may be based on sensor data from the computer mouse 130 to capture. In some embodiments, the input acquisition module 250 with the memory arrangement 220 work together to input data for the processors 210 based on the of the communication system 240 to generate received sensor data. The input capture module 250 for example, one of a mouse pointer on the display 120 distance covered on the basis of in the memory arrangement 220 stored scrolling speed information and sensor data of the computer mouse 130 calculate and route information to the processors 210 (or a renderer) to play mouse movement on the display 120 to hand over.

Although certain systems can not be specifically explained, they should be apparent to one of ordinary skill in the art as part of the system 200 be considered. The system 200 For example, a bus system for transmitting Have energy and / or data from and to its various systems.

It should be noted that the system 200 the explanation serves, whereby variations and modifications are possible. The system 200 can not have other functions explicitly described here. It is also noted that even if the system 200 With reference to particular blocks, these blocks are defined for ease of illustration and are not intended to imply a particular physical arrangement of components. In addition, the blocks need not correspond to any specific material components. The blocks may be configured to perform various functions, e.g. By programming a processor or providing a suitable control circuit, wherein various blocks may or may not be reconfigured depending on how they were originally formed.

Examples of the disclosed techniques may be implemented in various devices including electronic devices implemented using any combination of circuitry and software. Also, aspects and / or parts of the system 200 Depending on the design requirement, combined with other subsystems or operated with their help. For example, the input capture module 250 and / or the memory arrangement 220 within the processor (s) 210 instead of acting as a separate entity. In addition, the inventive concepts described in this document may be applied to various peripheral devices and are not limited to computer mice, keyboards, or microphones. The system 200 can be used with any of the embodiments explicitly, cognitively, or implicitly described in the embodiments (eg, as will be apparent to one of ordinary skill in the art for a particular peripheral device). The foregoing embodiments are not intended to be limiting, and numerous applications and possibilities will be apparent to one of ordinary skill in the art in light of this disclosure.

Examples of Generating Immersive Mixed Reality Experiences

3A - 3F illustrate examples of mixed-reality immersive experiences (MR experiences) using the system 100 from 1A and 1B were generated. How out 3A and 1B Obviously, the HMD 160 a camera picture 170 from the interaction of a user with, for example, a keyboard 140 take up. Instead of the user the camera image 170 to show the HMD 160 to create a composite image, some of the real elements (for example, the user's hand 180 ) of the camera image 170 with a virtual element 302 , such as a virtual keyboard, combine and show the composite image to the user. To create an immersive experience, a composite image can also have the same interaction as in the camera image 170 However, the interaction takes place between the hands 180 the user and the virtual keyboard 302 , While on the camera picture 170 for example one with several fingers of the hand 180 In the composite image, the same typing action may be the same in the composite image 180 but the typing action is instead of typing on the keyboard 140 on the virtual keyboard 302 is performed. While various examples of interactions between a user's hand and a keyboard are described in this disclosure, these examples are not intended to be limiting, and the techniques disclosed may be based on interactions of a user involving other body parts (eg, fingers, arms, etc.). Legs, feet, etc.) and other non-peripheral types of real objects involved.

3A also shows various ways of generating the composite image for generating an immersive MR experience. How out 3A You can see a composite picture 310 be generated by the camera image 170 in place of the keyboard 140 a virtual keyboard 312 is used, so the hand 180 the user as on the virtual keyboard 312 instead of the keyboard 140 is shown typing. In the composite picture 310 are all other real elements, including z. B. the display 120 , the mouse 130 , the user hand 180 , etc., continue to be included. In addition, a composite picture 320 also be generated by taking pictures of the user's hand 180 from the camera image 170 extracted and inserted into a virtual image representing a series of virtual elements, including, for example, a virtual flower 322 and virtual piano keys 324 , The user hand 180 may be at an image position of the composite image 320 be placed, the image position of the keyboard 140 in the camera picture 170 equivalent.

To enhance the immersive interactive experience, generating the composite images 310 and 320 also to a position and orientation of the HMD 160 whereby an effect is generated in which the real elements and the virtual elements are viewed from another position and / or at a different angle. For example, if the user rotates his head to the right (indicated by the character "A"), the user may Content of the ad in the HMD 160 move to the left (indicated by the sign "B"). At the in 3A illustrated composite image 310 Both the real elements (eg display 120 , Mouse 130 , User hand 180 , etc.) as well as the virtual elements (eg virtual keyboard 312 ) as indicated by the label "B" to the left. In addition, moving in the composite image 320 both the real elements (eg user hand 180 ) as well as the virtual elements (eg virtual piano keys 312 and virtual flower 322 ) as indicated by the label "B" to the left.

The HMD 160 may be the shift of the content of the ad in the composite images 310 and 320 based on tracking a position of a reference object relative to the HMD 160 make. The reference object may be a peripheral such as the keyboard 140 act. 3C shows examples of the tracking process. As in the left part of 3C shown is the sight line of a camera of the HMD 160 (represented by arrow 330 ) to a reference object 340 Aligned with a tracking position 345 of the reference object 340 in the middle of the camera image 170 located. As described below, the tracking position 345 be used for determining where in a composite image a virtual element (eg the virtual keyboard 312 ) or a real element (eg the user hand 180 ) are to be arranged. The tracking position allows the system to reference the object 340 relative to the HMD 160 follow. In addition, arises at the image position 350 in the camera picture 170 also an image of the reference object 340 , where the picture position 350 and the tracking position 345 are identical. In some examples, the image position 350 for example, based on a segmentation process for identifying and extracting pixels of the reference object 340 from the camera image 170 be determined.

As described below, the position coordinates of the HMD 160 and the reference object 340 in real space 360 (relative to the xyz coordinate system) as well as the orientation of the HMD 160 in real space 360 determined using various position and orientation sensing techniques. As in the right part of 3C is the line of sight of the camera of the HMD 160 with a rotation of the HMD 160 clockwise to the right at an angle α to the reference object 340 arranged. At an angle α assumed to be small, the HMD 160 the distance between the HMD 160 and the object 340 (denoted by "d") based on the position coordinates of the HMD 160 and the reference object 340 and a probable shift (labeled "s") of the tracking position 345 calculate on the basis of the following equation: $$\text{<figure-callout id="345" label="Postpurchase position shift" filenames="DE102019107103A1\_0008.png,DE102019107103A1\_0009.png" state="{{state}}">Postpurchase position shift</figure-callout> 345}\mathrm{=}\text{d}\times \text{sin}\left(\mathrm{\alpha }\right)$$

In addition, the camera of the HMD 160 also an updated camera picture 170 record when the HMD 160 located in the new orientation, taking the picture position 350 of the reference object 340 may have the same degree of shift as the tracking position 345 so the tracking position 345 and the picture position 350 stay aligned with each other.

As described above, the tracking position 345 be used for a determination where a virtual element (eg a virtual keyboard 312 ) or a real element (eg a user hand 180 ) in a composite image. In 3D Here are some examples of virtual and real element layout when creating composite images. As in the left part of 3D shown, the tracking position 345 the keyboard 140 in the camera picture 170 be identified, the composite image 310 by replacing the pixels located at the tracking position 345 keyboard 140 with pixels of the virtual keyboard 312 can be created. While the tracking position 345 due to the rotation of the HMD 160 moved to the left (indicated by "B"), can be viewed in the camera image 170 also all other real elements (eg display 120 , Mouse 130 , Keyboard 140 , User hand 180 , etc.) are shifted to the left when the camera image 170 is recorded when the HMD 160 after the rotation in the new orientation relative to the keyboard 140 located. In addition, the user's hand 180 as in the right part of 3D is shown at the tracking position 345 of the camera image 170 be extracted based on the assumption that the user's hand 180 in the same place as the keyboard 140 located. To create a composite image 320 For example, the extracted image of the hand can be inserted into a virtual image in which the virtual elements (e.g., virtual piano keys 324 , virtual flower 322 , etc.) of the shift of the tracking position 345 are shifted to the left accordingly. In both cases, moving is a virtual element of an interaction (such as a virtual keyboard 312 , virtual piano keys 324 , etc.) on a real element of the interaction (eg user hand 180 ), allowing for accurate interaction and enhancing the immersive interactive experience.

As described above, in order for the movement of the HMD to hang 160 reflects, in one composite image the alignment of the virtual element of interaction at the real element of interaction during a movement of the HMD 160 from that also the contents of the camera picture 170 (from which the real element is extracted or into which the virtual element is inserted). However, this is not necessarily the case. Due to processing delays (e.g., camera image buildup delay, system processing delay, etc.), it may happen that the camera image used to create composite images 170 not the current orientation / position of the HMD 160 relative to the reference object 340 reflects. 3E and 3F illustrate the effect that occurs when creating a composite image is an obsolete camera image 170 is used. As in 3E shown, the HMD 160 after being rotated by the angle α, the tracking position 345 of the reference object 340 as described above, based on the position coordinates of the HMD 160 and the reference object 340 and the detected rotational angle α move. Due to the processing delay, the actual image position remains 350 of the reference object 340 (eg keyboard 140 ) in the camera image 170 but the same as before the turn. This is the tracking position 345 no longer at the picture position 345 aligned.

3F illustrates the effect of an offset between tracking position 345 and picture position 345 when creating the composite images 310 and 320 , As in the left part of 3F Although this was the tracking position 345 of the reference object 340 moved to the left, the camera image 170 but not updated, so the real elements of the camera image 170 not moved to the left. If the virtual keyboard 312 at the postponed tracking position 345 is inserted, is the virtual keyboard 312 to the user hand 180 added. In addition, in the camera image 170 a part of the image of the keyboard 140 stand still and not through the virtual keyboard 312 be replaced. The representation of the keyboard in the composite image 310 as well as the relative position of the hand 180 the user's relative to the keyboard in the composite image can be very confusing to the user and severely affect the immersive interactive experience. Besides, as in the right part of 3F is shown, pixels from the camera image 170 at the offset tracking position 345 in anticipation of that extracts the extracted pixels hands 180 represent the user. The virtual elements in the virtual image (eg virtual flower 322 , virtual piano keys 324 , etc.) are based on the tracking position 345 also moved to the left. Because the camera image 170 however, if not updated, the extracted pixels may be the hand 180 of the user only partially (or not at all) as well as some other artifacts. Inserting the extracted pixels into the composite image 320 It also creates confusing display effects for the user and significantly degrades the immersive interactive experience.

Embodiments for image delay compensation

4A - 4D illustrate examples of an image delay compensation scheme for minimizing the delay caused by the processing delay of the camera image 170 conditional offset between the tracking position 345 and the picture position 350 , As in the left part of 4A the content of the camera image can be displayed 170 , which is a picture of the reference object 340 at the picture position 350 includes, relative to the tracking position 345 be moved so that the picture position 350 and the tracking position 345 can be aligned. By moving the contents of the camera image 170 the effect is similar to taking an updated camera image 170 through the HMD 160 at the new orientation relative to the reference object 340 after a rotation.

The shift distance of the camera image 170 can based on the position coordinates of reference object 340 and HMD 160 in real space 360 and the rotation angle α are determined. Based on the position coordinates of the reference object 340 and the HMD 160 can be a first location vector V1 from the HMD 160 to the reference object 340 be determined. In addition, one from the HMD 160 outgoing second location vector V2 , that of the uncorrected image position 350 of the reference object 340 corresponds, by turning of V1 be determined by the rotation angle α. The uncorrected image position 350 can based on V2 and the position coordinates of the HMD 160 be determined. If the tracking position 345 With " P2 "And the uncorrected image position 350 with "P3", a shift vector Vs for the camera image 170 be determined on the basis of the following equation: $${\text{V}}_{\text{S}}=VektOr\left(\mathrm{<figure-callout\; id="3"\; label="P"\; filenames="DE102019107103A1\_0018.png"\; state="\{\{state\}\}">P</figure-callout>}\mathrm{3,}P2\right)$$

In Equation 2, the displacement vector Vs may be a vector of the uncorrected image position 350 to the tracking position 345 be determined. The direction of the shift vector Vs may indicate the direction of the shift. It should be noted that, even if in 4A a displacement perpendicular to the z-axis is shown (which is a turning of the head of the user of the HMD 160 can correspond), the displacement can also be perpendicular to the x-axis or the y-axis, for example, a pitching motion of the user of the HMD 160 equivalent. In addition, the size of the shift vector Vs may indicate the shift distance. The route can be to the coordinate system of real space 360 calculated and reduced to pixel distances. For example, scaling may be based on a matrix expression that projects world coordinates onto display coordinates.

There are several ways to determine the positions of the HMD 160 and the reference object 340 (eg keyboard 140 ). For example, at the reference object 340 one or more sensor capsules may be mounted to detect directional light rays from light emitters (eg, lighthouses) placed at various predetermined positions in real space 360 are arranged. The sensor capsules can perform distance measurements based, for example, on the transit time of the directional light. The sensor capsules can read the results of the distance measurements to the HMD 160 transfer. The HMD 160 can change the position of the reference object 340 determine on the basis of the measurement results as well as the known positions of the lighthouses and the known direction of the light beams emitted by the lighthouses. In another example, the reference object 340 Directed light rays have emitting light emitter, of several at different predetermined positions in real space 360 arranged light sensors can be detected. The light sensors can also perform range measurements and the results of range measurements to the HMD 160 transfer. The HMD 160 can change the position of the reference object 340 based on the distance measurements and the known directions of the light rays and determine positions of the light sensors. Similar or other techniques (eg, simultaneous positioning and mapping, Wi-Fi based positioning, etc.) may also be used to determine the position of the HMD 160 in real space 360 be used.

Except for the positions of HMD 160 and reference object 340 The determination of the displacement vector Vs is also based on the rotation angle α. The angle of rotation α can correspond to the angle by which the HMD 160 between a first time at which the camera image 170 and a second time point at which a composite image to be displayed to the user is to be rotated. The time period between the first and the second time may correspond to the above-mentioned processing delay. The rotation angle α can be a product between an angular velocity of HMD 160 and the time from the first to the second time. The angular velocity can z. B. with an inertial measurement unit (IMU) of the HMD 160 be measured.

In addition, the time period between the first time and the second time can be determined in various ways. Like, for example 4B As can be seen, as part of a calibration process, the camera of the HMD is capable of capturing an image of an object that is at the first time (in 4 when T0 denotes) can be used. The object may include, for example, a clock indicating the first time, a display device displaying a time-varying image (eg, a bar code) indicative of the first time, a time-varying pattern, which is output from a peripheral device (eg, a keyboard, a mouse, etc.) and has the first time closed, etc.). The clock, display, peripheral and HMD timers can all be synchronized with a global clock. When the image is displayed on a display device at a second time (in FIG 4C when T1 denotes), the second time is recorded. A difference representing the processing delay between the first time and the second time can be determined. The HMD 160 may internally store the processing delay information and use the processing delay information for calculating the rotation angle α for the calculation of the displacement vector Vs when taking a camera image after recording 170 a rotational movement of the HMD 160 is detected. If after taking the camera picture 170 no rotational movement is detected, the HMD 160 determine that when composing a composite image, there is no displacement of the camera image 170 is required.

In some examples, the shift of the camera image 170 take place on a frame buffer in which the camera image 170 is stored. 4C and 4D illustrate examples of moves to a frame buffer 402 , As in 4C shown, the image buffer memory 402 Image data of the camera image 170 to save. The image buffer 402 can in a display area 404 and a non-display area 406 be divided. On in the display area 404 stored first part of the camera image 170 can as second picture 410 be kept available for the creation of a composite image and / or for an advertisement, while one in the non-display area 406 stored second section of the camera image 170 not available for creating a composite image or for display. The display area 404 and the non-display area 406 can affect different address groups or buffer locations in the frame buffer 402 Respectively.

In some examples, the display area and the non-display area of the frame buffer may 402 be moved to the effect of shifting the content of the camera image 170 within the second picture 410 to create. Like, for example 4D can be seen, to the effect of a shift of display 120 , Mouse 130 , Keyboard 140 and user hand 180 In the second picture 410 to the left (labeled " B "Marked), the display area 404 in the frame buffer 402 to extract the second image 410 to the right (labeled " A "Marked). The second picture 410 can then be made available for creating composite images. In the example of 4D depends on the postponed tracking position 345 (based on the movement of the HMD 160 ) at the corrected image position 350 the keyboard 140 out. The HMD 160 can then generate a composite image 310 the pixels of the keyboard 140 of the second picture 410 at the postponed tracking position 345 through the pixels of the virtual keyboard 312 replace. Likewise, to create a composite image 320 Pixel of the user hand 180 also at the postponed tracking position 345 of the second picture 410 extracted and into a virtual image (which also corresponds to the movement of the HMD 160 is moved). In some examples, the positions of display area 404 and non-display area 406 also be static, with the camera image 170 for generating the displacement effect within the frame buffer 402 relative to the static display area and the non-display area.

Embodiments for hand segmentation based on activity distribution

As described above, creating a composite image for creating an immersive AR / MR experience may be to distinguish skin pixels of a user from the pixels of a background object (eg, a keyboard, mouse, or other peripheral device). The skin pixels may be part of the user's hand, part of the hand, or other body parts that interact with the background object. As if looking back 3D . 3F and 4D become apparent when creating a composite image 310 Pixel of the keyboard 140 identified and by pixels of the virtual keyboard 312 replaced. Besides, pixels can also be hand-held 180 extracted and into the composite image 310 be inserted.

The identification of pixels of the keyboard 140 and the hand 180 For example, it may include filtering out pixels that are likely to become part of the hand 180 belong to the user. In addition, when creating a composite image 320 Pixel of the user hand 180 extracted and on the virtual piano keys 324 arranged. In either case, an image segmentation scheme may be used to distinguish the pixels of a body part of a user (eg, hands, fingers, etc.) from the pixels of a background object. For example, the image segmentation scheme may be based on a comparison of the pixels of an image with one or more thresholds representing, for example, a skin tone, a color of the background object, and so on. A determination of whether a pixel belongs to a user's hand may be based on the comparison result.

The accuracy of an image segmentation scheme can be influenced by a variety of factors. One factor can be the skin tone / color of the user. If the user's skin tone / color is apparently similar to the color of the background object (eg, a user-operated peripheral device), either due to the actual skin color or due to ambient light, the pixel color of the user's skin and the pixel color of the peripheral device, as they were taken by the camera are very similar, which can lead to errors in the image segmentation. The accuracy of the image segmentation process may further decrease if a fixed threshold is used to compare the pixels and the fixed threshold does not take into account factors such as the skin may have a different color / tint with different users, the ambient light conditions may change depending on the operating condition , etc.

In some embodiments, the pixel comparison thresholds may be adjusted to improve the accuracy of an image segmentation process based on a spatial distribution of the actions on the peripheral device. More specifically, the spatial distribution of the actions may indicate at which (which) location (s) of the peripheral device (eg, which key or area of a keyboard) certain user actions (e.g., typing) have taken place and at which (which) location (s) of the peripheral device no user action took place. The spatial distribution information can be used to adjust the thresholds. Specifically, an image may be obtained with a capture of the user's body part (e.g., hand, finger, etc.) interacting with the peripheral device, where pixels from different parts of the peripheral device may be identified in the image. Based on the spatial distribution of actions, pixels can be identified that are likely to be a capture of the body part of the user correspond. Various attributes of the identified pixels (eg, luminance, colors, etc.) may be used to set a threshold for comparing other pixels in determining whether those pixels belong to the body part. The information about the spatial distribution can be used as a secondary test. For example, the pixels may be compared to a different threshold that is not determined by the spatial distribution information. If the pixel comparison result is not unique or has a low confidence level, the spatial distribution information may be used to confirm or discard the comparison result. For example, if the comparison result for a pixel is ambiguous and the pixel belongs to a location of the peripheral device where, as indicated by the spatial distribution information, there are few or no user actions, it can be determined that the pixel is not part of the body owned by the user.

The action distribution of the interaction can be determined based on historical information or current information. In particular, the historical information may be in the form of a heat map or a probability map created based on a statistical analysis of previous interaction activities. 6A illustrates an example of a heat map 502 the keyboard 140 , The heat card 502 can control the statistical level of interaction anywhere on the keyboard 140 represent. In 6A can the areas 504 . 506 . 508 and 510 the most frequently used areas (eg keys) of the keyboard 140 represent while the rest of the range of the keyboard 140 less frequently used areas. Each dark area may also include a usage distribution within the area. The heat card 502 Can be used as a grayscale mask for every pixel of the keyboard 140 be encoded in a picture. For example, the grayscale mask represents 520 an example of a grayscale mask that is part of the heat map 502 was derived and on a picture of the keyboard 140 can be displayed. In the grayscale mask 520 For example, a black pixel may correspond to a 100% probability of finding a pixel of a body part of a user (eg, hand, finger, etc.) while a white pixel may correspond to a probability of 0%.

The likelihood information may be used to configure an image segmentation scheme, such as a secondary check, to confirm or discard a decision from a comparison result and / or to adjust the comparison thresholds. For example, assuming that a pixel is difficult to classify as belonging to the skin or not to the skin (eg, because the pixel value is very close to the threshold), the likelihood information may be used to decide on a comparison result confirm or reject. To give an illustrative example, if a comparison shows that a pixel belongs to a skin, then the difference between the pixel and the threshold is below a confidence level, and if the grayscale mask 520 for this pixel is dark, it can be determined that the pixel belongs to the skin. If the grayscale mask 520 however, for this pixel, it can be determined that the pixel does not belong to the skin.

In some embodiments, the thresholds for the comparison of a pixel may be based on the grayscale mask 520 be configured. Based on a picture of the keyboard 140 For example, pixels that are likely to belong to the user's body part / skin may be identified by the grayscale mask 520 be identified. The attributes of these pixels (eg, color, luminance, etc.) may be used to determine whether these pixels belong to the body part / skin of the user for establishing a comparison threshold for comparison with other pixels. The comparison procedure can also be based on the grayscale mask 520 be adjusted. For example, for pixels of an image, their values in the grayscale mask 520 are high, the requirement for qualifying as skin pixels will be looser than for pixels whose values are in the grayscale mask 520 are low. In one illustrative example, pixels whose values are in the grayscale mask may 520 even if those pixels are below the comparison threshold, they qualify as skin pixels if the difference is within a difference threshold, while pixels whose values are in the gray scale mask 520 are low, must be above the comparison threshold to qualify as skin pixels.

For example, in some examples, the historical information may be used to calculate the temporal differences of the same pixel, whereby a multi-level pixel comparison process for improving the computational efficiency may be performed. How out 5B For example, several gray scale masks can be seen 520 (eg grayscale masks 520a . 520b . 520c , etc.) are generated from different frames. A variation in the overall intensity (or color change) from one image to another may allow for pictorial mapping of the image changing parts used to identify one or more edges 530 (eg margins 530a . 530b , etc.) of the pixels that represent the moving objects, such. For example, the user's hand or other body parts that interact with the peripheral device. The edges For example, pixel areas 532 (eg areas 532a . 532b , etc.) that are likely to belong to the user or to other parts of the body, and only pixels originating from those areas are compared to the threshold (s) in determining whether the pixels belong to the body part of the user. With such a determination, the total number of pixels to be compared with the threshold value (s) in the segmentation process can be reduced, whereby the efficiency of the calculation can be improved.

In some embodiments, the thresholds may also be adapted to different operating conditions and users. For example, the thresholds can also be adjusted depending on the ambient light of the operating environment. In particular, the luminance threshold for hand / finger pixels can be corrected upward with high ambient light intensity and downward with low ambient light intensity. In addition, several thresholds, z. B. adapt to different users, different parts of their hands, etc. For example, different thresholds may be adapted for the extraction of pixels associated with fingernails, finger segments, palm, wrist, etc., each of which may have different colors. In addition, the thresholds can also be adapted to different users. For example, some users may have lacquered fingernails, and the thresholds for extracting pixels of the fingernails of these users may be adjusted taking into account the colors of the lacquered fingernails. In another example, some users may have tattoos and / or garments that completely or partially cover their hands, etc., and the thresholds for extracting hand pixels may be adjusted for these users, considering the covered portions of their hands. How out 5C can be seen a hand model database 550 are defined for multiple users to accommodate a distribution of thresholds (e.g., thresholds 552 - 558 for different areas of one hand with different users and / or for different ambient brightnesses).

There are many different ways to create the heat map 602 , For example, heat maps may be off-line from analysis data or other user's peripheral device usage records for other instances of a keyboard 140 to be created. Peripheral devices may also be categorized based on the type and distribution of motion sensors (which may determine a usage distribution pattern), for each category a heat map may be created representing a statistical usage pattern for all peripherals of the category. Statistical analyzes can also be made to determine the likelihood that interaction activity will take place at one site of the peripheral device based on a number of interactive activities taking place at the site and the time the number was determined, with statistical analysis at different levels Positions of the peripheral device can be repeated. In some examples, the thermal map may also be spontaneously created for a particular user to the usage pattern of the keyboard 140 by the user. In addition, several heat maps / probability maps can be created for different applications. For example, the gaming map / probability map shows a high probability that fingers will be found on certain buttons (eg, the "w", "a", "s", "d" buttons and surrounding buttons) while in coding and word processing applications, the probability of finding fingers may be more evenly distributed across the keys of the keyboard. By customizing the heat maps, the accuracy of the heat maps can be improved for accurate prediction of finger position for skin pixel extraction.

In some examples, the action distribution of the interactions may also be determined based on current information. 5D and 5E illustrate an example of capturing a current distribution of hand segmentation actions. How out 5D You can see pictures of the keyboard 140 from a camera 560 to be taken by a camera control 562 is controlled. At the camera 560 (as well as the camera control 562 ) may be, for example, a camera of the HMD 160 or other cameras with a clear view of the keyboard 140 act. The recording of images of the keyboard 140 can through the camera control 562 based on detecting a keystroke action on one or more keys using the keyboard motion sensors 140 to be triggered. For mapping various pixel areas (eg, pixel areas 572 . 574 , etc.) of the image on the keys of the keyboard 140 can be a sensor card 570 to be created. Even if in 5D While a keyboard is shown, other peripherals and motion sensors (eg, buttons, touch pads, etc.) may of course be used to capture up-to-date interaction information.

5E shows one from the camera 560 taken picture 580 the keyboard 140 , Based on the sensor card 570 and capturing the keystrokes can be camera control 562 a first pixel area 582 identifying the one or more keys on which a keystroke action has taken place. Of the Image processor can also have a second pixel area 584 identify who at the first area 582 borders or surrounds it and no keystroke action took place. The first pixel area 582 may include pixels of the user's hand involved in the keystroke action while the second pixel area 584 Pixel of the keyboard 140 may contain. For the first pixel range 582 can be a first histogram 592 a set of color components (eg, relative intensities for each color component). Also for the second pixel range 584 can be a second histogram 594 be determined for the same color component set.

The first histogram 592 and the second histogram 594 can be used to adjust thresholds for pixel comparison in image segmentation (eg, to determine if the pixels belong to the hands / fingers of a user). For example, a threshold value may be calculated for each color component, each threshold based on, for example, averaging (or weighted averaging) the intensities of the color component between the first histogram 592 and the second histogram 594 can be calculated. For a color component set, a set of thresholds can be set. To determine whether a pixel belongs to a hand or a background object, a set of intensities of the color components of the pixel can be calculated. The intensity set can be compared to the threshold set and from the comparison results it can be determined if the pixel belongs to a hand / fingers. In some examples, the histogram data may also be used to redefine a user's base skin tone. The base value information may be used to train a machine learning model to perform hand segmentation on image pixels captured under different lighting conditions.

In some examples, a combination of historical and current information may be used to set a second background detection threshold. For example, images of peripheral device locations where very little or no interaction activity has occurred may be periodically captured and a third histogram of the color components of those locations may be determined. Based on the third histogram, a second threshold for the background detection can be determined. The second threshold for background detection (eg, a threshold probability that a pixel belongs to the background) may be at the first threshold for hand / finger detection (eg, a threshold probability that a pixel belongs to the hand / fingers ) can be combined in different ways. In one example, the second background detection threshold may be used as the first step in the verification. If the pixel is very close to the second threshold and the result is ambiguous, the pixel may be compared to the first threshold for hand / finger recognition. In another example, a combined threshold may be determined by averaging the first and second thresholds, wherein the pixels of the image for identifying the hand pixels may be compared to the combined threshold.

The determination of the thresholds may be part of an image exchange based calibration process. As if looking back 5D it can be seen a hidden from the hand or the fingers of a user image of the keyboard 140 be recorded. The pixels of the image may be used as a reference / starting point for determining the thresholds for distinction between hand / finger pixels and pixels of the peripheral device. As part of the calibration process, a user can place their fingers on the keyboard in a prescribed manner 140 place a second image of the keyboard covered by the fingers 140 can be included. Based on the known pixel positions of the hand / fingers (eg based on recognition key strokes and the sensor card 570 ) as well as the known pixel positions of the keyboard 140 which do not contain hand / fingers, the thresholds can be adjusted from the reference, for example, the probability of a correct distinction between the pixels of the keyboard 140 and to maximize the user's hand / finger pixels. Taking pictures of the keyboard 140 may also be repeated to obtain an updated image for determining the reference thresholds in consideration of a change in operating conditions (eg, change in ambient brightness, inactivity of the peripheral device, etc.).

After identifying the pixels of the body part of the user (eg, hand, finger, etc.), identifying the pixels can create a composite image. For example, the extracted pixels (eg, a hand, the finger, etc.) may be used to represent the virtual interaction, as in FIG 4D are inserted into a virtual environment represented in the composite image. The extracted pixels and / or the virtual environment may also be modified to be matched. As if looking back 4D You can see the luminance of the extracted pixels of the hand 180 For example, they may be adapted to match the lighting conditions in the virtual environment of the composite picture 310 fits. In another example may also be by hand 180 (represented by the extracted pixels) on the virtual keyboard 312 thrown virtual shadows into the composite image 310 be inserted. In some examples, additional pixels of the virtual environment may also be replaced by hand / finger pixels to fill in the gaps in the extraction of the hand / finger pixels.

In some examples, a like in 5F represented database 596 for storing several heat maps / probability maps 597 (eg heat maps 597a . 597b , etc.). Each heat map / probabilistic map may be associated with various attributes that affect a user's interaction with a peripheral device, e.g. For example, in a game application as described above, a different pattern of user interactions may occur on a keyboard than in a word processing application. In another example, a keyboard and a joystick have different user interaction patterns due to the different modes of operation. In addition, different models of the same peripheral device (eg, a keyboard) may also have different user interaction patterns, e.g. B. due to different key arrangements. In the database 596 The heat cards / probability cards can be based on different application identifiers 598 (eg application identifiers 598a . 598b , etc.), which can identify various applications, as well as various peripheral device identifiers 599 (eg, peripheral device identifiers 599a . 599b , etc.), which can identify various types and models of peripherals, be organized. When performing segmentation, a thermal map / probabilistic map may be based on the application for which the user is interacting with the peripheral device and / or database based on the type and model of the peripheral device 596 be retrieved. In such an embodiment, the proper heat map / probability map for making hand segmentation (or segmenting other body parts) may be used, thereby improving the accuracy of the segmentation.

By adjusting hand segmentation based on a distribution of actions on the peripheral device, the hand segmentation scheme may be less prone to other factors that may affect the accuracy of hand detection, such as skin tone, light intensity, etc. Thus, a more robust segmentation scheme can be achieved, the hand pixel for can correctly identify a wider range of skin tones and in a wider range of environments.

system

6 illustrates an example of a process engine 600 that part of the HMD 160 and configured to generate composite images for enabling an immersive interactive AR / MR experience. In some examples, the process engine 600 consist of software instructions written on a hardware processor of the HMD 160 are executable. In some examples, the process engine 500 Also include logic circuits implemented in an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. As in 6 represented, the process engine 600 a motion detection module 602 , a latency determination module 606 , a compensation determination module 608 , a module 610 for generating composite images with an image segmentation module 612 and a frame buffer 402 out 4 exhibit. Although the process engine 600 in 6 is represented as a single entity, the components of the processes engine 600 of course be distributed to different units. So the module can 610 For example, to generate composite images on an external computer, and obtain image data and sensor data from an HMD, generate the composite image and transmit the composite image back to the HMD for display.

The motion detection module 602 can be a message of a movement of the HMD 160 and measurements of speed and direction (eg angular velocity, direction of rotation, etc.) of the HMD 160 from the IMU 620 of the HMD 160 receive. In addition, the latency determination module 506 a processing delay when recording and processing the camera image 170 from the camera 601 of the HMD 160 based on the techniques described above. For example, the latency determination module 606 the time to which the camera 601 the camera picture 170 based on the content of the camera image 170 (eg, the camera image has 170 an image of a clock, a time-varying pattern emitted by an object, etc.) received as a first time, take a second time to which the camera image 170 in the frame buffer 402 is stored and available for the generation of composite images, and determine the processing delay based on a difference between the first time and the second time.

For example, the compensation determination module 608 determine the displacement vector Vs based on the angular velocity information and the processing delay information that make up the rotation angle α for example, by multiplying the angular velocity by the processing delay. The displacement vector may also be based on the position coordinate information 602 of the HMD 160 and the reference object 340 be determined. The position coordinate information 603 As described above, for example, they can be obtained from measurements or based on measurements made with the aid of a reference object 340 attached sensor capsule, with the help of light sensors, that of the reference object 340 detecting emitted light, etc. are made available. The determination of the displacement vector V _S can by a message of the motion detection module 602 about a movement of the HMD 160 to be triggered. When the movement after taking the picture 170 is the compensation determination module 608 use the information on the displacement vector Vs to determine a displacement direction and a displacement distance, and use the direction / route information to determine the storage location of the display area 404 in the frame buffer 402 to change or the picture 170 directly in the image memory 402 to move to a second picture 410 to create and the second picture 410 the module 610 to provide composite images.

The module 610 For generating composite images, the image segmentation module comprises 612 that is a segmentation of the second image 410 based on the above techniques. For example, segmentation may be performed to distinguish pixels of a user's hand / fingers (or other body parts) from the pixels of a user-operated peripheral device. The segmentation may be based on a comparison of the pixels of the second image 410 are based on one or more thresholds determined, for example, based on a distribution of interactions with the peripheral device. Further, the one or more thresholds may be to the user (eg, based on the skin color / tint of the user, features of the user's fingernails, finger segments, etc.) as well as the operating condition (eg, ambient brightness). be adjusted. Based on the segmentation result, pixels of the peripheral device or pixels of the body part of the user can be extracted. In some examples, the module may 610 to create composite images, insert the extracted pixels to render a virtual interaction in a composite image that depicts a virtual environment. The module 610 For generating composite images, the composite image may also be created from the first image based on replacing the extracted pixels with pixels of a virtual object (eg, virtual hand / finger). Further, the extracted pixels and / or the virtual environment may be modified to be matched. For example, the luminance of the extracted pixels of the body part of the user may be adjusted to match the lighting conditions of the virtual environment. In another example, virtual shadows cast by the hand (represented by the extracted pixels) on a virtual peripheral may also be inserted into the composite image. In some examples, additional pixels of the virtual environment may also be replaced by hand / finger pixels to close gaps in the extraction of the hand / finger pixels. The composite image can then be displayed 530 of the HMD 160 be made available to the user to be presented.

method

7 shows a simplified flowchart for a method 700 to provide mixed reality (MR) interaction. The procedure 700 can be, for example, an HMD in conjunction with the process engine 600 from 6 based on the techniques described above.

The procedure 700 starts with step 702 in which the camera 601 an HMD of the user can capture a first image of an interaction of the user with a real object. For example, the real object may include a real keyboard, and the interaction may include, for example, tapping the user's finger on the real keypad.

In step 704 can the process engine 600 measure a movement of the HMD relative to the real object between the first time and a second time. For example, the motion detection module 602 as described above by the IMU 620 of the HMD 160 a message about a movement of the HMD 160 as well as measurements of speed and direction (eg angular velocity, direction of rotation, etc.) of the HMD 160 receive. In addition, the latency determination module 506 a processing delay in recording and processing the camera image 170 from the camera 601 of the HMD 160 based on the techniques described above. In one example, the latency determination module may be 606 the time to which the camera 601 takes the first picture based on the content of the camera image 170 (the first image has, for example, an image of a clock, a time-varying pattern emitted by an object, etc.) as the first one Received time to record a second time when the first frame in the frame buffer 402 stored and available to create a composite image, and determine the processing delay based on a difference between the first time and the second time. In addition, the compensation determination module 608 the displacement vector V _S for example, based on the angular velocity information and the processing delay information that can be used to calculate the rotation angle α, for example, based on multiplying the angular velocity by the processing delay. The displacement vector may also be based on the position coordinate information 602 of the HMD 160 and the reference object 340 be determined. The position coordinate information 603 As described above, for example, they can be obtained from measurements or based on measurements made with the aid of a reference object 340 attached sensor capsule, with the help of light sensors, that of the reference object 340 detecting emitted light, etc. are made available. The determination of the displacement vector Vs can be achieved by a message from the motion detection module 602 about a movement of the HMD 160 to be triggered.

In step 706 can the process engine 600 process the first image to create the second image based on the measurement of movement of the HMD. For example, if motion is detected after the first image has been captured, the compensation determination module may 608 to create the second image, the information about the displacement vector V _S for determining a shift direction and a shift distance, and the direction / route information for changing the storage location of the display area 404 in the frame buffer 402 or for direct displacement of the image 170 in the frame buffer 402 use.

In step 708 can the process engine 600 based on the second image, create a composite image of a user-involving virtual interaction. For example, the module 610 for generating composite images, extract pixels of the peripheral device or pixels of the body part of the user (eg, hand, finger, etc.). To play a virtual interaction, the module 610 For example, to create composite images, in some examples, insert the extracted pixels into a composite image that depicts a virtual environment. The module 610 For generating composite images, the composite image may also be generated from the first image based on replacing the extracted pixels by pixels of a virtual object (e.g., virtual hand / finger). The extracted pixels and / or the virtual environment may be further modified to be matched. For example, the luminance of the extracted pixels of the body part of the user can be adapted to the lighting conditions of the virtual environment. In another example, virtual shadows cast by the hand (represented by the extracted pixels) on a virtual peripheral may also be added to the composite image. Further, in some examples, additional pixels of the virtual environment may be replaced by hand / finger pixels to close gaps in the extraction of the hand / finger pixels.

In step 710 then can the composite image of the ad 530 of the HMD 160 be made available to the user instead of the real interaction. The presentation may be done substantially simultaneously with the real interaction to facilitate a mixed reality experience.

8th shows a simplified flowchart of a method 800 to provide mixed reality (MR) interaction. The procedure 800 For example, on an HMD in conjunction with the process engine 600 from 6 based on the techniques described above.

The procedure 800 starts with step 802 in which a first image of a limb of a user and a real peripheral input device is captured. The first picture can be z. B. from the camera 601 an HMD. The limb of the user may at least partially obscure the real peripheral input device. The first image may represent a real interaction between the limb of the user and the real peripheral input device. The limb may be z. As a hand, fingers, etc. act. The physical peripheral input device may be, for example, a keyboard, a mouse, etc.

In step 804 can the process engine 600 perform segmentation of the first image in a first pixel region associated with the limb of the user, wherein a second pixel region belongs to the real peripheral input device. In some embodiments, the segmentation may be based on a comparison of the pixels of the first image with one or more thresholds determined, for example, based on a distribution of the interactions with the physical peripheral input device. The distribution of the interactions may include a historical distribution of the interactions (of the user or a group of users), an actual distribution of the user's interactions, or a combination of both. The one or more thresholds may be based on of the techniques described above are also adapted to the user (eg, based on the skin color / tint of the user, the characteristics of the user's fingernails, finger segments, etc.) as well as the operating condition (eg, ambient brightness). Based on the segmentation result, pixels of the peripheral device or pixels of the body part of the user can be extracted.

In step 806 can the process engine 600 making a composite image from the first image based on rendering pixels representing a virtual peripheral input device instead of the second pixel region such that the virtual peripheral input device appears in the composite spatial image at the same spatial location as the real peripheral input device and at least partially through the peripheral image input device Limb of the user is covered. For example, if the capture of the first image shows the user typing on a real keyboard, a virtual keyboard may be spatially at the same location in the composite image as the real keyboard, thereby giving the user the impression that he is instead using the real keyboard typing on the virtual keyboard.

In step 808 For example, the composite image may be useful to the user e.g. B. on the display 630 to get presented. To give the user a mixed reality experience, the composite image may be displayed substantially simultaneously with the real interaction.

Briefly stated are methods and systems for facilitating mixed reality (MR) interaction. In one example, the method includes: capturing a first image of a limb of a user and a real peripheral input device, wherein the limb of the user at least partially obscures the real peripheral input device; Segmenting the first image into a first pixel region corresponding to the limb of the user, wherein a second pixel region corresponds to the real peripheral input device; Creating a composite image from the first image based on rendering pixels representing a virtual peripheral input device instead of the second pixel region such that the virtual peripheral input device spatially appears in the composite image at the same location as the real peripheral input device and at least the user's limb partially obscured; and displaying the composite image to the user.

The subject matter of the present disclosure includes other variations. Accordingly, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain embodiments of the techniques detailed above are illustrated in the figures. It is to be understood, however, that a limitation of the disclosure to the disclosed specific form (s) is not intended, but is intended to cover all modifications, alternative constructions, and equivalents, that are included in the spirit and scope of the disclosure as set forth in the is defined in the appended claims. For example, any of the embodiments, alternative embodiments, etc., and their concepts may be applied to any of the described and / or within the scope and spirit of the disclosure.

The use of the terms "a" and "the", "the", "the" and similar references in the context of a description of disclosed embodiments (particularly in the context of the following claims), should be construed as including both the singular and the plural unless otherwise stated in this document or clear contradiction arises from the context. The terms "comprise," "have," "include," and "contain," unless otherwise specified, are to be understood as non-exhaustive enumeration terms (that is, they include, but are not limited to). ). The term "connected" is contained as part or wholly in something, appropriate to something, or to be associated with something, even if there is something in between. The term "based on" should be understood to be extensible and in no way limiting, and should, where appropriate, be interpreted or read as "at least partially based." Unless specified otherwise in this document, a range of value ranges is intended to serve only as a shorthand for a process which refers individually to each of the values falling within the range, in which case each individual value is included as he would be directly mentioned in it. All methods described in this specification may be carried out in any suitable order unless stated otherwise or contradicted in context. All examples or examples of preliminary formulations (eg, "as") used in this specification are merely illustrative of the embodiments of the disclosure and, unless otherwise claimed, do not limit the scope of the disclosure The expressions used in the specification should be construed to designate an unclaimed element as essential to the practice of the disclosure.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 62645760 [0001]

Claims (22)

A method of facilitating mixed reality (MR) interaction, comprising: Capturing a first image of a limb of a user and a real peripheral input device, wherein the limb of the user at least partially obscures the real peripheral input device; Segmenting the first image into a first pixel region corresponding to the limb of the user, a second pixel region corresponding to the real peripheral input device, Creating a composite image from the first image based on rendering pixels representing a virtual peripheral input device instead of the second pixel region such that the virtual peripheral input device spatially appears in the composite image at the same location as the real peripheral input device and at least the user's limb partially obscured, and Displaying the composite image to the user. Method according to Claim 1 wherein the composite image is presented to the user at substantially the same time that the limb of the user at least partially obscures the real peripheral input device. Method according to Claim 1 or 2 wherein the segmenting of the first image is based on a distribution of activities at multiple locations of the physical peripheral input device. Method according to Claim 3 wherein segmenting the first image comprises determining an historical distribution of the activities at the plurality of locations of the physical peripheral input device, wherein the historical distribution of the activities is based on previous interactions between the user or a group of users with the physical peripheral input device, determining one or a plurality of thresholds based on the historical distribution of the activities, comparing the pixels of the first image with the one or more thresholds, and determining the first pixel region corresponding to the limb of the user and the second pixel region corresponding to the physical peripheral input device based on the comparison results. Method according to Claim 4 wherein segmenting the first image comprises determining a distribution of the probabilities that the user's limb is located at a respective one of the plurality of locations of the physical peripheral input device based on a statistical analysis of the historical distribution of the activities, identifying pixels likely to be corresponding to a portion of the limb of the user, based on the distribution of the probabilities, and determining the one or more thresholds based on one or more attributes of the identified pixels. Method according to Claim 5 method, further comprising: determining a first threshold of the one or more thresholds associated with a first portion of the limb of the user, determining a second threshold of the one or more thresholds associated with a second portion of the limb of the user, and comparing the pixels of the user first image having the first threshold and the second threshold to determine the first pixel region corresponding to the limb of the user. Method according to Claim 6 method further comprising: determining the first threshold and / or the second threshold based on an ambient brightness and / or an identity of the user. Method according to one of Claims 3 to 7 wherein the distribution of the activities comprises a current distribution of the activities at the plurality of locations of the real peripheral input device. Method according to Claim 8 further comprising: receiving an indication of an activity from a sensor located at a first location of the physical peripheral input device, capturing a third image of the physical peripheral input device based on the message, selecting a third pixel region corresponding to the first location, and one of the first Location, determining a first color component histogram for the third pixel region, determining a second color component histogram for the fourth pixel region, determining one or more thresholds based on the first histogram and the second histogram, and comparing pixels of the first image with the one or more thresholds to determine the first pixel region of the first image corresponding to the limb of the user. Method according to Claim 9 method further comprising: identifying a second location of the physical peripheral input device that has less than a predetermined number of previous activities based on a historical distribution of the activities, Identifying a fifth pixel area corresponding to the second location from the third image, determining a third color component histogram for the fifth pixel area, determining a second threshold of the one or more thresholds based on the third histogram, and comparing the pixels of the first image to the second threshold to determine the second pixel area of the first image corresponding to the real keyboard. Method according to one of Claims 3 to 10 method further comprising: determining a temporal change of the distribution of the activities, determining a region of interest based on the change, identifying pixels in the region of interest of the first image, and determining the first pixel region corresponding to the limb of the user from the identified pixels. Method according to one of Claims 3 to 11 wherein the distribution of the activities from a plurality of activity distributions is determined based on a model of the real peripheral input device and / or an application for which the user interacts with the physical peripheral input device. Method according to one of Claims 1 to 12 further comprising: capturing a second image of the real peripheral input device that is not obscured by a limb of the user, determining one or more thresholds based on the second image, sending an indication to the user to place the limb of the user at predetermined locations the real peripheral input device, capturing a third image of the limb of the user at the predetermined locations of the real peripheral input device, adjusting the one or more thresholds based on the predetermined locations of the real keyboard corresponding pixels of the third image, and comparing the pixels of the first image with the adjusted one one or more matched thresholds to determine the first pixel region corresponding to the limb of the user. Method according to one of Claims 1 to 13 wherein the composite image is created by pixels representing a virtual environment in which the virtual peripheral input device is located based on a replacement of pixels corresponding to a real environment in which the real peripheral input device is located. Method according to Claim 14 wherein the composite image is generated based on adjusting a luminance of the first pixel region based on a luminance of the virtual environment. Method according to Claim 14 or 15 wherein the composite image is created based on adding a virtual shadow cast of the user's limb into the virtual environment. Method according to one of Claims 1 to 16 wherein the first image is taken by a camera of a head-mounted display (HMD) and wherein the composite image is displayed on the HMD. Method according to one of Claims 1 to 17 wherein segmenting the first image into the first pixel region corresponding to the limb of the user, identifying two or more threshold values for the user's limb comprises rendering pixel values wherein pixels of the first image that are not within the two or more thresholds are discarded become. Apparatus comprising: a display device, a camera adapted to capture a first image of a limb of a user and a real peripheral input device, wherein the limb of the user at least partially obscures the real peripheral input device, and an image processor coupled to and configured with the camera and the display device: get the first picture from the camera, segment the first image into a first pixel region corresponding to the limb of the user, a second pixel region corresponding to the real peripheral input device, to construct a composite image from the first image based on rendering pixels representing a virtual peripheral input device instead of the second pixel region such that the virtual peripheral input device spatially appears in the composite image at the same location as the real peripheral input device and the limb of the user at least partially obscured, and to provide the composite image of the display device for presentation to the user. Device after Claim 19 wherein the apparatus for carrying out a method according to one of Claims 1 to 18 is trained. Non-transitory computer readable medium storing instructions that, when executed by a hardware processor, cause the hardware processor: receive from a camera a first image of the limb of a user and a real peripheral input device, wherein the limb of the user at least partially obscures the real peripheral input device to segment the first image into a first pixel region corresponding to the limb of the user, wherein a second pixel region is the one real peripheral input device corresponds to creating a composite image from the first image based on rendering pixels representing a virtual peripheral input device instead of the second pixel region so that the virtual peripheral input device spatially appears in the composite image at the same location as the real peripheral input device the limb of the user is at least partially obscured, and to provide the composite image of the display device for presentation to the user. Non-volatile, computer-readable medium after Claim 21 wherein the instructions are the hardware processor for performing a method according to any one of Claims 1 to 18 cause.

Images