KR101532854B1 - Using occlusions to detect and track three-dimensional objects - Google Patents

Abstract

The mobile platform detects and tracks three-dimensional (3D) objects using occlusions of two-dimensional (2D) surfaces. To detect and track the 3D object, an image of the 2D surface with the 3D object is captured and displayed, and the 2D surface is detected and tracked. Closure of the region designated as the region of interest on the 2D surface is detected. The shape of the 3D object is determined based on a predetermined shape or using the shape of the area of the 2D surface obscured by the position of the camera with respect to the 2D surface, and the shape is calculated. Any desired action is performed on the position of the 3D object on the 2D surface, such as rendering and displaying the graphical object on or near the displayed 3D object based on the position of the region of interest of the 2D surface and the shape of the 3D object .

Description

USING OCCLUSIONS TO DETECT AND TRACK THREE-DIMENSIONAL OBJECTS < RTI ID = 0.0 >

In augmented reality (AR) applications, real-world objects are imaged and displayed on a screen with computer generated information such as image or text information. At the AR, imaged real-world objects are detected and tracked to determine camera position and orientation (pose) information for the object. This information is used to render the graphic object accurately and to display it together with the real-world object. Real-world objects that are detected and tracked are generally two-dimensional (2D) objects. Detecting and tracking three-dimensional (3D) objects is algorithmically more complicated and computationally very costly as compared to detecting and tracking 2D surfaces. On a desktop computer, full 3D tracking is typically performed by recognizing the entire 3D object from the geometry of the object. Due to limited processing capability on mobile platforms such as smart phones, there is no full 3D object tracking solution for mobile platforms. However, 3D object detection and tracking remains an important goal in AR applications to build intense user experiences tailored to real-world 3D objects, not 2D images or planes.

Thus, what is needed is an improved method for detecting and tracking 3D objects.

The mobile platform detects and tracks 3D objects using the obliteration of 3D objects on the 2D surface. To detect and track the 3D object, an image of the 2D surface with the 3D object is captured and displayed, and the position of the mobile platform with respect to the 2D surface is detected and tracked. Closure of the region designated as the region of interest on the 2D surface is detected. The shape of the 3D object is determined based on a predetermined shape or by using the shape of the area of the 2D surface obscured with the position of the mobile platform with respect to the 2D surface, and the shape is calculated. Any desired action is performed on the position of the 3D object on the 2D surface, such as rendering and displaying the graphical object on or near the displayed 3D object based on the position of the region of interest of the 2D surface and the shape of the 3D object .

Figures 1a and 1b illustrate front and rear views, respectively, of a mobile platform capable of detecting and tracking 3D objects using occlusion of a 2D surface.
Figure 2 shows a front view of a mobile platform held in landscape mode displaying a 2D image of a checkerboard surface and a captured image of a 3D tubular object.
Figure 3 is a flow chart for detecting and tracking 3D objects using occlusions of a 2D surface as a mobile platform.
Figure 4 shows a top view of an imaged 2D checkerboard surface having two zones designated as "areas of interest ".
Figure 5 shows a side view of a mobile platform for capturing an image of a 2D surface and a 3D object, wherein the 3D object shadows the area of the 2D surface based on the shape of the 3D object and the pose of the mobile platform to the 2D surface.
Figure 6 shows a top view of a 2D checkerboard surface showing the area of the 2D surface obscured by the 3D object.
Figure 7 shows a front view of a mobile platform held in landscape mode displaying a captured image of a 2D checkerboard surface and a 3D tubular object with graphics objects rendered and displayed.
Figure 8 is a block diagram of a mobile platform that can detect and track 3D objects using occlusions of a 2D surface.

Figures 1a and 1b illustrate front and back views, respectively, of a mobile platform 100 that can detect and track 3D objects using occlusions of a 2D surface. The mobile platform 100 in Figures 1A and 1B is shown to include a housing 101, a display 102, which may be a touch screen display. The mobile platform 100 may also include a speaker 104 and a microphone 106, for example, where the mobile platform 100 is a cellular telephone. The mobile platform 100 further includes a front camera 108 for imaging the environment displayed on the display 102. [ The mobile platform 100 may further include motion sensors 110 such as accelerometers, gyroscopes and the like that may be used to detect movement of the mobile platform 100 or equivalently the motion sensors 110, May be used to assist in determining the pose of the camera 108, which may have a positional relationship. The pose of the mobile platform 100 may also or alternatively be determined using vision-based tracking techniques. The mobile platform 100 may be a cellular or other wireless communication device, a personal communication system (PCS) device, a personal navigation device (PND), a personal information manager (PIM), a personal digital assistant (PDA), a laptop, RTI ID = 0.0 > AR), < / RTI >

Figure 2 shows a front view of the mobile platform 100 held in landscape mode. The display 102 is shown displaying an image of a real-world 2D checkered surface 120 having a real-world 3D tubular object 122 on a surface 120. If desired, graphical objects that are rendered to the computer may also be displayed by the display 102. Real-world objects are created using the camera 108 (FIG. 1B), while any AR objects are computer-rendered objects (or information). The mobile platform 100 detects and tracks the 3D object 122 based on the occlusion of the 2D surface 120 at a known location and the shape of the determined 3D object. As a result, the mobile platform 100 is not required to directly detect and track 3D objects, thereby improving the calculation efficiency, which means improved frame rate and battery usage.

Figure 3 is a flow chart for detecting and tracking 3D objects using occlusions of a 2D surface. As shown, an image of a 2D surface with 3D objects is captured and displayed (202). It should be appreciated that the captured image may be a single image, e.g., a picture or frame of video, or, if desired, one or more pictures or frames of video may be used to create a single image. The position of the mobile platform relative to the 2D surface is detected and traced (204). For example, the position of the mobile platform 100 relative to the 2D checkerboard surface 120 shown in FIG. 2 is detected and tracked. Detecting and tracking the position of the mobile platform relative to the 2D surface may be performed using well-known vision-based tracking where features from additional images from the camera 108 are extracted and extracted from the image of the 2D surface It is compared with features. Alternatively, if desired, the relationship between the mobile platform and the 2D surface may be determined based on a known pose of the 2D surface in the real world, and a pose of the mobile platform determined using the motion sensors 110.

A particular region of the 2D surface, i.e., a region having a known location, is designated 206 as an "area of interest ". The area of interest is the area that is expected to be covered by the 3D object. Designation of the area of interest may be performed by the user while viewing the 2D surface through the display 102 using, for example, a graphical user interface on the mobile platform 100. Alternatively, the designation of the area of interest may be performed off-line and stored in the memory of the mobile platform 100. If desired, only the entire 2D surface or only a small area of the 2D surface may be designated as the region of interest. One or more regions may be designated as regions of interest within the same 2D surface. These regions of interest may be regular geometric shapes such as squares, rectangles, triangles, circles, etc., or any polygon that covers a particular area in a 2D image. For example, the location of the region of interest as well as the shape of the region of interest may be based on the predefined shape and location of the 3D object. Figure 4 shows a top view of an imaged 2D checkerboard surface 120 having, for example, two zones 124a and 124b with locations known to be designated as "areas of interest ". The zones 124a and 124b are circular, such as the 3D object 122 expected in this example, and have a tubular shape and thus a circular cross-section. If desired, more areas of interest may be designated. Also, if desired, the region of interest may have a shape that is different from the shape of the cross-section of the 3D object.

The process then detects if the region of interest is covered, i.e. covered (208). The occlusion is performed by performing a natural feature detection to match using a plurality of features within the region of interest and to determine that a significant portion of the features in the region of interest, It may be detected by confirming a case where a large value is not detected. The features may be detected and matched as part of a vision-based tracking technique. If the features are appropriately distributed over the area of interest, it is possible to find the area covered by the geometrically covered area and the area not covered. If desired, it may be required that the entire ROI, or a significant portion of the ROI, be obscured such that the AR object is rendered and displayed. In other words, covering only a portion of the area of interest may be insufficient. The occlusion of the region of interest may be presumed to be caused by a 3D object, although this may not necessarily be the case. To reduce the amount of false detections, i.e., the detection of occlusions of a region of interest that are not caused by a 3D object, the shape of the region of interest may have the same cross-section as the 3D object, If a different area is obscured, it may be determined that the occlusion is not caused by the 3D object. For example, if desired, certain shapes of the area of interest may be used, and only certain objects may be used to mark the area of interest. Since the region of interest has a known location on the 2D surface, the occlusion of the detected region of interest provides information on the location of the object to be masked, and the masked object is assumed to be a 3D object.

The shape of the 3D object is also determined (210). The shape of the 3D object may be predefined, i.e., the shape of the 3D object is provided by the user or preprogrammed and stored in memory. The shape of the 3D object may also be determined based on the occlusion area of the 2D surface. 5 illustrates a side view of a mobile platform 100 that captures an image of a 2D surface 120 and a 3D object 122. The 3D object 122 is arranged in a region 126 of the 2D surface 120, . The 3D object 122 will mask the area 126 of the 2D surface 120 and the area 126 of the 2D surface 120 will cover the 3D object 122 As well as the pose (i.e., position and position) of the mobile platform 100 relative to the surface 120. Figure 6 shows a top view of the checkered surface 120 imaged with the area 126 obscured by the 3D object shown as hatching. Thus, based on the known pose of the mobile platform 100 to the 2D checkerboard surface 120 determined by tracing the 2D surface 120, along with the shape of the obscured region of the 2D checkerboard surface 120, The shape of the light guide plate 122 may be calculated. As the mobile platform 100 moves and implements the 2D checkerboard surface 120 and the 3D object 122, additional information about the shape of the 3D object 122 is derived and stored in the memory of the mobile platform 100 May be stored.

Using the known location of the obscured region of interest and the shape of the determined 3D object; The mobile platform 100 can indirectly track 3D objects. Thus, instead of directly detecting and tracking the actual 3D object, the obscured region of interest on the 2D surface is detected and tracked, greatly reducing the computational requirements. Any desired operation may then be performed 212 for the position of the 3D object on the 2D surface. For example, the augmented reality objects may be rendered and displayed with reference to the location of the obscured area of interest, where the AR object is rendered (212) for the determined 3D object shape. If desired, any other operation, such as performing discrete operations, using a mobile platform, or switching on and off behaviors within the mobile platform, may be triggered based on the determined location of the 3D object. Real-world behaviors or actions may be controlled by the mobile platform, e.g., via radio frequency or optical signals, directly or via a wireless network. For example, real world or simulated electrical circuits may be controlled by placing 3D objects (tactile icons) on a wiring diagram. Switch points at the train station may be controlled by placing 3D objects on the plan of the train station. Other examples include affecting or controlling the flow simulation application by, for example, placing the 3D object as an obstacle in the flow, for example, the obstacle may be a flow source, such as a windmill ventilator. By placing the actual ventilator into a scene, the flow simulation changes accordingly. Another example is adjusting the collision response, for example, a virtual object may respond accurately to a collision of a 3D object placed as an obstacle in a physical simulation application.

Figure 7 shows a front view of a mobile platform 100 that displays a captured image of a 2D checkerboard surface and a 3D tubular object, which is similar to that shown in Figure 2, but also depicts graphics objects rendered and displayed . For example, the AR object 128 may be displayed relative to the size of the 3D object 122 and displayed with reference to the obscured region of interest, such that the 3D object 122 may be displayed do. Other AR objects 129a and 129b may be rendered for the size of the 3D object 122 but displayed near the 3D object 122 rather than over it. In this example, the AR objects 129a and 129b fly around the 3D object 122 and appear as bees disappearing behind the 3D object 122 as shown by the AR object 129b.

Figure 8 is a block diagram of a mobile platform 100 that can detect and track 3D objects using occlusions of a 2D surface. The mobile platform 100 includes means for capturing an image of a 2D surface having a 3D object, such as a camera 108, such as accelerometers, gyroscopes, electronic compasses, or other similar motion sensing elements And may include motion sensors 110. The mobile platform 100 may include other position determination methods such as object recognition using "computer vision" techniques. The mobile platform further includes a user interface 150, such as display 102, that includes means for displaying images and AR objects. The user interface 150 may also include a keypad 152 or other input device through which a user may enter information into the mobile platform 100. [ If desired, the keypad 152 may be removed by incorporating the virtual keypad into the display 102 using the touch sensor. The user interface 150 may also include a microphone 106 and a speaker 104, for example, if the mobile platform is a cellular telephone. Of course, the mobile platform 100 may include other elements not related to the present disclosure, such as a wireless transceiver.

The mobile platform 100 also includes a control unit 160 that is connected to and communicates with the camera 108, the motion sensors 110, and the user interface 150. The control unit 160 receives data from the camera 108 and the motion sensors 110, processes the data, and controls the display 102 in response thereto. The control unit 160 may be provided with a processor 161 and associated memory 164, hardware 162, software 165, and firmware 163. The control unit 160 includes an image processor 166 for processing images from the camera 108 to detect 2D surfaces to determine when the regions of interest are obscured and, if desired, to determine the shape of the obscured region It is possible. The control unit may also be adapted to determine and track the pose of the mobile platform 100 to the detected 2D surface based on visual data from the camera 108 and / or data received from the motion sensors 110. [ And a position processor 167. The control unit 160 may further comprise a graphics engine 168, which may be, for example, a game engine, for rendering the desired AR objects for the location of obscured regions of interest and for the shape of the determined 3D object . Graphics engine 168 may retrieve AR objects from the storage (e.g., at memory 164). The image processor 166, the position processor 167 and the graphics engine are shown separate from the processor 161 for clarity, but may be part of the processor 161, May be implemented within the processor. The processor 161 as used herein may include, but is not limited to, one or more microprocessors, embedded processors, controllers, application specific integrated circuits (ASICs), digital signal processors You do not need to do this. The term processor is intended to describe functions implemented by the system rather than the specific hardware. The term "memory" as used herein also refers to any type of computer storage media, including long term, short term, or other memory associated with a mobile platform, and may include any type of memory, Or a type of medium in which the memory is stored.

The device includes means for detecting and tracking a 2D surface, which may include an image processor 166, a position processor 167, as well as motion sensors 110 if desired. The device further comprises means for detecting occlusion of a region of interest on the 2D surface, which means may comprise an image processor 166. The means for determining the shape of the 3D object may include a graphics engine 168 that retrieves a predetermined shape of the 3D object from the memory 164. Also, for example, means may be provided for determining the shape of the occlusion region of the 2D surface by the image processor 166, for example, a position processor 167, as well as motion sensors 110 A means for determining the pose of the mobile platform with respect to the 2D surface may be provided by the means for determining the shape of the 3D object and the means for determining the shape of the 3D object may comprise means for determining the shape of the 3D object, May be processor 161 that utilizes the shape of the region and the pose of the camera with respect to the 2D surface. The means for rendering the graphic object with reference to the location of the region of interest of the 2D surface may be a graphics engine 168, wherein the graphic object is rendered relative to the shape of the 3D object.

The methodologies described herein may be implemented in various ways depending on the application. For example, these methodologies may be implemented with hardware 162, firmware 163, software 165, or any combination thereof. In a hardware implementation, the processing units may include one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs) Microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or a combination thereof, as will be apparent to those skilled in the art.

For a firmware and / or software implementation, the methodologies may be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. Any machine-readable medium that tangibly embodies the instructions may be used to implement the methodologies described herein. For example, the software codes may be stored in the memory 164 and executed by the processor 161. [ The memory may be implemented within the processor 161 or external to the processor 161. [

When implemented in firmware and / or software, the functions may be stored in one or more instructions or code on a computer readable medium. Examples include non-volatile computer-readable media encoded in a data structure and computer-readable media encoded with a computer program. For example, a computer readable medium having stored thereon program code is program code for displaying an image of a 2D surface having a 3D object, captured by a camera, program code for detecting and tracking a 2D surface in the image, 2D Program code for detecting the occlusion of the region designated as the region of interest on the surface, program code for determining the shape of the 3D object, and program code for rendering and displaying the graphics object with reference to the location of the region of interest on the 2D surface Where the graphics object is rendered for the shape of the 3D object. The computer readable medium may further comprise program code for determining the shape of the occlusion region of the 2D surface and program code for determining the pose of the camera with respect to the 2D surface, The code uses the shape of the occluded region of the 2D surface and the pose of the 2D surface to determine the shape of the 3D object. Computer readable media include physical computer storage media. The storage medium may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage or other magnetic storage devices, And any other medium that can be accessed by a computer; Disks and discs as used herein include compact discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy discs and Blu-ray discs, Discs usually reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer readable media.

For example, a computer readable medium having stored thereon program code is program code for displaying an image of a 2D surface having a 3D object, captured by a camera, program code for detecting and tracking a 2D surface in the image, 2D Program code for detecting an occlusion of a region designated as a region of interest on the surface, program code for determining the shape of the 3D object, and program code for rendering and displaying a graphics object with reference to the location of the region of interest on the 2D surface Where the graphic object is rendered for the shape of the 3D object.

While the present invention has been described in connection with specific embodiments for purposes of study, the invention is not so limited. Various adaptations and modifications may be made without departing from the scope of the present invention. Accordingly, the spirit and scope of the appended claims should not be limited to the foregoing description.

Claims (35)

Capturing and displaying an image of a two-dimensional (2D) surface having a three-dimensional (3D) object with a camera;
Detecting and tracking the 2D surface;
Designating a region of the 2D surface having a known location on the 2D surface as a region of interest;
Detecting an occlusion of the region of interest within the image of the tracked 2D surface by the 3D object;
Obtaining information about the position of the 3D object based on the detected occlusion of the region of interest;
Determining a shape of the 3D object; And
Rendering and displaying a graphical object with reference to the known location of the region of interest on the 2D surface, wherein the graphical object is rendered for a shape of the 3D object; , Way. delete The method according to claim 1,
Wherein the shape of the 3D object is predefined. The method according to claim 1,
Wherein determining the shape of the 3D object comprises: determining a shape of the occluded region of the 2D surface and a position of the camera with respect to the 2D surface; determining a shape of the occluded region of the 2D surface, Using the position of the 3D object to calculate the shape of the 3D object. The method according to claim 1,
Wherein the graphical object is rendered on the 3D object. The method according to claim 1,
Wherein the graphical object is rendered near the 3D object. The method according to claim 1,
Wherein a plurality of regions of the 2D surface are designated as regions of interest. The method according to claim 1,
Wherein the known location of the region of interest is predefined. The method according to claim 1,
Wherein the image is captured and displayed by a mobile platform. As a mobile platform,
camera;
A processor connected to the camera;
A memory coupled to the processor;
A display connected to said memory; And
Wherein the processor is maintained in the memory and is executed on the processor to cause the processor to display on the display an image of a two-dimensional (2D) surface captured by the camera and having a three-dimensional (3D) Detecting and tracking the 2D surface and detecting occlusion of a region designated as a region of interest on the traced 2D surface with a known location on the 2D surface; and detecting the occlusion of the 3D object based on the detected occlusion of the region of interest Software for obtaining information about a location, determining a shape of the 3D object, and rendering the graphic object on the display by referring to the known location of the area of interest on the 2D surface,
Wherein the graphical object is rendered for a shape of the 3D object. delete 11. The method of claim 10,
Wherein the shape of the 3D object is predefined. 11. The method of claim 10,
Wherein the software executing on the processor causes the processor to determine a shape of the 3D object based on a shape of the occlusion region of the 2D surface and a position of the mobile platform relative to the 2D surface. 11. The method of claim 10,
Wherein the graphical object is rendered on the 3D object in the display. 11. The method of claim 10,
Wherein the graphical object is rendered near the 3D object in the display. 11. The method of claim 10,
Wherein a plurality of regions of the 2D surface are designated as regions of interest. 11. The method of claim 10,
Wherein the known location of the region of interest is predefined. 11. The method of claim 10,
Further comprising motion sensors,
Wherein the software causes the processor to detect and track the 2D surface in the image using data from the motion sensors. 11. The method of claim 10,
Wherein the software causes the processor to detect and track the 2D surface within the image using additional images from the camera. Means for capturing an image of a two-dimensional (2D) surface having three-dimensional (3D) objects;
Means for displaying the image;
Means for detecting and tracking the 2D surface;
Means for detecting an occlusion of a region designated as a region of interest on the tracked 2D surface having a known location on the 2D surface;
Means for obtaining information about a position of the 3D object based on the detected occlusion of the region of interest;
Means for determining a shape of the 3D object; And
Means for rendering and displaying a graphics object with reference to the known location of the region of interest on the 2D surface, the graphic object being rendered for a shape of the 3D object; Including the system. delete 21. The method of claim 20,
Wherein the shape of the 3D object is predefined. 21. The method of claim 20,
Means for determining the shape of the occlusion region of the 2D surface; And
Further comprising means for determining a position of the means for capturing the image for the 2D surface,
Wherein the means for determining the shape of the 3D object utilizes a position of the means for capturing the shape of the occluded region of the 2D surface and the image relative to the 2D surface to determine the shape of the 3D object. . 21. The method of claim 20,
Wherein the graphical object is rendered on the 3D object. 21. The method of claim 20,
Wherein the graphical object is rendered near the 3D object. 21. The method of claim 20,
Wherein a plurality of regions of the 2D surface are designated as regions of interest. 21. The method of claim 20,
Wherein the known location of the region of interest is predefined. A computer readable medium comprising stored program code,
Program code for displaying an image of a two-dimensional (2D) surface having a three-dimensional (3D) object, which is captured by a camera;
Program code for detecting and tracking the 2D surface in the image;
Program code for detecting an occlusion of a region designated as a region of interest on the tracked 2D surface having a known location on the 2D surface;
Program code for determining a shape of the 3D object; And
Program code for rendering and displaying a graphical object with reference to said known location of said region of interest on said 2D surface, said graphical object being rendered against the shape of said 3D object, ≪ / RTI > delete 29. The method of claim 28,
Wherein the shape of the 3D object is predefined. 29. The method of claim 28,
Program code for determining a shape of the occlusion region of the 2D surface, and program code for determining a position of the camera relative to the 2D surface,
Wherein the program code for determining the shape of the 3D object utilizes the shape of the occluded region of the 2D surface and the position of the camera relative to the 2D surface to determine the shape of the 3D object. 29. The method of claim 28,
Wherein the graphical object is rendered on the 3D object in a display. 29. The method of claim 28,
Wherein the graphical object is rendered near the 3D object in a display. 29. The method of claim 28,
Wherein a plurality of regions of the 2D surface are designated as regions of interest. 29. The method of claim 28,
Wherein the known location of the region of interest is predefined.

Images