WO2019109106A1

WO2019109106A1 - Stereo correspondence search

Info

Publication number: WO2019109106A1
Application number: PCT/US2018/063699
Authority: WO
Inventors: James A. MCCOMBE; Christoph BIRKHOLD
Original assignee: Mine One Gmbh
Priority date: 2017-12-03
Filing date: 2018-12-03
Publication date: 2019-06-06
Also published as: EP3718051A1; EP3718051A4

Abstract

Methods, systems, devices and computer software/program code products enable efficiently finding stereo correspondence between a feature or set of features in a first image or signal, and a search domain in a second image or signal.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS AND INCORPORATION BY REFERENCE This patent: application claims the priority benefit of commonly owned U.S. Provisional Pat.

App. 62/594006 filed Dec, 3, 2017, entitled "Stereo Correspondence Search" (Attorney Docket MNE- 1 18-PR); and is a continuation-in-part of commonly owned, co-pending PCX patent app,

PC37US 16/23433 entitled "Virtual 3D Methods, Systems And Software" (Attorney Docket MNE-l l 1 - PCX), published as WO/2016/154123, which claims the priority benefit of U.S. Provisional Pat. App. 62/136494 (Attorney Docket MNB-i i l-PR),

This patent application is also related to commonly owned, co-pending PCX patent app

PCT/SJS 16/32213 entitled "Facial Signature Methods, Systems and Software" (MNE-113-PCT), published as WO/2016/.183380, which claims the priority benefit of U.S. Provisional Pat, App.

62/160563 (MNE-113-PR),

This application is also related to commonly owned, co-pending PCX patent application

PCT/11S 18/ 19243 entitled "Image .Reconstruction for Virtual 3D" (MNE-115~PCT), published as WO/20 IS/ 164852, which claims the priorit benefit of U.S. Prov Pat. App. 62/46:2307 (MNE-1 !S-PR).

This application is also related to commonly owned, co-pending PCX patent app,

PCT/US 18/48197, entitled "Visual Communications Methods, Systems And Software" {MNE-119-PCI}, which claims the priority benefit of U.S. Provisional App. 62/550685 (MNE-119-PR)

This application is also related to commonly owned, co-pending PCX patent app.

PCT/US 18/63531 entitled "Temporal De-Noising" (MNE-120-PCT), which claims the priority benefit of U.S. Provisional App. 62/593920 (MNE-120-PR)

This application is also related to commonly owned U.S. Provisional Application for Patent 62/595055 filed Dec. 5, 2017, entitled "Disparity Cache” (MNE-12 DPR).

Each of the above-listed patent applications is incorporated by reference herein as if set forth heretn in its entirety, including all text and drawings thereof.

BACKG m QF T¾£ 1 ¥£Nim

It would be desirable to provide methods, systems, devices and computer software/program code products that enable efficiently finding stereo correspondence between a feature or set of features in a first image or signal, and a search domain in a second image or signal

It would also be desirable to provide such functions at reasonable computational cost.

The present invention provides methods, systems, devices and computer software/program code produc ts that enable the foregoing aspects and others.

Although embodiments and practices of the present invention discussed herein can be used in conjunction with practices and embodiments of the Virtual 3-D ("V3D") inventions described in the above-noted, commonly -owned patent applications incorporated herein by reference, they may also he useful in other systems and in connection with other technologies, and are not limited to application in connection with the V3D inventions described in the patent applications incorporated by reference herein.

Aspects, examples, embodiments and practices of the invention, whether in the form of methods, devices, systems or computer software/program code products, will next be described in greater detail in the following Detailed Description of the Invention, in conjunction with the attached drawing figures.

Those skilled in the art will appreciate that while the following detailed description provides sufficient detail to enable one skilled in the art to practice the present invention, the various examples, embodiments and practices of the present invention that are discussed and described below, in conjunction with the attached drawing figures, are provided by way of example, and not by way of limitation. Numerous variations, additions, and other modifications or different implementations of the present invention are possible, and are within the spirit and scope of the invention.

SUM ARY OF .THE. INVENTION

The present invention provides methods, systems, devices, and computer software/program code products suitable for a wide range of applications, including, but not l imited to: facilitating video communications and presentation of image and video content in telecommunications applications; and facilitating video communications and presentation of image and video content for virtual reali ty (VR), augmented reality (AR) and head-mounted display (HMD) systems.

Methods, systems, devices, and computer software/program code products in accordance with the invention are suitable tor implementation or execution in, or in conjunction with, commercially available computer graphics processor configurations and systems including one or more display screens for displaying images, cameras for capturing images, and graphics processors for rendering images for storage or for display, such as on a display screen, and for processing data values for pixels in an image representation. The cameras, graphics processors and display screens can be of a form provided in commercially available smartphones, tablets and other mobile telecommunications devices, as well as in commercially available laptop and desktop computers, which may communicate using commercially available network architectures including client/server and chent/network/cloud architectures.

in the aspects of the invention described below and hereinafter, the algorithmic image processing methods described are executable by digital processors, which can include graphic processor units, including general purpose graphical processor units (GPGFUs) such as those commercially available on cellphones, smartphones, tablets and other commercially available telecommunications and computing devices, as well as in digital display devices and digital cameras. Those skilled in the art to which this invention pertains will understand the structure and operation of digital processors, GPGPUs and similar digital graphics processor units.

One aspect of the invention relates to methods, systems, devices and computer software/prograrn code products operable to search for correspondence between a first selected feature in a first linage and an element within a selected search domain in a second image, wherein the searching comprises: in a digital processing resource comprising a digital processor:

A) utilizing a multi-level aggregate digital data structure representative of the second image, the multi-level aggregate digital data structure comprising digital data representations at different resolutions;

8 comparing a relatively lower resolution digital date representation of the first selected feature against a relatively lower resolution digital data representation of a first selected subset of the selected search domain in the multi-level aggregate data structure, the comparing comprising computation of a minimum difference value and a maximum difference value;

C) iteratively executing the comparing (B) against other subsets of the selected search domain;

D) identifying a threshold difference value based on maximum difference values from the comparisons against subsets of the selected search domain; and

E) rejecting subsets for which the comparison results in minimum difference values that exceed the identified threshold difference value. In another aspect, the computation of a minimum difference value and a maximum difference value comprises computation of an estimated minimum difference value and an estimated maximum difference value.

Another aspect comprises iteratively executing the comparing (B), the comparing (C), the identifying (D), and the rejecting (E), «sing higher resolution digital data representations of the first selected feature and higher resolution digital data representations in the multi-level aggregate structure.

In another aspect, the first and second images are images captured by at least one camera from different viewpoints of substantially the same scene.

in another aspect, the selected search domain is selected based on the relative positions of the respective viewpoints of the first and second images.

In another aspect, the first and second images are rectified to a common projection such that selected search domain within the second image is oriented along a selected axis of the second image.

Another aspect of the invention relates to methods, systems, devices and computer

software/program code products operable to determine disparity between a first image and a second image, the determining of disparity comprising; in a digital processing resource comprising a digital processor;

A) generating a multi-level aggregate digital data structure representative of the second image, the multi-level aggregate digital data structure comprising digital data representations at: different resolutions;

B) selecting a first feature from the first image;

C) searching for a corresponding element within the second image, utilizing a selected search method, the corresponding element having correspondence with the selected feature from the first image;

D) base on results of the search, recording correspondence information relating to

correspondence between the selected feature and the corresponding element; and

E) iteratively executing the selecting (B), the searching (C) and the recording (D), using other selected features from the first image, to construct a digital map of correspondences for the selected features from the first image.

In another aspect, the selected search method comprises:

(i) comparing a relati vely lower resolution digital data representation of the first selected feat ure against a relatively lower resolution digital data representation of a first selected subset of a selected search domain in the multi-levd aggregate data structure, the comparing comprising computation of a minimum difference value and a maximum difference value;

(is) iteratively executing the comparing (i) against other subsets of the selected search domain;

(i i i) identifying a threshold difference value based on maximum difference values from the comparisons against subsets of the selected search domain; and

(iv) rejecting subsets for which the comparison results in minimum difference values that exceed the identified threshold difference value.

In another aspect, the selected search method further comprises: iteratively executing the comparing (i), the comparing (H), the identifying (Hi), and the rejecting (iv), using higher resolution digital data representations of tire first selected feature and higher resolution digital data representations in the multi-level aggregate structure; and

outputting correspondence information associated with the subset of the search domain for which the difference value is lowest.

In another aspect, the correspondence information is a transform.

In another aspect, the correspondence information is an offset representing a position within the search domain,

in another aspect, the correspondence information is 2 -dimensional value representing a position within an image

hi another aspect, the correspondence information is computed based on a known difference in viewpoints between first and second cameras that captured the first and the second image, respectively.

In another aspect, the correspondence information is a disparity value representing the distance between the location of the first selected feature in the first image and the location of the corresponding element in the second image

in another aspect, the feature is a pixel

in another aspect, the feature is a matrix of pixels

in another aspect, the matrix of pixels is a structure of adjacent pixels

in another aspect, the multi-level aggregate data structure contains dements; each lower resolution element comprising the minimum and maximum values for one or more higher resolution elements

A related, further aspect of the invention comprises methods, systems, devices and computer soft ware/program code products operable to; in a digital processing resource comprising a digital processor;

A) compare a digital data representation of the selected feature from the first image, at a given resolution, against a digital data representation of a first selected subset of a selected search domain in the second image, at a given resolution, in the mufti-level aggregate digital data structure;

B) determine, based on the comparing, a variance value for the first selected subset; and

C) utilize the variance value to determine whether to execute another comparing operation in a higher resolution digital data representation in the multi-level aggregate digital data structure, wherein, if the determined variance value is less than a selected threshold value, then not executing another comparing operation for the first selected subset in a higher resolution digital data representation in the multi-level aggregate digital data structure.

hi another aspect, the multi-level aggregate digital data structure has a configuration substantially similar to the configuration of a MTP map.

In another aspect, the comparing utilizes a selected error function

In another aspect, the comparing util izes a selected comparison function.

In another aspect the comparison function comprises a kernel . In another aspect applying the error function executing the comparing produces a scalar error value for a given transform.

In another aspect, the error function comprises a comparison function and selected logic applicable to weight results of individual comparisons.

In another aspect, the second image is pre-processed to generate a multi-level aggregate digital data structure, in which each element in a given resolution level subsumes the minimum and maximum values from the resolution level above the given resolution level.

In another aspect, the multi-level aggregate digital data structure is characterized by a selected reduction ratio.

in another aspect, only selected resolution levels of the multi-level aggregate digital data structure are created in an initial structure generating operation.

in another aspect., only selected lowest resolution levels of the multi-level aggregate digital data structure are generated in an initial structure generating operation.

In another aspect, all, or substantially all, levels of the multi-level aggregate digital data structure projected to be required in executing the comparing are generated in an initial structure generating operation.

sofiwarevprograra code products operable to search for correspondence between a first selected feature in an input signal and an element within a selected search domain in a search signal, wherein the searching comprises: in a digital processing resource comprising a digital processor:

A) utilizing a multi-level aggregate digital data structure representative of the search signal, the multi-level aggregate digital data structure comprising digital data representations at different resolutions;

8) comparing a relatively lower resolution digital data representation of the first selected feature against a relatively lower resolution digital data representation of a first selected subset of the selected search domain in the multi-ievel aggregate data structure, the comparison comprising computation of a minimum difference value and a maximum difference value;

C> iteratively executing the comparing (B) against other subsets of the selected search domain;

E) rejecting subsets for which the comparison results in minimum difference values that exceed the identified threshold difference value.

A related aspect comprises; iteratively executing the comparing (B), the comparing (€), the identifying (D), and the rejecting (E), using higher resolution digital data representations of the first selected feature and higher resolution digital data representations in the multi-level aggregate structure, and rejecting comparison results having difference values above the threshold difference value.

In another aspect, the input signal and the search signal are 1 -dimensional signals.

In another aspect, the input signal and the search signal are of different formats.

in another aspect either the input signal or the search signal comprises any of: image data, pixel depth data, 3-dimensional -point data, 3-dimensional vertex data, L1DA data, neural network weights, computer readable instructions, abstract signal descriptions, compressed representations, audio data, market data, price signal data, or analytics.

In another aspect, either the input signal or the search signal comprises a combination of any of: image data, pixel depth data, 3-dimensional point data, 3 -dimensional vertex data, LlDAR data, neural network weights, computer readable instructions, abstract signal descriptions, compressed

representations, audio data, market data, price signal data, or analytics,

In another aspect, one of the input signal or the searc h signa l comprises an abstract representation describing features of the given signal, and the other signal comprises image data.

software/program code products operable to execute variance-aware searching for correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, the searching comprising in a digital processing resource comprising a digital processor:

8) comparing a digital data representation of the first selected feature, at a given resolution, against a digital data representation of a first selected subset of the selecte search domain, at a given .resolution, in the multi-level aggregate data structure;

C) determining, based on the comparing, a variance value tor the first selected subset; and

D) utilizing the variance value to determine whether to execute another comparing operation in a higher resolution digital data representation in the multi-level aggregate structure, wherein, if the determined variance value is less than a selected threshold value, then not executing another comparing operation for the first selected subset in a higher resolution digital data representation in the multi-level aggregate structure.

in another aspect, determining a variance value comprises: utilizing a deviation calculation. in another aspect, determining a variance value comprises: utilizing a squared deviation calculation.

In another aspect, determining a variance value comprises: calculating the variance based on a running standard deviation.

In another aspect, determining a variance value comprises: utilizing a heuristic.

In another aspect, the comparing is executed for a plurality of subsets of the selected search domain, and the determining comprises utilizing the results of the comparing.

software/program code products operable to execute variance-aware searching for correspondence between a first selected feature in an input signal and an element within a selected search domain in a search signal, the searching comprising: in a digital processing resource comprising a digital processor:

A) utilizing a multi-level aggregate digital data structure representative of the search signal, the multi-level aggregate digital data structure comprising digital data representations at different resolutions; B) comparing digital data representation of the first selected feature, at a gi ven resolution, against a digital data representation of a first selected subset of the selected search domain, at a given resolution, in the multi-level aggregate data structure;

C) determining, based on the comparing, a variance value for the first selected subset; and

D) utilizing the variance value to determine whether to execute another comparing; operation in a hi gher resolution digi tal data representation in the multi-level aggregate structure, wherein, if the determined variance value is less than a selected threshold value, then not executing another comparing operation lor the first selected subset in a higher resolution digital data representation in the multi-level aggregate struc lure.

In a related aspect, the variance is the absolute value of the difference between the minimum and maximum values contained in a lower resolution element of the multi-level aggregate digital data structure.

Another aspect comprises selecting a threshold for variance below which corresponding image detail is treated as irrelevant.

in another aspect, the variance threshold is a selected fixed value.

in another aspect, the variance threshold is determined, in the processing structure, based on parameters of the images or of the devices or systems from which the images originate.

In another aspect, the variance threshold is determined based on physical properties of cameras and camera processing dements associated with a stereo image disparity search.

In another aspect, the variance threshold is determined based on a selected model.

in another aspect, the model alters the variance threshold based on selected criteria.

In another aspect, the selected criteria comprise any of: absolute signal values contained within a currently selected element of a currently selected multi-level aggregate digital data structure; overall image-representative values across larger sections of an image being searched; available processing power; human perceptual factors; display technology considerations; relative and absolute image positions; inputs from additional connected components; or user-supplied inputs or hints.

Another aspect comprises adjusting the variance threshold based on availability of computational or processing power or resources.

Another aspect comprises adjusting the variance threshold to selectively prioritize execution of computational operations.

In another aspect, multi-level aggregate digital data structure elements with highest variance are designated as candidates for further refinement based on higher resolution versions of a selected image.

Another aspect comprises iteratively repeating the designating, as variances associated with all transforms across a given image being processed become smaller, such that correlations between images are refined as available computational or processing power permits.

software/program code products operable to execute a lossless search lor correspondence between a first selected feature in a first image and an element within a selected search domain in a second image the searching comprising: in a digital processing resource comprising a digital processor:

A) utilizing a first multi-level aggregate digital data structure representative of the selected feature, the first multi-level aggregate digital data structure comprising digital data representations at different resolutions;

B) utilizing a second multi-level aggregate digital data structure representative of a first selected subset of the selected search domain, the second multi-level aggregate digital data structure comprising digital data representations at different resolutions;

C) c omparing a digital data representation of the selected feature, at a given resolution, to a digital data representation of the first selected subset of the selected search domain, at a given resolution, and computing, based on the comparing, minimum and maximum error for each eligible data element of the digital data representation at the given resolution in the multi-level aggregate digital data structure, wherein an data element is considered to be ineligible if a subsuming data element in a corresponding lower resolution was indicated to be ineligible;

D) and if the currently executed comparison was at a highest possible resolution, then determining the lowest value for the error, wherein (1 ) the error determination can incorporate error values from multiple resolutions of the multi-level aggregate digital data structure, and (2) the result of the search is a transform that produces the lowest error.

In a related aspect, features are selected from a multi-level aggregate digital data structure representat ive of the fi ts! image ; and if the currently executed comparison was not at the highest possible resolution, then accessing a next higher resolution digital data representation in the multi-level aggregate digital data structure representative of the first image to execute a subsequent comparison.

In another aspect, in executing a lossless search, aggregate down-sampled versions of the first image, represented by a currently selected multi-level aggregate digital data structure, represent the bounds of the higher resolution first image elements.

in another aspect, the aggregate down-sampled versions of the first image represent the bounds of the higher resolution first image elements in a manner analogous to the respective multi-level aggregate digital data structure representing the bounds of the second image.

In another aspect, the executed comparison function considers two ranges, wherein the first range ts the range of subsumed values from each element in the first image and the second range is the range of subsumed values from each element in the second image.

In another aspect, if the first and second ranges are disjoint, the minimum value for the error is based on the distance between the nearest points hi each range.

In another aspect, if the first and second ranges overlap the minimum error can he zero, and the maximum error can be based on the distance between the two points in each range that are furthest from each other.

softwarc/program code products operable to search for a correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, the searching comprising; in a digital processing resource comprising a digital processor;

(A) comparing a digital data representation of the selected feature from the first image against a digital data representation of a selected subset of the selected search domain in the second image;

(B) determining, for a given comparison, an error value corresponding to the given comparison, the error value being representative of the degree to which the selected feature from the first image corresponds with the selected subset of the selected search domain in the second image whe the comparing is executed in accordance with selected processing parameters;

(C) iteratively executing the comparing and determining, white applying selected changes to the processing parameters across the selected subset of the selected search domain, until processing parameters resulting in a smallest error value are found; and

(D) estimating minimum and maximum possible errors for correspondence for the selecte feature in the first image at a full resolution, by executing the comparing and determining at reduced resolution on multi-level aggregate digital data structures representative of the first image and the selected subset of the selected search domain in the second image, respectively.

In another aspect, such operations are executed to determine a transform applicable to the selected feature in the first image such that the selected feature in the first image is matchable to the selected subset of the selected search domain of the second image with a level of error lower than a selected error threshold, the determined transform corresponding to the processing parameters resulting in die smallest error value.

software/progra code products operable to search for a correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, the searching comprising; in a digital processing resource comprising a digital processor;

(A) comparing a digital data representation of the selected feature from the first image against a digital data representation of a selected subset of the selected search domain in the secon image;

(B) determining, for a given comparison, an error value corresponding to the given comparison, the error value being representative of the degree to which the selected feature from the first image corresponds with the selected subset of the selected search domain in the second image when the comparing is executed in accordance with selected processing parameters:

(C) iteratively executing the comparing and determining, while applying selected changes to the processing parameters across the selected subset of the selected search domain, until processing parameters resulting in a smallest error value are found; and

(D) utilizing, in executing the comparing, a multi-level aggregate digital data structure containing digital data representations at different resolution levels.

in another aspect, such operations are executed to determine a transform applicable to the selected feature in the first image such that the selected feature in the first image is matchable to the selected subset of the selected search domain of the second image with a level of error lower than a selected error threshold; wherein the determined transform corresponding to the processing parameters resulting in the smallest error value.

In another aspect each resolution level contains at least one element, and m which a gi ven element at a given resolution level can reference both minimum and maximum values of other elements subsumed within the given element at higher resolution levels of the multi-level aggrega te digital da ta structure.

Another aspect farther comprises: in a digital processing structure: applying correspondence searching operations to multi -dimensional data.

In another aspect, the image data has n dimensions, and is greater than or equal to 2.

in another aspect, applying correspondence searching operations to multi-dimensional image data comprises: in a digital processing structure: searching a multi-dimensional image data field using a selected comparison kernel

In another aspect, using a selected comparison kernel comprises: evaluating- the selected comparison kernel; then, based on the evaluation, associating with a representative element of the kernel, a designation of eligibility; and then utilizing the designation of eligibility to determine whether the representative element, and any containe higher resolution elements, should be refected from further consideration.

Another aspect further comprises: if a selected relatively lower resolution element is designated ineligible, rejecting from further consideration the selected lower resolution element and rejecting any corresponding subsumed higher resolution elements

Another aspect further comprises: if a selected relatively lower resolution element is designated eligible, executing comparison operations on contained higher resolution elements.

Another aspect further comprises: hi a digital processing structure; expressing an «-dimensional image data field in a form comprising fewer than n dimensions, by applying a dimension collapsing operation.

In another aspect, applying a dimension collapsing operation comprises executing a search characterized by a selected search pattern.

In another aspect, the search pattern comprises a scan line search.

Another aspect further comprises executing a search characterized by a selected search patern, wherein the search pattern comprises a scan line search

Another aspect further comprises: executing a search characterized by a selected search pattern, wherein the search pattern comprises a scan line search.

In another aspect, the search patern comprises a Hilbert curve.

in another aspect, the search pattern comprises a Morton order.

In another aspect, successive search operations are executed in multiple passes through data to be searched.

In another aspect, in one pass, evaluation of a comparison kernel finds a minimum and maximum error for the kernel

In another aspect a subsequent pass comprises determining a minimum value of the maximum error for the kernel to be used as a threshold when determining which elements are eligible for further consideration.

In another aspect., a further subsequent pass comprises comparing all eligible elements at a next resolution level

In another aspect, in a digital processing structure:

(A) the first image, representing an input image, is processed to create a first multi-level aggregate digital data structure;

(B) the second image, representing a search image, is processed to create a second multi-level aggregate digital data structure: and

(€) features from the input image and associated first multi-level aggregate digital data structure are searched within the search image and associated second multi-level aggregate digital data structure by evaluating a selected comparison kernel in parallel against as much of the search image as possible.

Another aspect further comprises: optimizing execution of processing operations to substantially fully utilize all available digital processing resources.

Another aspect further comprises:

(A) testing a selected feature of the input image against a selected search domain in the second image, an generating, based on the testing, resultant error values: and

(B) if the resultant error is value is within a selected threshold designated for further processing, then continuing processing at a next resolution level of the associated multi-level aggregate digital data structure.

Another aspect further comprises: optimizing execution of processing operations to minimize computational work and memory traffic.

In another aspect, multiple features of the input image are tested in parallel against the selected search domain.

In another aspect, processing operations are executed in accordance with digital instructions executing on a computing device.

In another aspect, the computing device comprises a GPU (General Processing Unit).

In another aspect, the computing device comprises a CPU (Central Processing Unit).

In another aspect, the computing device comprises a TPU (Tensor Processing Unit).

In another aspect, processing operations are executed by dedicated hardware circuitry.

In another aspect, data generated for use in a given processing operation are retained for future use.

In another aspect, data for use in a given processing operation are generated on an as-needed basis.

Another aspect further comprises; in a digital processing resource:

(A) generating a first multi-level aggregate digital data structure representati ve of the input image and a second multi-level aggregate digital data structure representative of the search image;

(B) selecting a first feature from the input image to locate within a selected search domain in the search image and executing a search operation for the selected first feature; and

(€) selecting an additional feature from the generated multi -level aggregate digital data structures and executing a search operation for the additional feature.

In another aspect, a search operation for the additional feature is executed after execution of the search operation for the selected first feature.

In another aspect, the search operation for the additional feature is executed in parallel with the search operation for the selected first feature.

In another aspect, the result of a given search operation is a transform indicating a best correspondence between a feature from the input image and a best match for the input image feature in the search image.

In another aspect, the transform is invertible to generate a corresponding transform between a corresponding feature in the search image and a corresponding feature in the input image.

Another aspect further comprises selecting a first portion of a gi ven image to be a search image, and selecting a second different portion of the given image to be an input image.

Another aspect further comprises; applying comparison and search processes across a plurality of images simultaneously, such that one input image is simultaneously compared for correspondence with more than one search image.

Another aspect further comprises; applying comparison and search processes across a plurality of images simultaneously, such that multiple input images are simultaneously compared for correspondence with a given search image,

in another aspect, the input image differs in format from the search image.

Another aspect further comprises; executing a transform to enable comparison between an input image and a search image of different formats,

in another aspect, the applie comparison function adaptively interprets different respective sets of data associated with each image, to enable comparison between an input image an a search image of different formats.

The present invention enables the features described herein to be provided at reasonable computational cost, and in a manner easily accommodated within the digital processing capabilities and form factors of modern mobile devices such as tablets and smartphones as well as the form factors of laptops, PCs, computer-driven televisions, computer-driven projector devices, and the like, does not dramatically alter the economies of building such devices, and is viable within current or near-current communications neiwork/connectivity architectures.

These and other aspects, examples, embodiments and practices of the invention, whether in the form of methods, devices, systems or computer software/program code products, will be discussed in greater detail below in the following Detailed Description of the invention and in connection with the attached drawing figures.

Those skilled in the art wall appreciate that while the following detailed description provides sufficient detail to enable one skilled in the art to practice the present invention, the various examples, embodiments and practices of the present invention that are discussed and described below, in conjunction with the attached drawing figures, are provided by way of example, and not by way of limitation. Numerous variations, additions, and other modifications or different implementations of the present invention are possible, and are within the spirit and scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. ! is a schematic block diagram depicting a MLADDS structure in accordance with an exemplar}' practice of the in vention.

PIG. 2 is a schematic flowchart depicting a lossless search process in accordance with an exemplary practice of the invention.

FIG. 3 depicts computational functions, in accordance with the invention, that represent one possible method for calculating minimum and maximum error values when comparing two elements, each expressing minimum and maximum signal bounds values.

FIG. 4, comprised of sub-figures 4.1 , 4.2, 4.3 and 4.4, shows exemplar}' plots of the minimum and maximum errors for a feature from an input signal compare across a search domain at four separate resolution levels.

FIG. 5 is a schematic diagram depicting operations performed, in accordance with an exemplars' practice of the invention, to search a two-dimensional (2-D) image or image signal

FIGS 6 - 22 are flowcharts depicting exemplary practices of the invention

FIG. 23 is a schematic block diagram depicting exemplar}' devices or systems in which aspects of the present invention may be practiced or embodied.

1.5 PET AlLED PESCBlFTlO OF THE I YE TIQN

Overview

The present invention relates to methods, systems, devices and computer software/prqgram products that enable efficiently finding stereo correspondence between a feature or set of features in a first signal and a search domain hi a second signal. The process of finding an acceptable correspondence results in the determination of the transformation (ofte a position or translation) for the feature from the first signal such that it matches a part of the second signal with an acceptable error. In accordance with exemplary practices and embodiments of the present invention, this is accomplished by estimating the minimum and maximum possible errors for correspondence between features at a full resolution, by performing correspondence testing at reduced resolution on aggregations of the signal data.

With regard to the terms "difference values", "error values" and "error functions”: A comparison between a feature at a given resolution and an element in a multi-level aggregate digital data structure (MLADDS) may result in a difference value indicating the difference between the feature and die element being compared. Thus, comparisons with Sow difference values can be said to have low error values, because there is a more precise match between the feature and the element. When a comparison function is applied across an entire search domain at a given resolution, it can be called an error function. A 2 -dimensional plot: of the values of the error function can be used to identi fy the elements within the search domain of the lowest error or best correspondence between the feature and the dement,

One known technique for determining stereo correspondence between an input signal and a search signal is to -perform a linear search, in such a search, a portion of the input signal is transformed and compared for errors against the corresponding part of the search signal using a comparison function or an error function. Changes to the transform parameters are iteratively applied in a linear fashion across the entire possible domain being searched, and the transform parameters resulting in the smallest error are chosen for that portion of the input signal.

In this Detailed Description of the present invention, a portion of the input signal or a processed version of the signa l for which the algorithm is searching for a corresponding match can be referred to, various, as a’’feature" or as a "snippet."

The comparison function is often a kernel. An error function is any function that produces a scalar error value for a given transform, and can include one or more comparison functions along with logic to interpret or weight the results of the individual comparisons.

Mu|ti-Resolution error function

Building on the foregoing, the linear search can be applied to multipie resolutions of the input signal and the search signal, as taught by the Applicants' commonly owned patent applications listed above and incorporated herein by reference. Further, an error function that combines the results from comparisons of the sigt _lal at more than one resolution can increase the accuracy of the overall results.

This is due to the fact that false error function minima found at a high resolution can be caused by repeating paterns.

Repeating paterns may cause a problem if the patern size is greater than the size of the comparison function or kernel. By including data from comparisons of a lower resolution version of the signal in the overall error function, the search is able to account for the proverbial’’forest" as well as the "trees."

The process by which a tower resolution signal is produced from a high resolution signal can vary, but can utilize averaging, down-sampling, or applying a low-pass filter. The lower resolution version of a signal can be referred to as an aggregate signal or a downsampled signal. The technique of parallel searching across multiple resolutions produces good results, but is computationally very costly.

An aspect of the present invention is that, unl ike executing the search operation in each resolution as an independent process, in exemplary practices of the present invention each resolution is searched serially, where the search at a higher resolution i predicated on the results of the search at the lower resolution.

Problems with eorwentional refinement search

A known optimization that can be employed is a refinement search to reduce the computational cost of the search operation. In a refinement search, the search space is sampled first at a coarse resolution, and then higher resolution samples are taken only when the error is below a threshold. This process is repeated tor successively higher resolutions until a final result is determined from samples in a high resolution version of the signal.

This is a valid optimization, but it requires a threshold value appropriate for the signal being examined. Furthermore, it susceptible to sampling errors that occur when the Nyquist frequency of the signal exceeds the frequency of the lower resolution sampling. Under-sampling can lead to the issue of potentially missing a high frequency global maximum (or error minimum), in favor of a lower frequency local maximum (or error minimum) elsewhere.

in a search signal that represents an image, this type of error might result in the search ignoring a small object because it does not make a distinct contribution to a low resolution representation of the image.

With regar to the terras "image" and“signal" as used heroin; whore this documents refers to processing of an image in accordance with the present invention, the same operations can be applied to processing of a signal in accordance with the invention, and the term "signal" can be used in place of the term "image" where this document speaks of processing of an image in accordance with the invention. Further, by way of example, such signals, which can be processed by the invention, can comprise, hut are not limited to, any of the following: image data, pixel depth data, 3-dimensional point data, 3- dtmensioaal vertex data, L1DAR data, neural network weights, computer readable instructions, abstract signal descriptions compressed representations, audio data, market data, price signal data or analytics. Multi-Level Aggregate Data Structure {MLADDS) / Bounds Structure

To solve the problems associated with a refinement search, exemplary practices of the present invention employ a multi-level aggregate digital data structure (MLADDS), of which a bounds structure is one form. A MLADDS is a multi-resolution representation of the signal in which each element at a given le vel can reference both the minimum and maximum values of the elements contained within it at all higher resol utions. Certain aspec ts of a bounds structure or MLAD DS are analogous to aspects of a class of structures known as a "MIP map." (The word MIP is an acronym of the Latin phrase "multum in parvo". which translates to English as "much in little.") The bounds structure or MLADDS can be represented as one or more MIP maps, but does not necessarily imply all of the specific limitations from the traditional use or construction of MIP maps. More particularly, as will be described herein, the bounds structure or MLADDS of exemplary practices of the present invention, and the manner in which it is employed in connection with the present invention, are distinct from and contrast with previously known uses and constructions of MΪR maps. While the term "bounds structure^*’ is also used herein, referring to one form of MLADDS, it will be understood that the invention can be embodied using various forms of MLADDS.

in an exemplary embodiment of the invention, an image (or a signal that ma be representative of an image) is preprocessed to create an MLADDS 100 like that schematically depicted in FIG. 1. Each element in a gi ven MIP level encompasses the minimum and maximum values from the MIP level abo ve it. This interval, expressed in an element of the MLADDS, therefore expresses the overall range of all of the contained elements from the original signal.

In particular, on the left-hand side of the MLADDS 100 of FIG. 1 is a column 102 setting forth discrete values from a given search signal (e.g., values 5,1, 5,3, 5/9, etc ). Moving from left to right across FIG. I , the next column 104 of FIG. i represents a MLADDS structure MIP Level 1 , followed by column 106 (MIP Level 2), and column 108 (MIP Level 3). Each bo of each MIP Level contains a Min value and a Max value representative of minimum and maximum values from the MIP level above it.

The dashed lines shown in FIG. 5 as flowing from a given MIP level to a next higher numerically- designated MIP level, indicate how two rows of values from a previous level are represented by a single row of the next higher numerically-designated level,

The example of FIG. 1 illustrates a 2 to I MIP reduction, but other MIP reductions are also valid, an may be useful, depending on tradeoffs associated with the hardware in volve in the practice of the invention.

A brief discussion about MIP level nomenclature is appropriate here. A longstanding convention in the technology of computer graphics and computer graphics programming, perhaps confusingly, refers to the highest level of a MIP structure which contains the largest number of elements as”MiP level 0" {zero). Subsequent levels of such a structure are referred to as being "lower levels," but counterintuitively they are conventionally referred to with higher numerical designations. For example, MIP level 3 is a lower level than MIP level 2. Because the higher resolution levels of a bounds structure in accordance with the present invention, or indeed any type multi-level aggregate structure, may be unneeded to search certain portions of the signal, it may be advantageous to generate only the lowest levels. However, the process of generating a MfP-map-like structure fundamentally offers the opportunity to amortize the work and memory traffic creating successive levels. This is due to the fact that there is a sizable reduction in the amount of data moving from higher resolu tion levels to lower resolution le vels. For this reason it may he more efficient to generate all neede levels of the aggregate structures) in one operation rather than beginning b generating only the lowest resolution level and producing higher levels as needed. Lossless Search using the MLADDS

One aspect of the present invention, depicted by way of example in FIG, 2, involves performing a lossless stereo search using an MLADDS. This technique avoids wasted computation by avoiding the application of the comparison function to data for which the combined error function cannot produce the best result across the search domain. The lossless search is capable of producing the same result as a multi-resolution linear search applied with tire same input signal across the same search domain

in this regard, FIG. 2 is a flowchart depicting a lossless search process 200 in accordance with an exemplary practice of the invention in particular, FIG. 2 descri bes an exemplars¹ process of lossless!}·' searching a search signal for correspondence with features from an input signal.

The left-hand side of FIG, 2 indicates the search signal 210 and input signal 212, The search signal 210 is used as an input to operation 214: create or retrieve MLADDS structure for search domain within search signal. The input signal 212 is used as an input signal to operation 216: generate or retrieve lowest resolution aggregate signal for input signal. The result or output of operation 216 is used as an input to operation 218: select feature from current resolution aggregate signal to locate within the corresponding resolution search signal.

The result or output of operation 218 is used as an input to operation 220: execute comparison function to compu te minimum and maximum error for each eligible element of the corresponding resolution in the MLADDS structure. As indicated hi box 220, in the exemplary practice depicted in FIG. 2, an element is considered to be ineligible if the containing element in the lower resolution was itself indicated to be ineligible,

The result or output of box 220 is used as an input to box 222, a determination of whether the currently executed comparison was at the highest resolution. If "Yes", then the process proceeds to operation 224: determine the lowest value for the error. As noted in box 224, in the exemplary practice depicted in FIG. 2, this error determination may incorporate error values from multiple resolutions of the aggregate map. The result or output of operation 224 yields box 226, which is the output of the lossless search process example depicted in FIG. 2: foe result of the feature search is the transform that produces the lowest error.

if the determination of box 222 {determination of whether the currently executed comparison was at the highest resolution) is "No”, then the process proceeds to operation 228: determine the lowest value for the maximum error, across every element in the MLADDS structure at the appropriate resolution.

The result or output of determination 228 leads to operation 230; indicate that each dement for which the minimum error is greater than the previously determined lowest maximum error is ineligible.

Next, the result or output of operation 230 leads to operation 232; generate or retrieve the next higher resolution aggregate signal for the input signal. The lossless search process thus loops back to bo 218; select a feature from the current resolution aggregate signal to locate within the corresponding resolution search signal; and the process iterates,

it should be noted that when this document rises the terms "iterate” or’’iteratively", this does not imply a particular order or sequence of execution, or a requirement that the multiple iterations be executed serially; and in fact, operations executed "iteratively" in accordance with the invention can be executed in parallel, by multiple processors or pipelines, and in various orders or sequences.

For a truly lossless search, the aggregate downsampled versions of the input signal must also represent the bounds of the higher resolution input signal elements, in a functionally similar way that the MLADDS structure represents the bounds of the search signal. Then the comparison function must consider two ranges; the range of contained values from each deme t in the input signal and die range of contained values from each element in the search signal if those ranges are disjoint, the minimum value for the error is based on the distance between the nearest points in each range. If those ranges overlap, the the minimum error can be zero. Similarly, the maximum error can be based on the distance between the two points in each range that are furthest from each other.

FIG. 3 is a depiction of computational functions 302, 304, in accordance with the invention, that represent one possible method for calculating minimum and maximum error values when comparing two elements, each expressing minimum and maximum_: signal bounds values. These exemplary functions 302, 304, respectively, are;

min error ~ greatest of (0.0, ( search signal min value - inpul signal max value ),

(input signal min value search signal max value ) ) max error ~ greater off absolute valm{ input signal max value - search signal min value), absolute valu ( search signal max _ value input signal min value) )

The formulas noted above are with reference to signals that range from 0 (zero) to a positive maximum value. The formulas may require appropriate adjustment for signal ranges that are permitted to cross zero.

Next, FIG 4, comprised of sub-figures 4.1 , 4 2, 4.3 and 4.4, shows exemplary plots of the minimum and maximum errors for a feature from an input signal compared across a search domain at four separate resolution levels. FIG. 4 highlights areas of the signal that are rejected from further consideration in a lossless search. By way of example, in FIG. 4.1, for comparisons at MIP level 3, the lowest value of foe

maximum error is roughly 850. Because the minimum error is zero across the entire search domain, the search must descend and compare MIP level 2 for all data points in the search domain.

In FIG. 4.2, based on comparisons at nu level 2, the minimum level for foe maximum error function across foe search domain is around 66. Therefore, all data points in foe search signal where the minimum error is higher than the threshold of 66 are ineligible, and therefore are rejected from further consideration. For areas of the search signal that are not rejected, the processing continues to MIP .level 1.

As shown in FIG. 4 3, based on comparisons at MIP level 1 , more data points can be rejected from the search domain based on foe minimum_: error exceeding the new minimum value for foe maximum error.

As shown in FIG. 4.4, the raw signal ca be compared for ail signal ele ments tha were not rejected in a previous operation.

Acceptable Error

With each increase in resolution, the resulting transform is able to express a correspondingly higher degree of precision. This is due to the increased resolution in the signal allowing finer details to be compared, producing a more accurate result. In many instances, however, this increased precision is not necessary, la some cases, the high precision portion of the transform may even be invalid, due to spurious noise in one or both of the signals. Contrastingly, a high variance in a lower resolution element indicates foe potential tor a high frequency transition or some other meaningful detail that can he correlated between the signals with high potential accuracy. (It should be noted that the term "variance” is used herein in a manner substantially consistent with its broad English definition. In particular, foe term "variance", as used herein, is NOT limited to meaning only the square of the deviation.)

Accordingly, one optimization is to use variance awareness when determining whether to continue the evaluation of the search in a higher resolution of the MLADDS structure. The variance is the absolute value of the difference between the minimum and maximum signal values contained within a lower resolution element of a ML ADDS structure. At the highest resolution of a signal, an element represents a singleton sample and therefore the variance is zero, because there is no minimum and maximum signal values that are different from foe singleton value itself in embodiments where it is possible to infinitely subdivide signal elements or manufacture new samples from an analog scarce, individual samples would not be singletons

Vai jce Awarg Search

When the variance of an element falls below a threshold, then the elemen t can be determined not to contain significant detail, and therefore a further examination of higher resolution levels of the signal will likely not produce valuable information.

By way of example when searching image data or an image data signal, a low resolution element in ie MLADDS structure containing many pixels that all display a featureless smooth background will have a very low associated variance. In such a case, ike lower precision transforms obtained from the lower resolution versions of the MLADDS structure are likely to be sufficient to adequately represent the transform.

The threshold for variance below which the detail is considered irrelevant may depend on many factors. In the simplest embodiment, the threshold ca be a fixed value, or a value calculated from parameters of the systems on which the signals originate. For example, the threshold may be based on the physical properties of the cameras and camera processing pipeline for a stereo image disparity search.

The threshold can also be based on a model. That model may alter the allowable variance threshold depending on numerous other factors and consideration including, but not limited to. one or more of the following: absolute signal values contained with the element, overall signal values across larger sections of a signal, available processing power, perceptual factors, display technology

considerations, relative and absolute image positions, input from additional connecte

components, or user supplied inputs or hints.

In one embodiment, the threshold can be adjusted to accommodate the availability of computational resource. For example, the variance at each element could function as a type of prioritization. Elements with the highest variance would then be candidates for further refinement based on higher resolution versions of the signal This would continue as the variances associated with all transforms across the signal became smaller, in this way, correlations between the signals could be refined as the available processing power allows.

Next, FIG. 5 is a diagram depicting operations performed, in accordance with an exemplary practice of the invention, to search a two-dimensional (2-D) image or image signal In particular, FIG. 5 illustrates how a multi-dimensional signal can be searched, by execution, for example in a digital processing resource, of operations 500.

The left-hand side of FIG. 5 shows a 12 x 12 table 502 schematically representing Search Signal Date: the central portion of FIG. 5 shows a 6 x 6 table 504 representing Bounds MIP Level 1; and the right-hand side of FIG. 5 shows a 3 x 3 table 506 representing Bounds MIP Level 2.

In this example, a 2-D input signal feature is compared with a 2-D search signal using a 3x3 comparison kernel 50S, designated by the dark outlines in, e.g„ the Search Signal Data table of FIG, 5. After the comparison kernel 508 is evaluated, an indication of eligibility is associated with a

representative element of the kernel - in the example depicted in FIG. 5, the center element of the kernel is used. This indication of eligibility is then used to determine whether the element, and any contained or subsumed higher resolution elements, should be rejected from further consideration.

Because the example signal of FIG. 5 uses a 2:1 MIP reduction in each dimension, four comparisons at the higher resolution mip level result from every comparison that is not rejected. If a low resolution element is inel igible, it may be rejected and all contained higher resolution elements are also rejected and therefore not compared. If the low resolution element is eligible, multiple comparisons on the higher resolution signal may be performed. In the noted example, four comparisons are performed for each eligible dement

This approach can be extended into higher dimensions, including images or signals with three or more dimensions. The number of higher resolution elements that must be tested for each lower

resolution element will depend on the dimensionality of the signal and the mip reduction

factor.

In addition, a signal comprising multiple dimensions may be expressed as a signal comprising fewer dimensions. For example, a given image may he a two dimensional signal, but searching a single scan line involves a simitar behavior to searching a one dimensional signal. Successively searching multiple scan lines still preserves the one dimensional behavior for the entire image.

A wide range of search patterns have the same dimension collapsing property, such as. for example, following a Hilbert curve or Morton order.

Implementation Options

in exemplars¹ embodiments and practices of the invention, successive search operations would occur in passes. In one pass, a comparison kernel would find the minimum and maximum error for the kernel. This would be followed by a subsequent operation to determine the minimum value of the maximum error, to be used as a threshold when considering which elements are eligible for further consideration. The next pass would then be responsible for comparing all of the eligible elements at the next raip level and so on.

In one exemplary practice of the invention, an entire image representing an input signal is processed to create a bound structure. A second image representing a search signal is also processed to create a second search MLADDS structure. Features from the input image and associated MLADDS structure are searched within the search image and associated MLADDS structure by evaluating the comparison kernel in parallel against as much of the image as possible. In this "breadth-first" approach, the algorithm is optimized to saturate all available computation elements.

in another exemplary practice of the invention, each feature from the input signal is tested. If the error is determined to be within acceptable bounds for further processing, then processing can continue to the next level of the MLADDS structure. In this "depth-first" approach, the algorithm is optimized to minimize unnecessary computational work and memory traffic. In spite of the more serial nature of this embodiment, multiple features from the input signal can be searched in parallel, so the algorithm can still be amenable to large vector processors.

The present invention can be embodied in software executing on a GPU or similar computational device, processor or resource. It could also be embodied in software executed by a CPU, software executed by a TPU (Tensor Processing Unit), or by dedicated hardware circuitry.

Data created for use in one operation may be retained for future use, or intermediate data can be created on an as-needed basis. This is implementation-specific, and does not alter the invention. For example, one implementation may construct MLADDS structures for an entire input signal and an entire search signal . Then the search operation may select a feature from the input signal to locate within the search domain in the search signal. An additional feature can be selected from the already constructed MLADDS structures and another search operation may occur subsequently or in parallel.

The result of each search operation is a transformation indicating the best correspondence between the feature, subset or snippet from the input signal and its best match in the search signal. A transformation may be inverted to produce a corresponding transformation between the feature in the search signal and the corresponding feature in the input signal. For this reason, which signal is considered the search signal and which signal is considered the input signal may be arbitrary in some

implementations. Such an inversion does not fundamentally alter the invention.

Further, some embodiments may choose parts of a signal to behave as a search signal, while parts of the same signal may behave as an input signal.

Various embodiments and practices of the invention have been described above in terms of two signals, but the invention can be extended or implemented to operate with multiple signals

simultaneously. For example, one input signal could be compared for correspondence with multiple search signals simultaneously. Alternatively, multiple input signals could be compared against the same search signal. This may be desirable in a system with more than two input devices producing signals requiring correlation, for example a device with three or more cameras.

in other embodiments, the input signal may differ substantially from the search signal in format. In one example, an input signal may comprise an abstract: representation to describe the features of the input signal while the search signal may comprise image data from a camera. Additionally either the input signal or the search signal or both may comprise one or more of: image data, neural network weights, computer readable instructions, abstrac t signal descriptions, compressed representations, or combinations of the aforementioned. In such an embodiment, the comparison function would appropriately interpret the data provided from the signal.

Digital Processing Environment In Which Invention Can Be implemented

Those skilled in the art will understand that the above described embodiments, practices and examples of the invention can be implemented using known network, computer processor and

telecommunications devices, in which the telecommunications devices can include known forms of cellphones, smartphones, and other known forms of mobile devices, tablet computers, desktop and laptop computers, and known forms of digital network components and se 'er/cloud network/client

architectures that enable communications between such devices.

Those skilled in the art will also understand that method aspects of the present invention can be executed in commercially available digital processing systems, such as servers, PCs, laptop computers, tablet computers, cellphones, smartphones and other forms of mobile devices, as well as known forms of digital networks, including architectures comprising server, cloud, network, and client aspects, for communications between such devices.

The terms "computer software," "computer code product,” and "computer program product" as used herein can encompass any set of computer-readable programs instructions encoded on a non- transitory computer readable medi um. A computer readable medium can encompass any form of computer readable element, including, but not limited to, a computer bard disk, computer floppy disk, computer-readable flash drive, computer-readable RA or ROM element or any other known means of encoding, storing or providing digital information, whether local to or remote from the cellphone, smartphone, tablet computer, PC, laptop, computer-driven television, or other digital processing device or system. Various forms of computer readable elements and media are well known hi the computing arts, and their selection is left to the imp!ementer.

in addition, those skilled in the art will understand that the invention can be implemented using computer program modules and digital processing hardware elements, including memory units and other data storage units, and including commercially available processing units, memory units, computers, servers, smartphones and other computing and telecommunications devices. The term "modules", "program modules", "components", and the like include computer program instructions, objects, components, data structures, and the like that can be executed to perform selected tasks or achieve selected outcomes. The various modules shown in the drawings and discussed in the description herein refer to computer-based or digital processor-based elements that can be implemented as software, hardware, firmware and/or other suitable components, taken separately or in combination, that provide the functions described herein, and which may be read from computer storage or memory, loaded into the memory of a digital processor or set: of digital processors, connected via a bus, a communications network, or other comraumeatiom pathways, which, taken together, constitute an embodiment of the present invention.

The terms "data storage module", "data storage element", "memory element" and the like, as used herein, can refer to any appropriate memory element usable for storing program instructions, machine readable files, databases, and other data structures. The various digital processing, memory and storage elements described herein can be implemented to operate on a single computing device or system, such as a server or collection of servers, or they can be implemented and inter-operated on various devices across a network, whether in a server-client arrangement, server-ctoud-elient arrangement, or other configuration in which client devices can communicate with allocated resources, functions or applications programs, or with a server, via a communications network,

it wtil also be understood that computer program instructions suitable for a practice of the present invention can be written in any of a wide range of computer programming languages, including Visual Basic, Java, C+ -K and the like it will also be understood that method operations shown in the flowcharts can be executed in different orders, and that not all operations shown need be executed, and that many other combinations of method operations are within the scope of the invention as defined by tire attached claims. Moreover, the functions provided by the modules and elements shown in the drawings and described in the foregoing description can be combined or sub-divided in various ways, and still be within the scope of the invention as defined by the attached claims.

The Applicants have implemented aspects of the present invention, in prototype form. One implementation comprises a complete device, including four cameras, capable of encoding content and receiving (full-duplex communication). Another is an Apple iPhone-based implementation that can receive and present immersive content (receive-oniy). The Applicants used the following hardware and software structures and tools, among others, to create the two noted implementations, collectively:

1. A seven inch 1280x800 IPS LCD display.

2. Four PomtGrey Chameleon3 fCM3~U3-13$2C-CS) 1.3 Megapixel camera modules with 1/3” sensor size assembled on. an aluminum plate with shutter synchronization circuit, 3. Sunex DSL377A-650-F/2.8 M12 wide-angle lenses.

4 An Intel Core i7-f>770HQ processor which includes on-chip the following:

a. An Intel HD Graphics 580 integrated Graphics Processing Unit; and

b. An Intel QuickSync video encode and decode hardware pipeline.

5, OpenCL API using Intel Media SDK running on Linux operating system to implement, among other aspects: image Rectification, Fast Dense Disparity Bstmiate(s) (FDDE) and Multi-level Disparity Histogram aspects. 6. OpcnGL API running on Linux operating system to implement Multiple Native Disparity Map Voting and image reconstruction.

7. Intel Media SDK to access Intel QuickSync video compression hardware.

8. PhidgetSpatial 0/0/3 Basie accelerometer module and Linux API to determine device orientation.

9 DUB Face Detection library to locate presence of viewer's face.

10, In addition, the Apple iOS SDK was used to access accelerometer, gyroscope and compass fo device orientation and to access video decode hardware; and the OpenGL BS API to implement multiple native disparity map voting and image reconstruction to enable an iPhone-based prototype of a receiving device.

Flowcharts of Exemplary Practices of the Invention

FIGS. 6 - 22 are flowcharts illustrating method aspects and exemplary practices of the invention. The methods and elements depicted in these flowcharts are examples only ; the organization, groupings, order and number of operations in the exemplary practices can be varied; and the exemplary practices and methods can be grouped, arranged or ordered differently, and include different or additional functions, whether singly or hi combination, while still being within the spirit and scope of the present invention.

Elements shown in the flowcharts in parentheses are, among other aspects, optional in a given practice of the in vention.

All the text and respective textual elements of the accompanying flowcharts are incorporated by- reference into this Detailed Description of the Invention as if set forth in their entireties in this Detailed Description of the Invention, in the respective order in which they appear in the flowcharts, while noting that the grouping, organization, order and number of operations in the exemplary practices can be varied; and the exemplary practices and methods can be arranged or ordered differently, and include different or additional functions.

In particular, FIG . 6 shows a method 6(H) for searching for correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, according to an exemplary practice of the invention, comprising the following operations; fe KX.Sgarc ls fo :^ BlPBtMBl idence between ¾ First selected feature in a first image and an element within a selected search domain in a second image:

601 . Utilize multi-level aggregate digital data structure representative of second image, die multi-level aggregate digital data structure comprising digital data representations at different resolutions;

602. Compare a relatively lower resolution digital data representation of the first selected feature against a relatively lower resolution digital data representation of a first selected subset of the selected search domain in the multi-level aggregate data structure, the comparing comprising computation of a minimum difference value and a maximum difference value;

603. Iteratively execute the comparing (602) against other subsets of the selected search domain;

604. Identify a threshold difference value based on maximum difference values from the comparisons against subsets of the selected search domain;

605. Reject subsets for which the comparison results in minimum difference values that exceed the identified threshold difference value.

FIG, 7A shows a method 700 for determining disparity between a first image and a second image, according to an exemplar practice of the invention, comprising the following operations:

700 Determining disparity between a first image and a second image: 70 . Generate a multi-level aggregate digital data structure representative of the second image, the multilevel aggregate digital data structure comprising digital data representations at different resolutions;

?02. Select a first feature from the first image;

703. Search for a corresponding element within the second image, utilizing a selected search method, the corresponding element having correspondence with foe selected feature from the first image;

704. Based on results of the search, record correspondence information relating to correspondence between the selected feature and the corresponding element;

705. Iteratively execute the selecting (702), the searching (703) and the recording (704), using other selected features from the first image, to cons truct a digital map of correspondences for the selec ted features from foe first i ase.

FIG. 7B shows a selected search method, useful, for example, in element 703 of FIG 7A (which m turn shows a method for determining disparity between a first image and a second image), according to an exemplary practice of the invention, comprising the following operations:

751 . Compare a relatively lower resolution digital data representation of the first selected feature against a relatively lower resolution digital data representation of a first selected subset of a selected search domain in the multi-level aggregate data structure, the comparing comprising computation of a minimum difference value and maximum difference value.;

752, Iteratively execute the comparing (751) against other su bsets of the selected search domain;

753. Identify a threshold difference value based on maximum difference values from the comparisons against subsets of the selected search domain;

754. Reject subsets for which the comparison results in minimu difference values that excee the identified threshold difference value;

755, Then, iteratively execute the comparing (751), the comparing ( 52), the identifying (753), and the rejecting (754), using higher resolution digital data representations of the first selected feature and higher .resolution digital data representations in the multi-level aggregate structure;

756. Output correspondence information associated with the subset of the search domain for which the difference value is lowest

FIG. 8 shows a method 800 for searching for correspondence between a first selected feature in an input signal and an element within a selected search domain in a search signal, according to an exemplary practice of the invention, comprising foe following operations: Searching for correspondence between a first selected feature in an input signal and an element

801. Utilize a multi-level aggregate digital data structure representative of foe search signal, the multilevel aggregate digital data structure comprising digital data representations at different resolutions;

802. Compare a relatively lower resolution digital data representation of the first selected feature against a relatively lower resolution digi tal data representation of a first selected subset of the selected search domain in the multi-level aggregate data structure, the comparison comprising computation of a minimum difference value and a maximum difference value;

803. Iteratively execute the comparing (802) against other subsets of the selected search domain;

804. Identify a threshold difference value based on maximum difference values from the comparisons against subsets of the selected search domain;

805. Rejecting subsets for which the comparison results in minimum difference values that excee the identified threshold difference value.

FIG. 9 shows a method 900 for variance-aware searching for correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, according to an exemplary practice of the invention, comprising the following operations:

900, Vartance-a arc searching for correspondence between a first selected feature in a first Image and an element within a selected search domain in a second image: 901. Utilize a multi-level aggregate digital data structure representative of the second image, the multilevel aggregate digital data structure comprising digital data representations at different resolutions;

902. Compare a digital data representation of the first selected feature, at a given resolution, against a digital data representa tion of a first selected subset of the selected search domain, at a given resolution, in the multi-level aggregate data structure;

903. Determine, based on the comparing, a variance value for the first selected subset; 904. Utilize the variance value to determine whether to execute another comparing operation in a higher resolution digital data representation in the multi-level aggregate structure, wherein, if the determined variance value is less than a selected threshold value, then not executing another comparing operation for the first selected subset in a higher resolution digital data representation in the multi-level aggregate structure.

FIG, 10 shows a method 1000 for variance-aware searching for correspondence between a first selected feature in an input: signal and an element within a selected search domain in a search signal, according to an exemplary practice of the invention, comprising the following operations:

IOO¾.. v ij grawiMg.me|hod of searchm .fecoiTe^n^ce betwe i.lr¾.^ <; .fegtyie. .

input signal and an element within a selected search domain in a search signal:

1001 Utilize a multi-level aggregate digital data structure representative of the search signal, the multilevel aggregate digital data structure comprising digital data representations at different resolutions;

1002. Compare digital data representation of the first selected feature, at a given resolution, against a digital data representation of a first selected subset of the selected search domain, at a given resolution, in the multi-level aggregate data structure;

1003 : determine, based on the comparing, a variance value for the first selected subset;

1004: utilize the variance value to determine whether to execute another comparing operation in a higher resolution digital data representation in the multi-level aggregate structure, wherein, if the determined variance value is less than a selected threshold value, then not executing another comparing operation for the first selected subset in a higher resolution digital data representation in the multi-level aggregate structure. FIG. 1 1 shows a method 1100 for lossless search for correspondence betwee a first selected feature in a first image and an element within a selected search domain in a second image, according to an exemplary practice of the invention, comprising the following operations:

1 100. Lossless search for correspondence between a first selected feature in a first i age and an dement within a selected search domain in a second image:

110.1. Utilize a first multi-level aggregate digital data structure representative of the selected feature, the first multi-level aggregate digital data structure comprising digital data representation at different resolutions;

1102. Utilize a second multi-level aggregate digital data structure representative of a first selected subset of the selected search domain, the second multi-level aggregate digital data structure comprising digital data representations at different resolutions;

1 103. Compare a digital data representation of the selected feature, at a given resolution, to digital data representation of the first: selected subset of the selected search domain, at a given resolution, and compute, based on the comparing, minimum and maximum error for each eligible data element of the digital data representation at the given resolution in the multi-level aggregate digital data structure, wherein an data element is considered to be ineligible if a subsuming data element in a corresponding lower resolution was indicated to be ineligible;

1 104. if the currently executed comparison was at a highest possible resolution, then determine the lowest value for the error, wherein (!) the error determination can incorporate error values from multiple resolutions of the multi-level aggregate digital data structure, and (2) the result of the search is a transform that produce the lowest error.

FIG. 12 shows a method 1200 for searching for correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, according to an exemplary practice of the in vention, comprising the following operations:

w i in a selected search domain in a second 1 mage;

1201. Compare a digital data representation of the selected feature from the first image against a digital data representation of a selected subset of the selected search domain in the second image: 1202 Determine, for a given comparison, an error value corresponding to foe given comparison, the error value bei ng representative of the degree to which the selected feature from the first image corresponds with the selected subset of the selected search domain in the second image when the comparing is executed in accordance with selected processing parameters;

1203, Iteratively execute the comparing and determining, while applying selected changes to the processing parameters across the selected subset of the selected search domain, until processing parameters resulting in a smallest error value are found;

1204, Estimate minimum and maximum possible errors for correspondence for the selected feature in the first image at a foil resolution, by executing the comparing and determining at reduced resolution on multi-level aggregate digital data structures representative of the first image and the selected subset of the selected search domain in the second image, respectively.

FIG, 13 shows a method 1300 for searching for a correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, according to an exemplary practice of the invention, comprising the fol lowing operations:

1300. Searching for a correspondence between a first selected feature in a first linage and an element within a selected search domain in a second image.

1301. Compare a digital data representation of the selected feature from the first image against a digital data representation of a selected subset of the selected search domain in the second image;

1302. Determine, for a given comparison, an error value corresponding to foe given comparison, the error value being representati ve of the degree to which the selected feature from the first image corresponds with the selected subset of the selected search domain in the second linage when the comparing is executed in accordance with selected processing parameters;

1303. Iteratively execute the comparing and determining, while applying selected changes to the processing parameters across the selected subset of the selected search domain, until processing parameters resulting in a smallest error value are found;

1304. Utilize, in executing the comparing, a multi-level aggregate digital data structure containing digital data representations at different resolution levels. FIG. 14 depicts additional aspects relating to die MLADDS, searching, and other aspects of the invention, as follows:

(1401 , MLADDS has configuration substantially similar to a MIP map.)

(1402. The comparing can utilize a selected error .function.)

(1403. The comparing can utilize a selected comparison function.)

(1404 The comparison function can comprise a kernel )

(1405, Apply error function in executing the comparing produces a scalar error value for a given transform.)

(1406. Error function comprises a comparison function and selected logic applicable to weight results of individual comparisons. )

(1407 Second image is pre-processed to generate a MLADDS structure in which each element in a given resolution level subsumes the minimum and maximum values from the resolution level above the given resolution level. )

(1408. MLADDS is characterized by a selected reduction ratio.)

(5409. Only selected resolution levels of the MLADDS are created in an initial structure generating operation.) (1410. Only selected lowest resolution levels of the MLADDS structure are generated in an initial structure generating operation.)

(141 1. All levels of the MLADDS projected to be required in executing the comparing are generated in an initial structure generating operation.)

( 1412, Ail levels of the MLADDS are generated in an initial, digital aggregation data structure generating operation.) (1412. Execute a lossless search operation using the MLADDS structure.)

FIG. .15 depicts a further aspect 1500 of processing in which:

(1501 , if the currently executed comparison was not at: the highest possible resolu tion, searching can further comprise:

1501.1 determining the lowest value for the maximum error across every element in the

MLADDS at the currently selected resolution;

1501.2 indicating that each element for which the minimum error is greater than the determined lowest maximum error is ineligible:

1501.3 access a next higher resolution MLADDS representative of the first image; and

1501.4 returning to selecting a feature from the current resolution MLADDS structure to locate within the corresponding resolution MLADDS representative of the second image.)

FIG. 16 shows additional aspects relating to lossless search and other aspects, as follows:

(1601. In executing a lossless search, aggregate down-sampled versions of the first image, represented by a currently selected bounds structure, represent the bounds of the higher resolution first image elements)

(5602, Aggregate down-sampled versions of the first image represent 1¾e bound of the higher resolution first image elements in a manner analogous to the respective bounds structure representing the bounds of the second image)

(1603, Executed comparison function considers two ranges, wherein the first range is the range of subsumed values from each element in the first image and the second range is the range of subsumed values from each element in the second image)

(1604. If the first and second ranges are disjoint, the minimum value for the error is based on the distance between the nearest points in each range)

(1605. If the first and second ranges overlap, the minimum error can be zero, and the maximum error can be based on the distance between the two points in each range that are furthest from each other)

(1606. Applying variance awareness to determine whether to continue execution of search operations in a next-higher resolution level of a bounds structure)

(1607, Variance is defined as the absolute value of the difference between the minimum and maximum values contained in a lower resolution element of the bounds struc ture)

(1608 Select threshold for variance below which corresponding image detail is considered irrelevant)

(1609 Variance threshold is a selected fixed value)

(1610 Variance threshold is determined, in the processing structure, based on parameter of the images or of the devices or systems from which the images originate)

(161 1 Variance threshold is determine based on phy sical properties of cameras and camera processing elements associated with a stereo image disparit search)

( 1612 Variance threshold is determined based on a selected model)

(1613 Model alters the variance threshold based on selected criteria)

(1614, Selected criteria comprise any of absolute signal values contained within a currently selected element of a currently selected bounds structure; overall image-representative values across larger sections of an image being searched; a vailable processing power; human perceptual factors; display technology considerations; relative and absolute image positions; inputs from additional connected components; or user-supplied inputs or hints),

P1.G. 17 shows features relating to variance threshold and other aspects, as follows:

1700: Variance Threshold, other aspects

(1701. Adjust the variance threshold based on availability of computational or processing power or resources);

( 1702, Adjust variance threshold to selectively prioritize execution of computational operations); (1703. M LADDS elements with highest: variance are designated as candidates for further refinement based on higher resolution versions of a selected image);

(1704. iteratively repeal the designating as variances associated with ah transforms across a given image bein processed become smaller, such that correlations between images are refined as available computational or processing power permits).

FIG 18 shows features relating to multi -dimensional signal or image data and other aspects, as follows;

1800: Multi-dimensional aspects other aspects

(1801 , Apply correspondence searching operations to multi-dimensional image data)

(1802. Image data has n dimensions, and n is equal to or greater than 3)

( 1803. Applying correspondence searching operations to multi-dimensional image data comprises; in a digital processing structure, search a multi-dimensional image data field using a selected comparison kernel )

(1804, Using a selected comparison kernel comprises; evaluate the selected comparison kernel; then, base on the evaluation, associating, with a representative element of the kernel, designation of eligibility; and then utilize the designation of eligi bility to determine whether the representative element, and any contained higher resolution elements, should be rejected from further consideration)

( 1805, If a selected relatively lower resolution element is designated ineligible, rejecting the selected lower resolution element and rejecting any corresponding subsumed higher resolution elements)

(1 06, If a selected relatively lower resolution element is designated eligible, execute comparison operations on contained higher resolution elements.)

(1807. Express an «-dimensional image data field in a form comprisin fewer than n dimensions, by applying a dimension collapsing operation)

( 1808, Applying a dimension collapsing operation comprises executing a search characterized by a selected search pattern) (1809. Search pattern comprises a scan Hoe search.)

(1810. Search pattern comprises a Hilbert curve.)

(1 11 , Search patern comprises a Morton order.)

(1812. Successive search operations are executed in multiple passes through data to be searched.) (18.13. Evaluation of a comparison kernel finds the minimum and maximum error for the kernel.)

(1814. Subsequent pass comprises determinin the minimum value of the maximu error for the kernel, to be used as a threshold when determining which elements are eligible for further consideration.)

(1815, Further subsequent pass comprises comparing all eligible elements at a next resolution level.)

FIG. 19 shows features relating to execution of processing operation and other aspects, as follows:

1909: Execution of Processing Operations: Other Aspects

(1901 , First image, representing an input image, is processed to create a first bounds structure; second image, representing a search image, is processed to create a second bounds structure; and features from the input image and associated first bounds structure are searched within the search image and associated second bounds structure by evaluating a selected comparison kernel in parai!el against as much of the search image as possible)

{ 1 02, Optimize execution of processing operations to substantially fully utilize all available digital processing resources)

(1 03, Tests selected feature of the input image against a selected search domain in the second image, and generate therefrom, resultant error values; and if the resultant error is value is within a selected threshold designated for further processing, then continuing processing at a next resolution level of the associated bounds structure)

( .1 04, Optimize execution of processing operations to minimize computational work and memory traffic.) (1905. Multiple features of the input image are tested in parallel against the selected search domain.)

(1906. Processing operations are executed in accordance with digital instructions executing on

comp uiing device . )

(1907. Computing device comprises a GPU; a CPU; or a TFU).

(1908. Processing operations are executed by dedicated hardware circuitry )

( 1909. Data generate for use in a given processing operation are retained for future use.)

(1910 Data for use in a given processing operation are generated on an as-needed basis.)

FIG. 20 shows additional aspects, as follows;

2000. Additional Aspects:

(2001. Generate first MLADDS representative of the input tmage and a second MLADDS representative of the search image; selecting a fi rst feature from the input image to locate within a selected search domain in the search image and executing a search operation for the selected first feature; and select additional feature from the generated structures and execute a search operation for the additional feature)

(2002. Search operation for the additional feature is executed after execution of the search operation for the selected first feature.)

(2003. Search operation for the additional feature is executed in parallel with the search operation for the selected first feature.) (2004. Result of a given search operation is a transform indicating a best correspondence between a feature from the input image and a best match for the input image feature in the search image.)

(2005. Transform is invertible to generate a corresponding transform between a corresponding feature in the search image and a corresponding feature in the input image.)

(2006, Selecting a first portion of a given image to he a search image, and selecting a second, different portion of the given Image to be an Input image.) (2007. Applying comparison and search processes across a plurality of images simultaneously, such that one inpu image is simultaneously compared for correspondence with more than one search image.)

(2008. Applying comparison and search processes across a plurality of images simultaneously, such that multiple inpu images are simultaneously compared for correspondence with a given search image.)

(2009, Input image differs in format from the search image.)

(2010. Input signal and search signal can be 1 dimensional signals.)

(201 1. Executing a transform to enable comparison between an input image and a search image of different formats.)

(2012 Applied comparison function adaptively interprets different respective sets of data associated with each image, to enable comparison between an inpot image and a search image of different formats,)

FIG. 2 f shows features relating to input and search imagcs/signals, as follows:

: Search image/signa|

(2101 , Comparison function enables comparison of an input signal and a search signal.)

(2102. Input signal and the search signal are of different formats.)

(2103. Either the input signal or the search signal comprises any of: image data, pixel depth data, 3- dirnensionai point data, 3 -dimensional vertex data, LIDAR data, neural network weights, computer readable instructions, abstract signal descriptions, compressed representations, audio data, market data, price signal data, or analytics.)

(2104, Either the inpu signal or the search signal comprises a combination of any of: image data, pixel depth data, 3-dimensional point data, 3-dimensional vertex data, LIDAR data, neural network weights, computer readable instructions, abstract signal descriptions, compressed representations, audio data, market: data, price signal data, or analytics.)

(2105. One of the input signal or the search signal comprises an abstract representation descri bing features of the given signal, and the other signal comprises image data.) FIG. 22 shows additional aspects of exemplary practices of the invention, as follows:

2200. Additional aspects:

(2201. The computation of a minimum difference value and a maximum difference value can comprise computation of an estimated minimum difference value and an estimated maximum difference value.)

(2202. Executing search and further: iteratively executing the comparing and rejecting aspects, using higher resolution digital data representations of the first selected feature and higher resolution digital data representations in the multi-level aggregate structure)

(2203, The first and second images are images captured by at least one camera from different vie points of substantially the same scene.)

(2204. The selected search domain is selec ted based on the relative positions of the respective viewpoints of the first and second images.)

(2205 The first and second images are rectified to a common projection such that selected search domain within the second image is oriented along a selected axis of the second image)

(2206. The correspondence information can be a transform)

(2207. The correspondence information can be an offset representing a position within the search domain.)

(2208, The correspondence information can be a 2-dimensional value representing a position within an image.)

(2209, The correspondence information can he computed based on a known difference in viewpoints between first and second cameras that captured the first and the second image, respectively.)

(2210. The correspondence information can be a disparity value representing the distance between the location of the first selected feature in the first image and the location of the corresponding element in the second image.) (2211. The selected feature can be a pixel)

(2212. The selected feature can be a matrix of pixels.)

(2213. The matrix of pixels can be a structure of adjacent pixels.)

(2214. The multi-level aggregate data structure contains elements; each lower resolution element comprising the minimum and maximum values for one or more higher resolution elements) (2215. Process can further include comparing a digital data representation of the selected feature from the first image, at a given resolution, against a digital data representation of a first selected subset of a selected search domain in the second image, at a given resolution, in the multi-level aggregate digital data structure; determining, based on the comparing, a variance value for the first selected subset; and utilizing the variance value to determine whether to execute another comparing operation in a higher resolution digital data representation in the multi-level aggregate digital data structure, wherein, if the determined variance value is less than a selected threshold value, then not executing another comparing operation for the first selected subset in a higher resolution digital data representation in the multi-level aggregate digital data structure.) (2216, Process can further include iteratively execute the comparing, identifying and rejecting functions, using higher resolution digital data representations of the first selected feature and higher resolution digital data representations in the multi-level aggregate structure, and rejecting comparison results having difference values above the threshold difference value).

Block Diagram of Exemplary Embodiment! s) of the Invention

FIG. 23 is a schematic block diagram depicting exemplary devices or systems in which aspects of the present invention may be practiced or embodied.

in particular, FIG. 23 is a schematic block diagram showing a device or system 2300 in which the invention may be practiced. The device or system 2300 may be implemented «sing known forms of di gital processing hardware, such as known forms of smartphones, tablets and/or other forms of digital processing and imaging devices, supplemented a necessary in accordance with the teachings of the present invention. Arrows in FIG. 23 indicate flow of digital data and/or signals between elements.

By way of example, device of system 2300 can comprise at least a first digital camera or camera pipeline 2302, a second digital camera or camera pipeline 2304, and a digital processing resource 2306 comprising one or more digital processors) 2308, memory elements) 2310, bidler(s) 2312, other storage 2314, and interfaces 2316 to external devices and/or networks. In accordance with the teachings of the invention, such as discussed above, the digital processing resource 2306 is operable to receive digital data from the cameras or camera pipelines 2302, 2304, process the data in accordance with the invention, and provide outputs, based on such processing, to internal (t.e., within device or system 2300} display element 2330 or storage 2320: and/or to external display, storage or network elements (collectively 2350).

The externa! display, storage or network elements 2350 may comprise the Internet devices, processors or other networks connected to or via the Internet, or other network-connected elements or destinations.

in addition, the digital processing resource may receive or consume digital information from such a network or networks 2350, such as for processing by the digital processing resource 2306.

Conclusion

While the foregoing description and the accompanying drawing figures provide detail that will enable those skilled in the art to practice aspects of the invention, it should be recognized that the description is illustrative in nature and that many modific ations and variations thereof will be apparent to those skilled in the art having the benefit of these teachings. It is accordingly intended that the invention herein be defined solely by any claims that may be appended hereto and that the invention he interpreted as broadly as permitted by the prior art.

Claims

We claim: i . A method of searching for correspondence between a first selected feature in a first image and an dement within a selected search domain in a second image, the method comprising:

in a digital processing resource comprising a digital processor:

B) c omparing a relatively lo wer resolution digital data representation of the first selected feature against a relatively lower resolution digital data representation of a first selected subset of the selected search domain in the multi-level aggregate data structure, the comparing comprising computation of a minimum difference value and a maximum difference value;

D) Identifying a threshold difference value based on maximum difference values from the compari sons against subsets of the selected search domain; and

2. The method of claim 1 wherein the computation of a minimum difference value and a maximum difference value comprises computation of an estimated minimum difference value an an estimated maximum difference value

3 The method of claim i further comprising:

iteratively executing the comparing (B), the comparing (C), the identifying (D), and the rejecting (E), using higher resolution digital data representations of the first selected feature and higher resolution digital data representations in the multi-level aggregate structure.

4. The method of claim I wherein the first and secon images are images captured by at least one camera from different viewpoints of substantially the same scene.

5. The method of claim 4 in which the selected search domain is selected based on the relative positions of the respective viewpoints of the first and second images.

6. The method of claim 4 in which the first and second images are rectified to a common projection such that selected search domain within the second image is oriented along a selected axis of the second image.

? A method for determining disparity between a first image and a second image, tire method comprising:

in a digital processing resource comprising a digital processor:

A) generating a multi-level aggregate digital data structure representative of the second image, the multi-level aggregate digital data structure comprising digital data representations at different resolutions;

B) selecting a first feature from the first image;

C) searching for a corresponding element within the second image utilizing a selected search method, the corresponding element having correspondence with the selected feature from the first image;

D) based on results of the search, recording correspondence information relating to

correspondence between the selected feature and the corresponding element;

£} iteratively executing the selecting (B), the searching (€) and the recording (D), using other selected features from the first image, to construct a digital map of correspondences for the selected features from the first image.

8. The method of claim 7 wherein the selected search method comprises:

(i) comparing a relatively lower resolution digital data representation of the first selected feature against a relatively lower resolution digital data representation of a first selected subset of a selected search domain in the multi-level aggregate data structure, the comparing comprising computation of a minimum difference value and a maximum difference value;

fii) iteratively executing the comparing (i) against other subsets of the selected search domain; fiii) identifying a threshold difference value based on maximum difference values from the compari sons aga inst subsets of the selected search domain; and

9. The method of el aim 8 wherein the selected search method further comprises:

iteratively executing the comparing (i), the comparing (ii), the identifying (tit), and the rejecting (iv), using higher resolution digital data representations of Hie first selected feature and higher resolution digital data representations in the multi-level aggregate structure; and

10. The method of claim 9 wherein the correspondence information is a transform.

1 1. The method of claim 9 wherein the correspondence informa t ion is an offset representing a position within the search domain.

12. The method of claim 9 wherein the correspondence information is a 2-dimensional value representing a position within an image.

13. The method of claim 9 wherein the correspondence information is computed based on a known difference in viewpoints between first and second cameras that captured the first and the second image, respectively

14. The method of claim 9 wherein the correspondence information is a disparity value representing the distance between the location of tire first selected feature in the first image and the location of the corresponding element is the second image.

15. The method of claim 8 where the feature is a pixel.

16. The method of claim 8 where the feature is a matrix of pixels.

17. The method of claim 16 wherein the matrix of pixels is a structure of adjacent pixels,

18. The method of claims 1, 3, 7, 8 or 9 wherein the multi-level aggregate data structure contains elements; each lower resolution element comprising the minimum and maximum values for one or more higher resolution elements.

19. The method of claim 7 further comprising;

in a digital processing resource comprising a digital processor;

A) comparing digital data representation of the selected feature from the first image, at a given resolution, against a digital data representation of a first selected subset of a selected search domain in the second image, at a given resolution, in the multi-level aggregate digital data structure;

B) determining, based on the comparing, a variance value for the first selected subset; and

C) utilizing the variance value to determine whether to execute another comparing operation in a higher resolution digital data representation in the multi-level aggregate digital data structure, wherein, if the determined variance value is less than a selected threshold value, then not executing another comparing operation for the first selecte subset in a higher resolution digital data representation in the multi-level aggregate digital data structure.

20. The method of claims 1, 3_» 7, 8 or 9 wherein the multi-level aggregate digital data structure has a configuration substantially similar to the configuration of a MIT map.

21. The method of claim 20 wherein the comparing utilizes a selected error function.

22. The method of claim 20 wherein the comparing utilizes a selected comparison function.

23. The method of claim 22 wherein the comparison function comprises a kernel.

24. The method of claim 21 wherein applying the error function in executing the comparing produces a scalar error value for a gi ven transform.

25. The met hod of claim 21 wherein the error function comprises a comparison function and selected logic applicable to weight results of individual comparisons.

26. The method of claim 20 wherein the second image is pre-processed to generate a multi-level aggregate digital data structure, in which each element in a given resolution level subsumes the minimum and maximum values from the resolution level above the given resolution level.

27. The method of claim 26 wherein the multi-level aggregate digital data structure is characterized by a selected redaction ratio.

28. The method of claim .1 , 3, 7, 8 or 9 wherein only selected resolution levels of the multi-level aggregate digital data structure are created in an initial structure generating operation.

29. The method of claim 28 wherein only selected lowest resolution levels of the multi-level aggregate digital data structure are generated in an initial structure generating operation.

30. The method of claim 27 wherein ail levels of the multi-level aggregate digital data structure projected to be required in executing the comparing are generated in an initial structure generating operation.

35. A method of searching for correspondence between a first selected feature in an input signal and an element within a selected search domain in a search signal, the method comprising;

in a digital processing resource comprising a digital processor:

A) utilizing a multi-level aggregate digital data structure representative of foe search signal, foe multi-level aggregate digital data structure comprising digital data representations at different resolutions; B) comparing a relatively lower resolution digital data representation of the first selected feature against a relatively lower resolution digital data representation of a first selecte subset of the selected search domain in the mn! ti-levei aggregate data structure, the comparison comprising computation of a minimum difference value and a maximum difference value;

C) iteratively executing the comparing (B) against other subsets of the selecte search domain;

D) identifying a threshold difference value based on maximum difference values from the comparisons against s ubsets of the selected search domain ; and

32. The method of claim 31 further comprising:

iteratively executing the comparing (B), the comparing (C), the identifying (D), and the rejecting (B), using higher resolution digital data representations of the first selected feature and higher resolution digital data representations in the multi -level aggregate structure, and rejecting comparison results having difference values above the threshold difference value.

33. The method of claim 31 wherein the input signal and the search signal are 1 -dimensional signals.

34. The method of claim 31 wherein the input signal and the search signal are of different formats.

35. The method of claim 31 wherein either the input signal or the search signal comprises any of image data, pixel depth data, 3-dimenstonal point data, 3-dimensional vertex data, L1DAR data, neural network weights, computer readable instructions, abstract signal descriptions, compressed representations, audio data, market data, price signal data, or analytics,

36. The method of claim 34 -wherein either the input signal or the search signal comprises a combination of any of: image data, pixel depth data, 3 -dimensional point data, 3-dimensional vertex data, L1.DAR data, neural network weights, computer readable instructions, abstract signal descriptions, compressed representations, audio data, market data, price signal data, or analytics.

37. The method of claim 34 wherei n one of the input: signal or the search signal comprises an abstract representation describing features of the given signal, and the other signal comprises image data,

38, A variance-aware method of searching for correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, the method comprising:

in a digital processing resource comprising a digital processor:

A) utilizing a multi-level aggregate digital data structure representative of the second image, the multi-level aggregate digital data structure comprising digital data representations at. different resolutions;

B) comparing a digital data representation of the first selected feature, at a given resolution, against a digital data representation of a first selected subset of the selected search domain, at a given resolution, in the multi-level aggregate data structure;

C) determining, based on the comparing, a vari nce value for the first selected subset; and

D) utilizing the variance value to determine whether to execute another comparing operation in a higher resolution digital data representation in the multi -level aggregate structure, wherein, if the determined variance value is less than a selected threshold value, then not executing another comparing operation for the first selected subset in a higher resolution digital data representation in the multi-level aggregate structure.

39. The method of claim 38 in which determining a variance value comprises: utilizing a deviation calculation.

40. The method of claim 38 in which determining a variance value comprises utilizing a s uared deviation calculation.

4.1. The method of claim 38 in which determining a variance value comprises: calculating the variance based on a running standard deviation.

42. The method of claim 38 in which determining a variance value comprises: utilizing a heuristic.

43. The method of claim 38 in which the comparing is executed for a plurality of subsets of the selected search domain, and the determining comprises utilizing the results of the comparing.

44. A variance-aware method of searching for correspondence between a first selected feature in an input signal and an element within a selected search domain in a search signal, the method

comprising:

in a digital processing resource comprising a digital processor;

A) utilizing a multi-level aggregate digits! data structure representative of the search signal, the multi-level aggregate digital data structure comprising digital data representations at different resolutions;

B) comparing digital data representation of the first selected feature, at a given resolution, against a digital data representation of a first selected subset of the selected search domain, at a given resolution, in the multi-level aggregate data structure;

C) determining, based on the comparing, a variance value for the first selecte subset; and

D) utilizing the variance value to determine whether to execute another comparing operation in a higher resolution digital data representation in the multi-level aggregate structure, wherein, if the determined variance value is less than a selected threshold value, then not executing another comparing operation for the first selected subset in a higher resolution digital data representation in the multi-level aggregate structure,

45. The method of claim 44 wherein the variance is the absolute value of the difference between the minimum and maximum values contained in a lower resolution element of the multi-level aggregate digital data structure.

46. The method of claim 45 further comprising: selecting a threshold for variance below which corresponding image detail is treated as irrelevant

47. The method of claim 45 wherein the variance threshold is a selected fixed value.

48. The method of claims 46 or 47 wherein the variance threshold is determined, in the processing structure, based on parameters of the images or of the devices or systems from which the images originate.

49. The method of claim 48 wherein the variance threshold is determined based on physical properties of cameras and camera processing elements associated with a stereo image disparity search.

50. The method of claim 46 where in the variance threshold is determined based on a selected model.

51. The method of claim 50 wherein the model alters the variance threshold based on selected criteria

52. The method of claim 51 wherein the selected criteria comprise any of: absolute signal values contained within a currently selected element of a currently selected multi-level aggregate digital data structure; overall image-representative values across larger sections of an image being searched;

available processing power; human perceptual factors; display technology considerations; relative and absolute image positions; inputs from additional connected components; or user-supplied inputs or hints,

53. The method of claim 45 further comprising: adjusting the variance threshold based on availability of e-oraputational or processing power or resources

54. The method of claim 45 further comprising: adjusting the variance threshold to selectively prioritize execution of computational operations.

55. The method of claim 54 wherein multi-level aggregate digital data structure elements with highest variance are designated as candidates for further refinement based on higher resolution versions of a selected linage.

56. The method of claim 55 further comprising: iteratively repeating the designating, as variances associated with all transforms across a given image being processed become smaller, such that correlations between images are refined as available computational or processing power permits.

57. A method of executing a lossless search for correspondence between a ftrst selected feature in a first image and an element within a selected search domain in a second image, the method comprising:

in a digital processing resource comprising a digital processor:

A) utilizing a first multi-level aggregate digital data structure representative of the selected feature, the first multi -level aggregate digital data structure comprising digital data representations at different resolutions;

C) comparing a digital data representation of the selecte .feature, at a given resolution, to a digital data representation of the first selected subset of the selected search domain, at a given resolution, and computing, base on the comparing, minimum and maximum error for each eligible dat element of the digital data representation at the given resolution in the multi-level aggregate digital data structure, wherein an data element is considered to be ineligible if a subsuming data element in a corresponding lower resolution was indicated to be ineligible;

!.)) and if the currently executed comparison was at a highest possible resolution, then determining the lowest value for the error, wherein (1) the error determination can incorporate error values from multiple resolutions of the multi-level aggregate digital data structure, and (2) the result of the search is a transform that produces the lowest error.

58. The method of claim 57 wherein features are selected from a multi-level aggregate digital data structure representative of the first linage; an i f the currently executed comparison was not at the highest possible resolution, then accessing a next higher resolution digi tal data representation in the multi-level aggregate digital data structure representative of the first image to execute a subsequent comparison.

59. The method of claim 58 wherein, in executing a lossless search, aggregate downsampled versions of the first image, represented by a currently selected multi-level aggregate digital data structure, represent the bounds of the higher resolution first image elements.

60. The method of claim 59 wherein the aggregate downsampled versions of the first image represent the hounds of the higher resolution first image elements in a manner analogous to the respective multi-level aggregate digital data structure representing the bounds of the second image.

61. The metho of claim 59 wherein the executed comparison function considers two ranges, wherein the first range is the range of subsume values from each element in the first image an the second range is the range of subsumed values from each element in the secon image.

62. The method of claim 61 wherein, if the first and second ranges are disjoint:, the minimum value for the error is based on the distance between the nearest points in each range.

63. The method of claim 61 wherein, if the first and second ranges overlap, the minimum error can be zero, and the maximum error can be based on the distance between the two points in each range that are furthest from each other.

64 A method of searching for a correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, the method comprising;

in a digital processing resource comprising a digital processor;

comparing a digital data representation of the selected feature from the first image against a digital data representation of a selected subset of the selected search domain in the second image;

determining, for a given comparison, an error value corresponding to the given comparison, the error value being representative of the degree to which the selected feature from the first image corresponds with the selected subset of the selected search domain in the second image when the comparing is executed in accordance with selected processing parameters; and

iteratively executing the comparing and determining, while applying selected changes to the processing parameters across the selected subset of the selected search domain, until processing parameters resulting in a smallest error value are found; and

estimating minimum and maximum possible errors for correspondence for the selected feature in the first image at a full resolution, by executing the comparing and determining at reduced resolution on multi-level aggregate digital data structures representative of the first image and the selected subset of the selected search domain in the second image, respectively.

65. The method of claim 64 executed to determine a transform applicable to the selected feature m the first image such that the selected feature in the first image is matehabfe to the selected subset of the selected search domain of the second image with a level of error lower than a selected error threshold, the determined transform corresponding to the processing parameters resulting in the smallest error value.

66. A method of searching for a correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, the method comprising;

in a digital processing resource comprising a digital processor;

determining, for a gi ven comparison, an error value corresponding to the given comparison, the error value being representative of the degree to which the selected feature from the first image corresponds with the selected subset of the selected search domain in the second image when the comparing is executed in accordance with selected processing parameters; and

utilizing, in executing the comparing, a multi-level aggregate digital data structure containing digital data representations at different resolutio levels. 67. The method of claim 66 executed to determine a transform applicable to the selected feature in the first image such that the selected feature in the first image is matchable to the selected subset of the selected search domain of the second image with a level of error lower than a selected error threshold; wherein the determined transform corresponding to the processing parameters resulting in the smallest error value.

68. The method of claim 66 wherein each resolution level contains at least one element, and in which a given element at a given resolution level can reference both minimum and maximum values of other elements subsumed within the given element at higher resolution levels of the multi-level aggregate digital data structure.

69. The method of claims 1 , 3. 7, 8, 9 or 68, further comprising:

in a digital processing structure:

applying correspondence searching operations to multi“dimensional data.

70. The metho of claim 69 wherein the image data has n dimensions, and n is greater than or equal to 2.

7.1. The method of claim 70 wherein applying correspondence searching operations to multidimensional image data comprises:

in a digital processing structure:

searching a muiti-di ncnsional image data field using a selected comparison kernel.

72 The method of c laim 71 wherein using a selected comparison kernel comprises: evaluating the selected comparison kernel; then, based on the evaluation, associating, with a representative dement of the kernel, a designation of eligibility; and then utilizing the designation of eligibility to determine whether the representative element, and any contained higher resolution elements, should he rejected from further consideration.

73. The method of claim 72 further comprising, if a selecte relatively lower resolution element is designated ineligible, rejecting from further consideration the selected lower resolution element and rejecting any corresponding subsumed higher resolution elements,

74. The method of claim 72 further comprising: if a selected relatively tower resolution element is designated eligible, executing comparison operations on contained higher resolution elements.

75. The method of claim 74 further comprising:

in a digital processing structure;

expressing an «-dimensional image data field in; a form comprising fewer than « dimensions, by applying a dimension collapsing operation.

76. The method of claim 75 wherein applying a dimension collapsing operation comprises executing a search characterized by a selected search patern.

77. The method of claim 76 wherein the search pattern comprises a scan line search.

78. The method of any of claims 38 or 44 further comprising: executing a search characterized by a selected search pattern, wherein the search pattern comprises a scan line search.

79. The method of any of claims l , 7 31 or 57 further comprising; executing a search

characterized b a selected search pattern, wherein the search pattern comprises a scan line search.

80. The method of claim 76 wherein the search pattern comprises a Hilbert curve.

81. The method of claim 76 wherein the search pattern comprises a Morton order.

82. The metho of claims i or 81 wherein successive search operations are executed in multiple passes through data to he searched.

83. The method of claim 82 wherein, in one pass, evaluation of a comparison kernel finds a minimum and maximum error for the kernel.

84. The method of claim 83 wherein a subsequent pass comprises determining a minim u value of the maximum error for the kernel , to be used as a threshold when determining which elements are eligible for further consideration,

85. The method of claim 84 wherein a further subsequent pass comprises comparing all eligible dements at a next resolution level.

86. The method of claims l, 3, 7, 8, 9, 31, 38, 44, 57, 64 wherein;

in a digital processing structure;

the first image, representing an input image, is processed to create a first multi-level aggregate digital data structure;

the second image, representing a search image, is processed to create a second multi-level aggregate digital data structure; and

features from: the input image and associated first multi-level aggregate digital data structure are searched within the search image and associated second multi-level aggregate digital data structure by evaluating a selecte comparison kernel in parallel against: as much of the search image a possible.

87. The method of clai 86 further comprising: optimizing execution of processing operations to substantially fully utilize all available digital processing resources.

88. The method of claim 86 further comprising:

testing a selected feature of the input image against a selected search domain in the second image, and generating, based on the testing, resultant error values:

and tf the resultant error is value is within a selected threshold designated for further processing, then continuing processing at a next resolution level of the associated multi-level aggregate digital data structure.

89. The metho of claim 88 further comprising: optimizing execution of processing operations to minimize computational work and memory traffic.

90. The method of claim 89 wherein multiple features of the input image are tested in parallel against the selected search domain.

91. The method of claim 90 wherein processing operations are executed in accordance with digital instructions executing on a computing device.

92. The method of claim 91 wherein the computing device comprises a GPU (General

Processing Uni t).

93. The method of claim 91 wherein the computing device comprises a CPU (Central

Processing Unit),

94. The method of claim 91 wherein the computing device comprises a TPU ( Tensor

Processing Unit)

95. The method of claim 90 wherein processing operations are executed by dedicate hardware circuitry.

96. The method of claim 90 wherein data generated for use in a given processing operation are retained for future use.

97. The method of claim 90 wherein data for use in a gi en processing operation are generated on an as-needed basis.

98. The method of claim 86 further comprising:

generating a first multi-level aggregate digital data structure representative of the input image an a second multi-level aggregate digital data structure representative of the search image;

selecting a first feature from the input image to locate within a selected search domain in the search image and executing a search operation for the selected first feature; and

selecting an additional feature from the generated multi-level aggregate digital data structures and executing a search operation for the additional feature.

99. The method of claim 98 wherein the search operation for the additional feature is executed alter execution of the search operation for the selected first feature,

100. The method of claim 98 wherein the search operation for the additional feature is executed in parallel with the search operation for the selected first feature.

101. The method of claim 86 wherein the result of a given search operation is a transform indicating a best correspondence between a feature from the input image and a best match for the input image feature in the search image.

102. The metho of claim 101 wherein the transform is invertible to generate a corresponding transform between a corresponding feature in the search image and a corresponding feature in the input image.

103. The method of claim 86 further comprising: selecting a first portion of a given image to be a search image and selecting a second, different portion of the given image to be an input image.

104. The method of claim 86 further comprising: applying comparison and search processes across a plurality of images simultaneously, such that one input image is simultaneously compared for correspondence with more than one search image.

105. The method of claim 86 further comprising: applying comparison and search processes across a plurality of images simultaneously, such that multiple input images arc simultaneously compared for correspondence with a given search image,

106. The method of claim 86 wherein the input image differs in format from the search image.

107. The method of claim 106 further comprising: executing a transform to enable comparison between an input image and a search image of di fferent formats.

108. The method of claim 10? wherein the applied comparison function adaptively interprets different respective sets of data associated with each image, to enable comparison between an input image and a search image of different formats.

109. A system operable to search for correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, the system comprising:

a digital processing resource comprising a digital processor, the digital processing resource being configured to:

A) utilize a multi-level aggregate digital data structure representative of the second image, the multi-level aggregate digital data structure comprising digital data representations at different resolutions;

B) compare a relatively lower resolution digital data representation of the first selected feature against a relatively lower resolution digital data representation of a first selected subset of the selected search domain in the multi-level aggregate data structure, the comparing comprising computation of a minimum difference value and a maximum difference value;

C) iterati vely execute the comparing (B) against other subsets of the selected search domain;

D) identify a threshold difference value based on maximum difference values from the comparisons against subsets of the selecte search domain; and

E) reject subsets for which the comparison results in minimum difference values that exceed the identified threshold difference value.

1 10. A program product for use with a digital processing system to enable the digital processing system to search for correspondence between a first selected feature in a first image and an element within a selected search domain in a second image, the digital processing system comprising a digital processing resource, the digital processing resource comprising at least one digital processor, the program product comprising digital processor-executable program instructions stored on a iron-transitory digital processor-readable medium, which when execute in the digital processing resource cause the digital processing resource to:

B) compare a relati vely lower resolution digital data representation of the first selected feature against a relatively lower resolution digital data representation of a first selected subset of the selected search domain in the multi-level aggregate data structure, the comparing comprising computation of a minimum difference value and a maximum difference value:

C) iteratively execute the comparing (8) against oilier subsets of the selected search domain;