Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
  • G06F17/10—Complex mathematical operations
  • G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
  • G06T7/269—Analysis of motion using gradient-based methods
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/97—Determining parameters from multiple pictures
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10004—Still image; Photographic image
  • G06T2207/10012—Stereo images
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30248—Vehicle exterior or interior
  • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

• the present invention relates to a method for determining an association between a matrix element of a matrix and a comparison matrix element of a comparison matrix and to a corresponding device.
• correspondences of elements of two matrices are made and mappings made between the elements.
• the matrix and the comparison matrix may each represent a first and a second image.
• the formation of correspondence or correspondences are used in the field of computer vision, in particular optical flow or stereo disparity. In the optical flow, correspondences are formed in temporal direction to determine how the projection of a point in a 3D scene has moved into a 2D image from an old coordinate to a new coordinate. It can the
• Motion in the image caused by the movement of the scene point or by the movement of a camera or both at the same time.
• Stereo vision uses different images and correspondences between individual pixels to capture simultaneous images from two cameras located at different locations.
• the relative arrangement of the cameras is usually known and fixed.
• Correspondence formation can thus be used to determine a distance to a point in the 3D scene. From DE 1 1 2009 001 727 B4, DE 10 2014 009 522 A1 and DE 10 2013 224 502 A1, methods for image processing and for calculating correspondences are known, wherein a correspondence of pixels in different images are generated. While the first two
• the method according to the invention for determining an association between a matrix element of a matrix and a comparison matrix element of a comparison matrix comprises a first step, which comprises carrying out a
• Writing process for writing the position of a matrix element of the matrix in a table field of a correspondence table provides.
• the writing process is performed for a plurality of matrix elements of the matrix.
• a reading process is carried out in which a table field of the correspondence table is read out and a position of a matrix element of the matrix stored in the table field is determined.
• a further step provides for the creation of an assignment from the read-out position from the table field and a current position of a comparison matrix element of a comparison matrix.
• the writing process comprises the steps of determining a feature of a matrix element of the matrix, determining an access feature from the matrix element, determining a table field of the correspondence table from
• the reading process for reading out the table field comprises the steps of reading out a feature of a comparison matrix element of a comparison matrix, determining a
• the reading process comprises determining a table field of the correspondence table from the access feature of the comparison matrix element and reading the stored position of the matrix element from the ascertained table field.
• the readout of a feature of a comparison matrix element of a comparison matrix, and thus the read process, can therefore preferably begin only when a plurality of matrix elements of the matrix have been processed, that is, e.g. the
• Matrix element of the matrix can be time-controlled by reading out or during processing or during the process, as a so-called “on the fly” method
• the feature can not be formed immediately, if needed, or a short time before it is needed (then with caching), for example, when it is determined from the environment of a pixel, or when the local environment is one pixel or one pixel describes.
• the matrix does not have to be fully occupied but may have gaps or only partially exist. For example, a sparse matrix may be present if the matrix elements represent a pixel or a pixel.
• the present invention can be applied not only to the processing of image data and images in the form of matrices, but rather represents a universal tool with which correspondences of various kinds can be efficiently determined. For example, letters can be recognized in texts and corresponding correspondences can be determined.
• the matrices used are not limited to Cartesian matrices, but include other matrix shapes, such as hexagonal matrices.
• Comparison matrix element is indicated in each case by coordinates. In this way, a unique position of the corresponding elements can be defined.
• the coordinates can be stored individually or in the form of vectors.
• the position in X- Coordinates and Y-coordinates can be specified. Hexagonal matrices are conceivable.
• the first matrix is a first image and the comparison matrix is a second image.
• a matrix element is advantageously a pixel in the first image, while a comparison matrix element is a pixel in the second image.
• the first image may be shifted in time relative to the second image, or the first image may be an image of a first camera and the second image may be an image of a second camera at the same time
• Matrix element or describes a comparison matrix element.
• the feature represents the local environment of an image coordinate or a pixel of an image, or the local environment around an element of a matrix.
• a disc-shaped or rectangular environment around an image coordinate is used, the feature being, for example, made of
• Gray values or color values can be formed.
• the feature is designed so that a similar-looking environment in all matrices results in an identical feature, i. H. in the case of image processing, the same environment around an image coordinate leads to identical features. Methods to generate features are well known to those skilled in the art.
• Form matrix element or from the feature of a comparison element.
• the feature and the access feature are the same.
• To determine the access feature the entire feature of a matrix element
• the value range of the feature corresponds to the size of the table.
• the feature can be used directly as an access feature and specify the address of the corresponding table field of the correspondence table.
• the value range from 0 to 2 16 -1 is directly comparable to a correspondence table of length 65,536.
• Describe correspondence table If the value range of the feature is greater than the table size or its length, that is, the number of table fields, then there are several possibilities. For example, only a small part of the feature can be used as a table address. The remaining part is either discarded or used as additional information, which may be stored in the correspondence table, for example in the form of an attribute. It is also possible to map a too large feature to a feature of lesser length via a function, which can be lossy.
• Matrix elements - or in the case of images from neighboring pixels - or from further away to a feature with a larger word length (concatenate).
• the number of table fields may be smaller or significantly smaller than the number of characteristics in the matrix for which the assignments or correspondence are to be determined.
• the access features are then not unique, since an access feature in the same picture usually occurs several times.
• At least one invalid access feature is provided, that is one that should not be used to form the assignment or correspondence.
• these can be completely white areas in which an image sensor has reached its saturation.
• Other examples are areas where there is no significant texture that is different from the noise of the sensor, or areas marked as irrelevant, such as the (uninteresting) area of the hood covered by a camera mounted behind the windshield. It is advantageous if a table field of the correspondence table next to
• Position of a matrix element comprises further attributes. For example, this additional information may help identify ambiguities in the formation of the association and identify preferred associations or preferred correspondences. Also it is possible, if too big
• the processing of the matrix and the comparison matrix takes place sequentially.
• the individual matrix elements are thus processed one after the other.
• line-by-line processing of the matrix elements can take place, i. H. the writing process or the reading process only become the first one
• the (sequential) processing preferably takes place via a
• predetermined number of matrix elements preferably over all matrix elements of the matrix.
• the number of writing processes and the reading processes is preferably the same.
• the offset between the writing process and the reading process is fixed or variable.
• the offset is chosen so that the reading process follows the writing process; a reversal of this principle is possible.
• a fixed offset has the advantage that the largest difference, which should be findable as correspondence or assignment, can be determined.
• a variable offset has the advantage that the search range determined by the offset is likewise variable and, for example, special features of the image processing can be taken into account. For example, the optical flux vectors used as an assignment in camera systems, ie the
• mapping vectors The lower the point in an image, the longer the mapping vectors. This feature can be due to the variable offset on simple Be taken into account, the writing processes and
• Reading processes may need to be adjusted. For example, it may be advantageous to suspend one of the processes for a certain duration if the offset is variable.
• the search field comprises a plurality of matrix elements that have already been processed by the writing process, for example, have been entered in the table. It can also advantageously comprise the current matrix element of the writing process, ie the last written matrix element.
• Matrix elements and predicates occur only for those matrix elements that are located within the search field. In this way, the
• Search range for assignments between the matrix and the comparison matrix are limited, for example, to avoid ambiguity.
• the search field may not only have a rectangular or square shape, but also other shapes such as circle, ellipse, polygon, or the like.
• mapping is a vector.
• the assignment vector is preferably selected from the position read from the table field and
• the assignment vector can be understood as a motion vector or optical flow vector in image processing methods. He can easily be processed.
• the reset can be done as a one-time reset (initialization), for example, for the entire
• Correspondence table Preferably, it takes place before the writing process.
• the reset may be a continuous process that may be performed in parallel with the writing process or the reading process, i. Each stored position is preferably processed several times per image pass and possibly deleted. This results in the "throughput" of the reset function.
• a continuous reset of the correspondence table preferably takes place at least twice per scroll, as a rule
• Table entries exist that can not yet be deleted because they can still lead to correspondence. These "leftovers" will be preferably deleted in a later, time-shifted run. In this way, old values in the table fields that are no longer suitable for correspondence formation are discarded or deleted.
• the offset for the time-shifted pass is advantageously selected so that it corresponds to at most half the image height or matrix height.
• all items stored in the table are processed (and, if necessary, deleted) in a predetermined order (e.g., ascending address). Once all the addresses have been processed, they start again in the same order. At least two such reset passes occur per image pass (or sub-matrix pass), wherein the reset can continue regardless of the image change.
• Association formation or correspondence formation are used because they are outside the search area, so the reading process has already processed the corresponding matrix elements. It is also advantageous to perform two suitably time-delayed reset passes per image pair or per matrix pair in order to ensure correct and complete processing in the continuous reset mode and to ensure that old entries in the
• Correspondence table be deleted in time before when processing a next matrix pair an erroneous correspondence formation or assignment formation can take place.
• the device for determining an association between a matrix element of a matrix and a comparison matrix element of a comparison matrix which is set up to carry out the method according to the invention, is advantageous and has all the advantages of the method according to the invention.
• Figure 1 is a schematic representation of the correspondence formation with a first image and a second image.
• FIG. 1 shows a matrix 101, which is a first image 102.
• a comparison matrix 201 is shown, which is a second image 202.
• the first image 102 and the second image 202 correspond to a shot at different times.
• the first matrix 101 comprises a plurality of matrix elements 103.
• the comparison matrix 201 has a multiplicity of comparison matrix elements 203.
• FIG. 1 shows a snapshot in which the first image 102, the second image 202 and a correspondence table 300 are shown, as well as a
• Result matrix 400 in which mappings can be stored and stored.
• the matrix elements 103, 203 each contain features that correspond to the access features 1 10 in the simplified form shown here.
• Access features are preferably coded as a number and indicate the address of a table field 301.
• the table field 301 corresponds to a row in the table 300.
• the table 300 shown here has four columns 302 to 305, the column 302 indicating the addresses of the table field. This column 302 is typically implicit and not explicit. It usually does not require one
• Columns 303 and 304 describe the positions, where column 303 is the X values and column 304 is the Y values of the coordinate or position of
• the writing process according to the method according to the invention has already advanced in FIG. 102.
• the writing process has a part of
• the processed matrix elements 103 are hatched in FIG.
• the matrix element 104 has the access feature 1 10, im
• the writing process thus writes the coordinate or position of the matrix element 104 in the 17th table row 301 of the correspondence table, so that the x coordinate and the y coordinate are entered into the columns 303 and 304 in the corresponding position.
• the comparison of the images 102 and 202 shows that the reading process performed in the image 202 follows the writing process carried out in the image 102.
• the offset between the writing process in image 102 and the reading process in image 202 is one line and several columns, but this is chosen only for the sake of illustration.
• Offset be chosen significantly larger, for example 32 lines and 64
• the writing process leads the reading process by this offset.
• the processing of the correspondence formation or assignment formation, in which identical access features 1 10 in the matrix 101 and the comparison matrix 201 are linked to one another, preferably takes place sequentially, for example line by line, z. B. starting with the upper left matrix element 103.
• the processing can also be done in columns or in a different order.
• the order is chosen to be the same both in image 102 and in image 202, but not started simultaneously, but with an offset that can be considered as a time offset or a local offset.
• a phase is first carried out in which writing is performed while the reading process has not yet started. This is followed by a phase in which both processes work. After reaching the last matrix element 103 within the matrix 101, the writing process stops while the reading process within the comparison matrix 201 is continued.
• the time-shifted reading process which reads out the current comparison matrix element 204, determines an access feature 110 for this element 204 with the value "21".
• the coordinate values from columns 303 and 304 are now read in the 21st line of the correspondence table 300.
• the table field refers to the matrix element 106 having the same access feature with the value "21”.
• An association between the comparison matrix element 204 and the matrix element 106 is now formed, the coordinates of the two elements 106, 204 being compared with one another.
• a difference formation is performed.
• Search box 120 is located. In the present case, this is given, so that the determined assignment u, v, which is present as an assignment vector, into the
• Result matrix 400 is entered.
• the search field 120 always refers to the image 102 of the writing process or to the matrix 101. It is in the comparison matrix 201 or in the second image for reasons of clarity
• the search field 120 is preferably selected such that it comprises only matrix elements 103 which have already been processed by the writing process, for which the position has thus been written into the table and which is now represented by the
• Figure 1 shows the case in which the flow is just large enough to meet the above criterion.
• the position of the matrix element 104 currently being processed in the write process lies just within the search field 120.
• the matrix element 104 currently to be processed by the write process is located in the lower right corner of the search field 120.
• the preprocess could also be larger, so that the matrix element 104 currently being processed lies, for example, to the right of the search region 120 or in the next line. It is possible that the search field 120 is asymmetrical with respect to the
• Comparison matrix element 204 is what is the case here. It would also be possible that the current reference position, ie the currently edited
• Comparison matrix element 204 is outside the search field. Importantly, however, the writing process has progressed so far that the positions of the matrix elements 103 located in the search field 120 have already been noted in the correspondence table 300 and written before the read process accesses the correspondence table. Therefore, the search field 120 must always lie completely in the region of the matrix 101 shown hatched here, in which the already processed matrix elements 103 are arranged.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Mathematical Physics (AREA)
• Multimedia (AREA)
• Pure & Applied Mathematics (AREA)
• Mathematical Optimization (AREA)
• Data Mining & Analysis (AREA)
• Mathematical Analysis (AREA)
• Computational Mathematics (AREA)
• Algebra (AREA)
• Databases & Information Systems (AREA)
• Software Systems (AREA)
• General Engineering & Computer Science (AREA)
• Computing Systems (AREA)
• Image Analysis (AREA)
• Complex Calculations (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=60191393

Family Applications (1)

Country Status (6)

Family Cites Families (24)

• 2016
  • 2016-11-23 DE DE102016223079.4A patent/DE102016223079B4/de active Active
• 2017
  • 2017-10-30 WO PCT/EP2017/077729 patent/WO2018095698A1/de unknown
  • 2017-10-30 JP JP2019547765A patent/JP6876818B2/ja active Active
  • 2017-10-30 EP EP17791681.4A patent/EP3545492A1/de not_active Ceased
  • 2017-10-30 CN CN201780072446.7A patent/CN109983503B/zh active Active
  • 2017-10-30 US US16/349,090 patent/US11341654B2/en active Active

Also Published As

Similar Documents

Legal Events