Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/14—Transformations for image registration, e.g. adjusting or mapping for alignment of images
  • G06T3/147—Transformations for image registration, e.g. adjusting or mapping for alignment of images using affine transformations
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/25—Fusion techniques
  • G06F18/253—Fusion techniques of extracted features
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/24—Aligning, centring, orientation detection or correction of the image
  • G06V10/245—Aligning, centring, orientation detection or correction of the image by locating a pattern; Special marks for positioning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
  • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
  • G06V10/757—Matching configurations of points or features
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
  • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
  • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features

Definitions

• the invention relates to a system and method for mapping still or moving images of different types and/or different views of a scene or object at the same or different points in time such that a specific object or location in the scene may be identified and tracked in the respective images.
• the system and method may be applied to virtually any current or future image types, both 2-D and 3-D, both single-frame and multi-frame (video).
• the system and method also may be used in association with any known method of image alignment by applying the same form of transformation to images as applied by that image alignment method.
• images may be different types (e.g., x-ray, photograph, line drawing, map, satellite image, etc.), similar image types taken from different perspectives (e.g., different camera angle, rotation, focal length or subject-focal plane relationship), similar or different images taken at different points in time, or a combination of all of these.
• images may be different types (e.g., x-ray, photograph, line drawing, map, satellite image, etc.), similar image types taken from different perspectives (e.g., different camera angle, rotation, focal length or subject-focal plane relationship), similar or different images taken at different points in time, or a combination of all of these.
• imaging types may be used with such imaging types or other imaging types that capture and present images of 3-D space (e.g., CAT and MRI, which use multiple 2-D slices) or that create 3-D renderings from 2-D images (e.g., stereoscopic slides such as are used in the current ophthalmology image comparison gold standard or "3-D” technologies such as used in entertainment today).
• the image types may also include video and film, which are composed of individual images (2-D or stereoscopic).
• a user may (1) estimate using various visual and intuitive techniques, (2) estimate using mathematical techniques, or (3) use computer image morphing techniques to align and overlay the images using, e.g., flicker chronoscopy, which is used in many other disciplines such as engineering and astronomy to identify change or motion.
• Each of these techniques has important shortcomings, including relatively low accuracy, being slow or time consuming, requiring high levels of skill or specialized knowledge, and being highly prone to error.
• An improved technique without such shortcomings is desired.
• the cross-image mapping (CIM) technique of the invention is designed to increase the ease, speed and accuracy of mapping objects across images for a variety of applications. These include - but are not limited to - flicker chronoscopy for medical tracking and diagnostic purposes, cartographic applications, tracking objects across multiple sequential images or video frames, and many others.
• the CIM technique of the invention makes it possible to locate specific coordinates, objects or features in one image within the context of another.
• the CIM technique can be applied to any current or future imaging technology or representation, whether 2-D or 3-D, single-frame (still) images or multi-frame (video or other moving image types).
• the process can be easily automated, and can be applied in a variety of ways described below.
• the CIM technique of the invention generally employs three broad steps:
• mapping and/or alignment parameters to identify and highlight the pixel in one image corresponding to the comparable location in another (i.e., identify the pixel that shows the same location relative to some landmark in each image).
• the method may also include the ability to indicate the accuracy or reliability of mapped pixel locations.
• This accuracy or reliability assessment may be based on outputs or byproducts of the alignment algorithm(s) or tool(s) employed in the mapping, or on assessment of aligned images after the fact.
• Such accuracy or reliability measures may be presented in many ways, including but not limited to visual modification of the mapped marking (through modification of line thickness, color, or other attributes) and quantitative or qualitative indicators inside or outside of the image area (e.g., red/yellow/green or indexed metrics).
• the scope of the invention includes a method, computer system and/or computer readable medium including software that implements a method for mapping images having a common landmark or common reference point (e.g., global positioning system tags, latitude/longitude data, and/or coordinate system data) therein so as to, for example, enable the creation, location and/or mapping of pixels, coordinates, markings, cursors, text and/or annotations across aligned and/or unaligned images.
• a common landmark or common reference point e.g., global positioning system tags, latitude/longitude data, and/or coordinate system data
• the computer- implemented method includes selecting at least two images having the common landmark or common reference point, mapping the selected images so as to generate mapping parameters that map a first location on a first image to the corresponding location of the first location on a second image, and identifying at least one pixel on the first image and applying the mapping parameters to at least one pixel on the first image to identify the corresponding pixel or pixels in the second image.
• the mapping parameters then may be used to locate or reproduce any pixels, coordinates, markings, cursors, text and/or annotations of the first image at the corresponding location of the second image.
• the two images may be of different image types including: x-ray image, photograph, line drawing, map image, satellite image, CAT image, magnetic resonance image, stereoscopic slides, video, and film.
• the images also may be taken from different perspectives and/or at different points in time.
• the images may be aligned using an automated image matching algorithm that aligns the first and second images and generates alignment parameters, or a user may manually align the first and second images by manipulating one or both images until they are aligned.
• Manual or automatic landmark mapping may also be used to identify the common landmark in the first and second images.
• associated software may generate mapping parameters based on the locations in the first and second images of the common landmark.
• the first image may be morphed to the second image whereby the common landmark in each image has the same coordinates.
• an indication of a degree of accuracy of the alignment and/or mapping of the selected images at respective points in an output image may also be provided.
• Such indications may include means for visually distinguishing displayed pixels for different degrees of reliability of the alignment and/or mapping of the display pixels at respective points.
• different colors or line thicknesses may be used in accordance with the degree of reliability of the alignment and/or mapping at the respective points or, alternatively, a numerical value for points on the output image pointed to by a user input device.
• the mapping may also be extended to pixels on at least one of the images that is outside of an area of overlap of the first and second images.
• Figure 1 illustrates images of a single, unchanged object where unaligned image A illustrates the object as taken straight on from a specific number of feet and unaligned image B illustrates the same object from a lower vantage, further away, with the camera rotated relative to the horizon, and a different placement of the object in the image.
• Figure 2 illustrates how image B is modified to correspond to image A.
• Figure 3 illustrates the mapping parameters for mapping unaligned image B to unaligned image A.
• Figure 4a illustrates the mapping of a user-drawn circle at a user-defined location from the input image B to the output image (aligned image B or input image A).
• Figure 4b illustrates the application of alignment parameters (e.g. lines) to the images to indicate shift by mapping "before and after" marks from two or more images onto the marked images or other images from the image set.
• alignment parameters e.g. lines
• Figure 5 illustrates two types of images of the same object where common identifying features are provided in each image.
• Figure 6 illustrates the alignment of the images of Figure 5 using a common feature by modifying one or more of the images to compensate for camera angle, etc. using a manual landmark application or an automated algorithm.
• Figure 7 illustrates the parameters for mapping from one image in a set to another, based on alignment of the two images (note the parameters are the same as in Figure 3 except that the images are not aligned).
• Figure 8 illustrates the mapping of a user-entered input marking in image A to image B or aligned image B.
• Figure 9 illustrates an exemplary computer system for implementing the CIM technique of the invention.
• Figure 10 illustrates a flow diagram of the CIM software of the invention.
• Figure 11 illustrates the operation of a sample landmark tagging application in accordance with the invention whereby corresponding landmarks are identified in two images either manually or through automation.
• Figure 12 illustrates the expression of a given "location" or "reference point” in an image in terms of a common landmark or by using a convention such as the uppermost left-hand pixel in the overlapping area of aligned images.
• Figure 13 illustrates examples of displaying accuracy or reliability in the comparison of images using the CIM techniques of the invention.
• Figure 14 illustrates aligned and mapped images in which image A covers a small portion of the area covered by image B, and illustrates a means for identifying coordinates of a landmark in image B relative to image A coordinate system but beyond the area covered by image A.
• the CIM technique of the invention employs computer-enabled image morphing and alignment or, alternatively, mapping through landmark tagging or other techniques, as the basis of its capabilities. Specifically, two or more images are aligned and/or mapped to each other such that specific landmarks in either image fall in the same spot on the other. It is noted that the alignment may be of only part of each of the images. For example, the images may depict areas with very little common overlap, such as images of adjacent areas. In addition, one image may cover a small area included in a second, larger area covered by the second image.
• the landmarks or pixels shown in the overlap area though bearing the same relationship to each other in both images and ostensibly representative of the same spot in space, might fall in very different locations within the image relative to the center, corner or edge.
• This alignment can be achieved by changing one image to match the other or by changing both to match a third, aligned image (in the case of multiple input images or video images, the same principles are applied several times over) or by mapping the images (mapping one image to match the other or by mapping both to match a third, aligned image) to each other without actually changing the images.
• mapping software such as Photoshop
• technologies such as the Dual Bootstrap algorithm.
• the end result is (1) a set of images including two or more unaligned images (unaligned image A, unaligned image B, and so on) and two or more aligned images (aligned image A, aligned image B, and so on), such that the aligned images can be overlaid and only those landmarks that have moved or changed will appear in different pixel locations and/or (2) a set of parameters for mapping one or more image in the set to another such that this mapping could be used to achieve alignment as in (1). It is important to note that the CIM technique is indifferent to the mechanism used for aligning or mapping images and does not purport to accomplish the actual alignment of images.
• aligning means to transform a first image so that it overlays a second image.
• image alignment may include the modification of one or more images to the best possible consistency in pixel dimensions (size and shape) and/or location of specified content within the image (e.g. , where only part of images are aligned).
• mapping means to identify a mathematical relationship that can be used to identify spot or pixel in one image that corresponds to the spot or pixel in another image. It is not necessary to modify either image to establish mapping. For example, mapping is used to create alignment and/or to represent the operations performed to achieve alignment. Mapping parameters are the output of the mapping operation and are used to perform the calculations of pixel locations when performing landmark tagging.
• this common pixel can be located in each image in any of several ways, typically (but not limited to) relative to a specified landmark or relative to a common reference point or pixel (e.g., uppermost left-hand) in the overlapping portion of the images.
• an "input image” is the image used to identify the pixel or location used for generating mapping
• an “output image” is the image upon which the mapped pixel or location is located and/or displayed. Both input and output images may be aligned or unaligned.
• mark tagging refers to various forms of identifying common landmarks or registration points in unaligned images, either automatically (e.g. through shape recognition) or manually (i.e. user-identified).
• the CIM technique of the invention first creates formulae for mapping a specific location within any image in the input or aligned image sets to any other in the set.
• the formulae contain parameters for shifting image centers (or other reference point) up/down and left/right, rotating around a defined center point, stretching one or more axes or edges or dimensions to shift perspective, and so on.
• These formulae can (1) be captured as part of the output of automated alignment algorithms such as Dual Bootstrap, or (2) be calculated using landmark matching in a landmark tagging or other conventional application.
• the landmark tagging application will present the user with two or more images, allow the user to "tag" specific, multiple landmarks in each of the images, and use the resulting data to calculate the formulae or parameters that enable a computer program to map any given point in an image to the comparable point in another image within the set.
• landmark tagging may be achieved through automated processes using shape recognition, color or texture matching, or other current or future techniques.
• the user selects two (or more) images from the image set for mapping. These may be all aligned images, a mix of unaligned and aligned images, or all unaligned images. These may be a mix of image types, for example drawings and photographs, 2-D video frames and 3-D still or moving renderings, etc. (e.g., CAT, MRI, stereoscopic slides, video, or film).
• the selected images are displayed by the landmark tagging application in any of several ways (e.g., side by side, or in overlapping tabs).
• the user may then identify a pixel, feature or location in one of the selected images (the input image), and the CIM application will identify and indicate the corresponding pixel (same object, landmark or location) in the other selected images (output images).
• the manner of identification can be any of several, including clicking with a mouse or pointing device, drawing shapes, lines or other markings, drawing freehand with an appropriate pointing device, marking hard copies and scanning, or other computer input techniques.
• Selected pixels or landmarks can be identified with transient indicators, by translating the lines or shapes from the input image into corresponding display in the output image, or by returning coordinates in the output image in terms of pixel location or other coordinate system.
• the input image can be an unaligned or aligned image, and the output image(s) can also be either unaligned or aligned.
• two or more images are selected for mapping. These input images may have differences in perspective, camera angle, focal length or magnification, rotation, or position within a frame.
• Figure 1 illustrative images of a single, unchanged object are shown.
• input image A the object is shown as taken straight on from a specific distance (e.g., 6 feet), while input image B illustrates the same object from a lower vantage point, further away, with the camera rotated relative to the horizon, and a different placement of the object in the image.
• the object will have changed shape, size, or position relative to other landmarks.
• Parameters for aligning and/or mapping the images are calculated. If an automated matching/morphing algorithm such as Dual Bootstrap is used, this process is automated. Alternatively, if manual landmark tagging is used, the user identifies several distinctive landmarks in each image and "tags" them (e.g., see the triangles in Figure 2). Finally, the images may be aligned through manual morphing/stretching (such as in Photoshop transforms). In either case, the best alignment or mapping possible is established. It is noted that, in some cases, some features may not align or map across images but that cross-image mapping may still be desirable.
• a photograph of a small area of landscape may be mapped to a vector map covering a far larger area (e.g., see Figure 14).
• Figure 2 shows how the input image B is modified to correspond to input image A.
• the resultant alignment and/or mapping parameters are recorded and translated into appropriate variables and/or formulae for aligning and/or mapping any two images in the image set.
• the mapping parameters for mapping input image B to input image A are shown. Typically, these parameters are expressed as a series of mathematical equations.
• Alignment and/or mapping parameters are applied to align and/or map a location in one image to the equivalent location in another image within the image set.
• a specific spot along the edge of a shape has been circled by the user, and the CIM application displays the equivalent shape on another image from the set (note that on the output image, the circle is foreshortened due to morphing for alignment).
• the item tagged may be a specific pixel, a circle or shape, or other form of annotation.
• Alignment and/or mapping parameters also may be applied to indicate shift by mapping "before and after" marks from two or more images onto the marked images or other images from the image set.
• two lines are drawn by the user (e.g., tracing the edge of a bone structure in an x-ray), and the two lines are plotted together onto a single image.
• the image onto which the lines are plotted may be a third image from the same set and that more than two markings and/or more than two images may be used for this technique.
• the drawing of the lines may be automated using edge detection or other techniques.
• mapping two images of different types may be used as input images, and a shared element of the two images may be used to calculate mapping parameters. Examples of this form include combining (1) x-rays and photos of tissue in the same spot, (2) photos and line maps or vector maps such as those used in Computer Aided Mapping (CAM) applications used to track water or electrical conduits beneath streets, (3) infrared and standard photographs, or (4) aerial or satellite photographs and assorted forms of a printed or computerized map.
• CAM Computer Aided Mapping
• a common feature is used to align and/or map - and if necessary conform through morphing - two or more images.
• Examples of a common feature include: teeth visible in both a dental x-ray and a dental photograph; buildings visible in photographs and CAM maps; known coordinates in both images, e.g., a confluence of rivers or streets or latitude and longitude.
• input image A in Figure 5 may represent image type A such as an x-ray of teeth or a vector drawing such as in a CIM map.
• the illustrated white shape may be an identifying feature such as a tooth or a building.
• Input image B may represent image type B such as a photo of tissue above an x-ray or an aerial photo of an area in a vector map.
• the white shape may be an identifying feature such as a tooth or building.
• Figure 6 illustrates the alignment of the input images using the common feature (e.g., tooth or building) by morphing one or more of the images to compensate for camera angle, etc. using a CIM landmark tagging application, an automated algorithm, or using manual alignment (e.g., moving the images around in Photoshop until they align). In some cases, alignment and/or mapping may be achieved automatically using shapes or other features common to both images (such as teeth in the above example).
• the parameters for mapping from one image in a set to another are calculated and expressed as a series of mathematical equations as shown in Figures 3 and 7.
• mapping capability can now be used to identify the location of a landmark or point of interest in one image within the area of another from the set. This is illustrated in Figure 8, where a user-entered input marking in input image A is mapped to output image B using the techniques of the invention. If required, morphing of images may be applied in addition to re-orientation, x,y shift, rotation, and so on.
• mapping technique of the invention need not be limited to mapping visible markings. It could, for instance, be used to translate cursor location when moving a cursor over one image to the mapped location in another image.
• FIG. 9 illustrates an exemplary computer system for implementing the CIM technique of the invention.
• a microprocessor 100 receives two or more user-selected input images 110 and 120 and processes these images for display on display 130, printing on printer 132, and/or storage in electronic storage device 134.
• Memory 140 stores software including matching algorithm 150 and landmark tagging algorithm 155 that are optionally processed by microprocessor 100 for used in aligning the images and to generate and capture alignment parameters. Matching algorithm 150 and landmark tagging algorithm 155 may be selected from conventional algorithms known by those skilled in the art.
• CIM software 160 in accordance with the invention is also stored in memory 140 for processing by microprocessor 100.
• FIG 10 illustrates a flow diagram of the CIM software 160 of the invention.
• the CIM software 160 enables the user to select two or more images or portions of images at step 200.
• the selected images are then aligned in step 210 using the automated matching algorithm 150, and alignment parameters (e.g., Figure 3) are generated/captured from the algorithm at step 220.
• the alignment may also be performed manually by allowing the user to manipulate, reorient and/or stretch one or both images until they are aligned.
• the mapping would document the manipulation and alignment parameters would be generated at step 220 based on the mapping documentation.
• landmark tagging also may be used to map images by determining transformations without changing the images at step 230 and generating the mapping parameters generated by the mapping application (e.g., CIM matching algorithm) at step 240.
• the alignment and/or mapping parameters are used to define formulae for aligning/mapping between all image pairs in a set of images (e.g., unaligned-unaligned, unaligned-aligned, aligned-aligned).
• a pixel or pixels on any image in an image set (e.g., an input image) is then identified at step 260 and the afore-mentioned formulae are applied thereto to identify the corresponding pixel or pixels in other images in the image set (e.g., output images) at step 270.
• the pixel location is mapped, any markings, text or other annotations entered on an input image are optionally reproduced on one or more output images, the pixel location is identified and/or displayed, and/or pixel coordinates are returned at step 280.
• the degree of accuracy or reliability is calculated and/or displayed to user, as described below in connection with Figure 13.
• FIG 11 illustrates a sample landmark mapping application used in step 230 in accordance with the invention in which the user selects two or more images that are displayed side-by-side, in a tabbed view, or in some other manner.
• the user selects landmarks such as corners of the same object in the two images and marks each landmark in each image using a mouse or other input device.
• the selected landmarks are identified as comparable locations in each image (e.g., by entering numbers or using a point-and- click interface).
• the CIM software 160 uses the corresponding points to calculate the best formulae for translation from one image to another, including x,y shift of the image(s), rotation, and stretching in one or more dimensions.
• the images need not be actually aligned; rather, the mapping formulae are used to map pixels, coordinates, markings, cursors, text, annotations, etc. from one image to another using the techniques described herein.
• Additional levels of functionality may easily be added to the CIM software 160.
• manual tagging or automated edge detection may be used to identify a specific landmark in two images, as well as a reference landmark of known size (e.g., a foreign object introduced into one image to establish a size reference) or location (e.g., the edge of a bone that has not changed).
• a CIM application or module within another application can calculate distances or percentage changes between two or more images.
• Additional information about the mapping may be displayed visually or in other ways. For example, statistical measures of image fit may be used to estimate the accuracy and/or reliability of the mapping, and to display this degree of accuracy or "confidence range" through color, line thickness, quantitative displays or other means. Furthermore, such information may be a function of location within an image (e.g., along an edge that has been greatly stretched versus an edge that has not); these differences may be reflected in the display of such additional information either visually on an image (e.g., through line thickness or color of markings) or through representations such as quantitative measures.
• a specific pixel in an input image may correspond to a larger number of pixels in the output image (for example, a ratio of 1 pixel to four).
• the line on the output image may be shown as four pixels wide for every pixel of width in the input image.
• this can be shown with colors, patterns or other visual indicators by, for example, showing less accurate location mappings in red instead of black, or dotted instead of solid lines.
• the mapped locations might be one fourth the width; in this case, the line can be shown as using one quarter the pixel width, or as green, or as bold or double line.
• This approach to showing accuracy of mapping can be based on factors other than resolution.
• descriptive statistics characterizing the accuracy of alignment may be used, including measures derived from comparison of each pixel in an input and output image, measures derived from the number of iterations, processing time or other indications of "work" performed by the alignment algorithm, and so on. Such statistics may be employed as a measure of accuracy or fit.
• the uniformity of morphing applied can be used. For instance, if an image is stretched on one edge but not on another, the accuracy can be shown as greatest on the portion of the image that has been stretched the least.
• any indication of accuracy of alignment, reliability of/confidence in an alignment or other qualifying measures may be used as the basis of indicating these confidence levels.
• Figure 13 illustrates examples of displaying accuracy or reliability as just described.
• the input image on the left of the figure
• the mapping of pixels to the output image will be more accurate for the bottom line than the top line.
• this can be indicated through changes in the thickness of the line (Output A), the color of the line (Output B), attributes of the line (Output C), or by other, similar means.
• accuracy or reliability also may be indicated using a quantitative or qualitative display linked to the cursor, as in Output D.
• the cursor is pointed at various locations in the image and a "score" showing accuracy or reliability of the alignment is shown for that location in the image.
• GPS global positioning system
• GIS global information system
• coordinates may be extended beyond the area of overlap in the one or more images.
• the CIM technique of the invention may be used to infer the location of a pixel or object in image B based on extrapolation of coordinates attached to image A and mapped to image B using the overlapping area.
• Figure 14 illustrates aligned and mapped images in which image A covers a small portion of the area covered by image B.
• image A has associated coordinate data (e.g. latitude/longitude) and image B does not.
• a location in image B outside of the area of overlap with image A is selected as an input location, making image B the input image.
• the common landmark in the overlap area is at known coordinates in image A.
• CIM the parameters for mapping the overlapped areas are known and by extension areas that do not overlap are known. This allows one to establish the location of any pixel in image B by (1) applying the image A coordinate data within the overlap area to image B within the overlap area, and (2) extending the mapping beyond the area of overlap to infer the coordinates within the image A coordinate system of a pixel in image B, even if it is outside of the area covered by image A.
• the output location cannot be shown on image A but can be expressed in the coordinate system applied to image A.
• CIM can be used to establish mappings outside the area of overlap.
• MRI, CT, stereoscopic photographs, various forms of 3-D video or other imaging types may all have CIM techniques applied to and between them.
• CIM techniques applied to and between them.
• an MRI and CT scan can be mapped using CIM techniques, allowing objects visible in one to be located within the other.
• the structures that appear to have moved or changed in the respective input images may be located on the input images using the technique of the invention.
• structures or baselines e.g., jaw bone in dental images
• the technique may also be used to show corresponding internal and external features in images (e.g., abscesses on x-rays or gum surface in dental x-rays). This technique may also be used to show structures or baselines in successive frames of a video or other moving image source.
• a frame from a video of a changing perspective may be aligned to a map or satellite image.
• landmark tagging Once landmark tagging has been established, a given object in the video may be tracked in subsequent frames of the video by applying landmark tagging or other techniques establishing mapping parameters to the subsequent frames of the video.
• the CIM techniques described herein may be employed within a single moving image source by applying the technique to successive frames.
• a moving object in a video from a stationary perspective may be identified using landmark tagging or other techniques establishing mapping parameters and then tracked from frame to frame using successive applications of CIM .
• a stationary object in a video taken from a moving perspective e.g., from an aircraft
• CIM applications or CIM modules within other applications include:
• a patient may be exhibiting jaw bone loss, a very common problem.
• the doctor may compare two or more dental x-rays of the same area of the patient's jaw taken months apart. By marking the bone line in one and using CIM to map this marking to other images, the doctor, patient or other parties can see how much the bone has moved during the period between image captures, thus quantifying both the pace and magnitude of change.
• the doctor could highlight the bone line along the top of the bottom jaw in each of the two images as well as a baseline (for example the bottom edge of the bottom jaw).
• the CIM application could then calculate bone loss as a percentage of total bone mass.
• a reference object could be included in one or more images, and the CIM application could then express bone loss in millimeters.
• Examples of how this technique might be employed include using the overlay of a CIM map of gas mains and an aerial photo of a city block to pinpoint a location for digging which can be found by workers using landmarks rather than surveying equipment.
• GPS coordinates associated with one or both images may be used to identify additional images or areas of images contained in databases with which to align.
• This application can also use various measures of the accuracy and precision of alignment to indicate precision of mapping.
• the technique may also be used to examine the bones underneath a specific area of inflamed tissue or to locate a specific object visible in one photograph by mapping it against a shared feature in a map or alternate photograph.
• the input can take a variety of forms.
• Input mechanisms include (1) drawing lines, shapes or other markings using a mouse, touch-screen or other input device, so they are visible on the input image, (2) drawing lines, shapes or other markings using a mouse, touch-screen or other input device so they are not visible on the input image, (3) entering coordinate data such as latitude/longitude or map grids, such that specific pixels are identified on an input image with such associated coordinates, or (4) entering other types of information associated with specific locations within an image. Examples of other types of information include altitude data on a topographical map or population density in a map or other database.
• the form of input could be to specify all areas corresponding to a specific altitude or range of altitudes, or to a specific population density or range of population densities.
• Other means of input either existing or invented in the future, may be used to achieve the same result.
• the output can take a variety of forms. These include (1) showing lines, shapes or other markings on the output image, (2) returning the pixel location(s) of corresponding pixels in the output image, (3) returning latitude and longitude or other coordinates associated with pixels or specific locations in the output image, and (4) other forms of information associated with specific locations within an image.
• some input or output methods do not require the display of one or both images to be effective.
• the location to be mapped may be indicated by inputting appropriate coordinates, or alternatively values such as altitude ranges or population densities even if the input image is not displayed. These locations may then be displayed or otherwise identified or indicated in the output image.
• these coordinates can be identified or returned, without the output image itself being displayed.
• the user may then identify a feature or location in one of the selected images (the input image), and the CIM application will identify and indicate the corresponding pixel (same object, landmark or location) in a second selected image (output image).
• the manner of identification may be any of several, as described above.
• Selected pixels or landmarks may be identified with transient indicators or by translating the lines or shapes from the input image into corresponding display in the output image, or by returning coordinates or other location indicators in the output image.
• the input image may be either an aligned or unaligned image, and the output image(s) also may be either an unaligned or aligned image.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Evolutionary Computation (AREA)
• Artificial Intelligence (AREA)
• Multimedia (AREA)
• Data Mining & Analysis (AREA)
• Software Systems (AREA)
• Computing Systems (AREA)
• Medical Informatics (AREA)
• Health & Medical Sciences (AREA)
• General Health & Medical Sciences (AREA)
• Databases & Information Systems (AREA)
• Bioinformatics & Computational Biology (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Evolutionary Biology (AREA)
• General Engineering & Computer Science (AREA)
• Apparatus For Radiation Diagnosis (AREA)
• Image Processing (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Image Analysis (AREA)
• Magnetic Resonance Imaging Apparatus (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (1)

Family

ID=41255452

Family Applications (1)

Country Status (6)

Cited By (3)

Families Citing this family (13)

Citations (2)

Family Cites Families (12)

• 2009
  • 2009-05-01 EP EP09739953.9A patent/EP2286370A4/en not_active Withdrawn
  • 2009-05-01 AU AU2009242513A patent/AU2009242513A1/en not_active Abandoned
  • 2009-05-01 US US13/501,637 patent/US20120294537A1/en not_active Abandoned
  • 2009-05-01 JP JP2011507690A patent/JP2011520190A/ja active Pending
  • 2009-05-01 CA CA2723225A patent/CA2723225A1/en not_active Abandoned
  • 2009-05-01 WO PCT/US2009/042563 patent/WO2009135151A1/en not_active Ceased

Patent Citations (2)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events