EP0946919A1

EP0946919A1 - Real-time remote inspection of high-resolution images

Info

Publication number: EP0946919A1
Application number: EP97947822A
Authority: EP
Inventors: Andrea Basso; Raffaele Noro; Olivier Verscheure; Jean-Pierre Hubaux
Original assignee: Ecole Polytechnique Federale de Lausanne EPFL
Current assignee: Ecole Polytechnique Federale de Lausanne EPFL
Priority date: 1996-12-30
Filing date: 1997-12-30
Publication date: 1999-10-06
Also published as: AU5405898A; WO1998029819A1; GB9627037D0

Abstract

There is provided a method for storage, retrieval and transmission comprising: the steps of receiving hierarchical digital image data from the source (1), where images are analysed and further decomposed as a set of elementary areas in which any such area in one image at a given level of hierarchy corresponds to a group of elementary areas in the upper level of the hierarchy and in which for each image of the hierarchy, consecutive elementary areas are sorted out in a first file, where the position in said first file of each area is indexed in a second file and the length in bytes of each area is stored in a third file; the steps of passing and storing the index file and the length file of the areas in each image of the hierarchy in which these files are further used when retrieving from the storage means (3) a portion of a given image in the hierarchy selected by a control signal (5) issued by the display apparatus (10) and where the selection refers to an image or portion of an image at a coarser resolution level in the hierarchy; the steps of buffering the requested portion of the images at higher resolution where the elementary areas are organised in order to form a sequence of consecutive areas to be transmitted across the transmission means (9) and that are suitable for the display apparatus.

Description

Real-Time Remote Inspection of High-Resolution Images

FIELD OF THE INVENTION

This invention relates to Real-Time Remote Inspection of High-Resolution Images and more particularly to coordinated storage, retrieval and transmission of digital images with different levels of resolution.

BACKGROUND OF THE INVENTION

There is the need of accessing high resolution image information in several fields of pure and applied technical and scientific environments, such as cartography, medicine, industrial design and professional editing.

In the past this information was typically stored in large "film rooms" in the form of printed films or papers. The available technologies permit today to have and store the same information in large database systems containing images in digital format.

There is also the additional need to remotely access this data from a display station that is connected with long-distance transmission means to the storage apparatus; the most relevant user requirement in this case is the so called distance independence, i.e., the ability to remotely perform display operations like browsing, navigation and zooming in a natural way. This requirement is usually addressed as real-time interactivity.

The interactivity is handled by letting the user to select, within a low-resolution version of the image he/she is interested in, only a portion of the image to be viewed with higher resolution or higher priority with respect to the remainder: this implies a hierarchical structure of the image data as well as the definition of Region of Interest (Rol) that indicates a specific portion of the image to be viewed at higher resolution. It turns out that the Rol have two possible interpretations: in the first, the Rol is viewed in a display window independent from the low-resolution image while in the second, the Rol is displayed within the same display window but with an enhanced level of resolution for the specific region (EP-A-0 246 010 and WO-A-96 29818 teach about said second interpretation). The coder defined in "JPEG Committe Draft, Digital Compression and Coding of Continuous-tone Still Images, Jan. 1991" also teaches about multi-resolution hierarchical coding. The concept of Rol makes the system interactive because it reduces drastically the amount of information and thus the delay for the display and long-distance transmission at a given instant. Moreover, the extensive use of coding-decoding techniques, in particular targeted to hierarchical image structures further increases the degree of interactivity. The publication of H. Torbey, D. Freilander, J. Barda, "Selective Decompression on a Hierarchically Coded Image", in Multidimensional Signal Processing, San Francisco, Mar. 1992 [1] teaches about mathematical relations enabling decoding/decompression for image hierarchies that operates on a user selectable Rol.

The aim of the present invention is to provide coordinated integrated storage, retrieval and transmission methods to support the decoding operations required in particular by [1], although it is not exclusively targeted to the mentioned system.

For the scope of this invention, it is addressed a system that receives a hierarchy of images at different resolution levels, either in original or coded format, and, by using only index numbers as identificators of portions of images, establishes direct relationships between corresponding areas in all consecutive levels of the hierarchy, before the storage in the database. It is also addressed a system for retrieving in a second time both full images and Rol that are selected by the user through the display apparatus. Finally, it is addressed a system for the interactive transmission of images or Rol as a sequence of consecutive elementary areas.

It is noticeable that, although this invention fits both with coded and uncoded images, in most of the implemented systems it will operate on coded material and thus it will select elementary areas as blocks of the coded image, defined by appropriate block headers in the coded stream.

SUMMARY OF THE INVENTION

The first object of this invention is to provide a suitable technique for the storage of high-resolution images that are generated by the image source as a hierarchy of digital images.

The second object of this invention is to provide a suitable technique for retrieving images and portion of images that are organised in a hierarchical structure in a storage apparatus. The third object of this invention is to provide a suitable technique for the transmission to a display apparatus of images and portion of images that are retrieved from a storage apparatus and are organised in a hierarchical structure.

In order to achieve the above objects, there is provided a method for storage, retrieval and transmission comprising: the steps of receiving hierarchical digital image data from the source, where images are analysed and further decomposed as a set of elementary areas in which any such area in one image at a given level of the hierarchy corresponds to a group of elementary areas in the upper level of the hierarchy and in which for each image of the hierarchy, consecutive elementary areas are sorted out in a first file, whereas the position in said first file of each area is indexed in a second file and the length in bytes of each area is stored in a third file; the steps of passing and storing the index file and the length file of the areas in each image of the hierarchy in which these files are further used when retrieving from the storage means a portion of a given image in the hierarchy selected by a control signal issued by the display apparatus and where the selection refers to an images or a portion of an image at a coarser resolution level in the hierarchy; the steps of buffering the requested portion of images at higher resolution where the elementary areas are organised in order to form a sequence of consecutive areas to be transmitted across the transmission means and that are suitable for the display apparatus.

The first object of the invention is more particularly achieved by the steps of storing digital hierarchies of images corresponding to increasing resolution levels in which the whole information content of the image is aggregated as a set of indexes, each of them corresponding to one and only one elementary area and where:

- images are received from a source of images that independently generates the image hierarchy

- one elementary area at a given level of the hierarchy matches exactly a down- sampled aggregation of elementary areas in the adjacent image at higher resolution

- a relationship between one elementary area and the group of matching areas of the upper resolution level is established through index numbers with a preprocessing phase prior of the storage.

The step of storing digital hierarchies of images may be a step of storing hierarchies of uncoded images in the form of consecutive elementary areas of M x N pixels, independently from the hierarchical level, or a step of storing hierarchies of coded images where elementary areas are obtained through a mask composed of consecutive regular rectangles of M x N pixels.

The image at level i may be stored as a set of consecutive regular elementary areas for which an index is produced in the increasing order from 0 to H_j x V_j , where H_j and V_j are respectively the number of areas in the horizontal and vertical directions.

Each image at level i may be decomposed in H_j x Vj consecutive elementary areas and in which H = Z_j x H_j and V_j.^Z_j x V_s are the number of elementary areas in the upper level i-1 of the hierarchy and Z₍ is the zooming factor between image l_j and l_j..,.

An index may be associated and used to identify a unique elementary area of the image at level i and in which this index corresponds to Zj² indexes of the image at level i-1 and each image may be decomposed as a set of indexed elementary areas and for each index the length in bytes may be written in a file at the corresponding index value. In a variant, each image may be decomposed in a set of indexed elementary areas and for each index the position of the first byte of the information relative the indexed area may be written in a file at the corresponding index value.

The second object of the invention is more particularly achieved by retrieving steps for stored hierarchies of images where:

- the hierarchy is stored as a unique file for each coded or uncoded image in the order of consecutive elementary areas

- the display apparatus queries images or portion of images in the increasing order of resolution level

- relationships between elementary areas of successive levels of the hierarchy are maintained in the form of files that represents position and length of each individual area within the stored image.

The image at the lowest resolution level n may be used to define a portion of the image at level n-1 in the form of a rectangular mask defined by the upper-left x_{1 t} y₁ and bottom-right x₂, y₂ corners.

At any level i of the hierarchy may be defined a portion of the image at level i-1 to be retrieved as a rectangular area x_{1 (} y_η, x₂, ^₂ , until the level 1 of the hierarchy is reached. The queries may be received from the display apparatus as a query of a rectangular area x₁, y_{1 ;} x₂, y₂ of the image at level i that is mapped into a corresponding rectangular area x^, y.,', x₂', y₂' of the image i-1 to be retrieved, and a rectangular area x^, y^, x₂', y₂' of the image at level i-1 of the hierarchy may be mapped into a set of indexed areas for which the indexes are in the increasing order and the set of elementary areas covers the defined rectangle.

The database may be queried for a portion of an image in the form of a set of rectangular elementary areas and this elementary areas may be identified as index numbers in the increasing order and in which the index is associated with the position and length of the elementary area to be retrieved. In a variant the database may be queried for a portion of an image in the form of a set of elementary areas, where the indexed elementary areas are identified by indexed positions and lengths to be retrieved without any specific ordering.

A sequence of portions of the same reference image may be queried.

The third object of the invention is more particularly achieved by the steps of temporarily storing the information relative to the elementary area to be transmitted in a buffer organised to store the exact amount of information to be transmitted when buffer is filled up.

The buffer may be organised as consecutive indexed slots in the increasing indexing order, each slot corresponding to an indexed length.

The transmission may be organised for sending the data contained in the buffer and data being readout in the increasing index order.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a block diagram of a preferred embodiment of the method of the present invention.

Fig. 2 represents the operations performed in storage mode, where the Y_j x X_j image of level i is decomposed in H_j x V₍ elementary areas of size M x N pixels. Fig. 3 represents the mapping of one M x N elementary area of level i with Z_j ² elementary areas of level i-1 , where Z_j is the zooming factor between image at level i and image at level i-1 .

Fig. 4 represents the selection of the active elementary areas in the image at level i- 1 once the selection is performed in the image at the level i . The rectangle x^, y-,', x₂', y₂' is derived from the reference image, then the indexes of the active elementary areas are computed.

Fig. 5 represents the set of indexes that are queried to the database after the selection and the computation of active elementary areas on the reference image.

Fig. 6 represents the buffer space allocated for each control signal received from the display apparatus, once the selection on the reference image has been performed, as well as the queries to the storage apparatus have been sent: the selected elementary areas are received from the database and placed in the appropriate slot, until the buffer is filled up.

DETAILED DESCRIPTION OF THE INVENTION

A block diagram of a preferred embodiment of the method of the present invention is represented on Fig. 1. Block 1 is the source of images in hierarchical format, block 2 is the storage system of the invention, block 3 is the storage apparatus, block 4 is the memory that stores the length and position of the indexed elementary areas for subsequent retrieval, block 5 generates the control signals received from the display apparatus, block 6 is the retrieval system of the invention, block 7 is the memory to organise the buffer slots of the retrieved elementary areas, block 9 is the transmission support and block 10 is the display apparatus.

The method for storage, retrieval and transmission of the present invention includes the following described methods in storage, retrieval and transmission modes.

Method in storage mode

The storage apparatus is composed of one or several disks and/or CD juke-box that permanently store any digital information, in particular high-resolution images in the embodiment of the invention. The method in the storage mode is receiving the hierarchy of digitised images from the image source, either in original or coded format. The image with the lowest resolution level, say In, is analysed first and subdivided into elementary areas of equal size M x N pixels to form a regular grid (Fig. 2). Each area of the grid is then indexed starting from the upper-left corner with the number zero and increasing the index for the consecutive areas in the horizontal direction or for the left area of the following line of the grid if the right image border is reached.

The number of areas in the horizontal scan and in the vertical scan are, respectively:

H_n = X_n/N V_n = Y_n/M where:

X_n is the horizontal size in pixels Y_n is the vertical size in pixels

This gives H_n x V_n adjacent areas indexed from 0 to (H_n x V_n)-1 .

For each indexed area, the method in storage mode generates a trace of the length l_k, in number of bytes, occupied by the k-th indexed area of the image (such length is the same for all areas if the image is in uncompressed format).

The elementary areas are then sent to the storage apparatus in their exact indexing order, so that the consecutive bytes of each area are adjacent for the subsequent retrieval and transmission.

Similarly, the method applies for all the images with higher resolution l_n.-,, ..., ^ , where I-, is the image with highest resolution in the hierarchy. All elementary areas have M x N pixels and in order to provide the hierarchical structure, each given area of the image l_j is covered by the low-resolution version of an aggregation of an integral number of elementary areas in the image lj._{1 t} thereof if the zooming factor from image l_j to image l is Z_j in both the horizontal and vertical axes, each elementary area in l_j corresponds to Z_j ² elementary areas in l_μι (Fig. 3).

Further, for each digital image of the hierarchy and for each indexed elementary area, the method computes the position and the length, in number of bytes, of the indexed areas in the order they are sent to the storage apparatus. These positions and lengths are then recorded and sent as two distinct files to the retrieval module for subsequent hierarchical access to the images.

Method in retrieval mode

The method in retrieval mode is remotely controlled by a user and is receiving control signals from the display apparatus. The storage apparatus contains all the information that can be potentially accessed at a given instant by the user in the hierarchy of resolution levels, to be displayed either as separate windows for separate resolution levels or in the same window with enhanced resolution levels.

For the object of this invention, it is assumed that the control signal issued by the user through the display apparatus represents a request for zooming within the image l_j at the resolution level i , the rectangular region defined by his upper-left corner x^ y^ and his bottom-right corner x₂, y₂ . The method operates as follows: for the image l_n with the lowest resolution level, the retrieval system reads from the storage apparatus l_k consecutive bytes at the position p_k , where k ranging from 0 to (H_n x V_n)-1 is the index of the elementary areas in that image and p_k and l_k are respectively the position and the length in bytes of the k-th indexed elementary area as contained in the retrieval memory of the method; in subsequent operations and with reference to the image l_j in the i-th level of the hierarchy, the retrieval is performed under control of the display apparatus that indicates simultaneously the level i of the reference image, the upper-left and bottom-right corner of the rectangular area x_{1 t} y^ x₂, y₂ to be viewed with magnification; the indexes of the elementary areas of the image l_j.-, to be then displayed are computed from the index of the upper-left corner of the portion of image lj already displayed (Fig. 4), say Cj , from the values of x^ y₁, x₂, y₂ and from the zooming factor between the two images Zj , first by determining the absolute position x-)', y^, X₂ ¹, V₂ ^{in tne} image lj_ι of the corresponding rectangle x-_|, y_{l t} x₂, y₂ of the image lj as:

x = Z_j x (C_j MOD H_j + x ) y-,' = Zj x (Cj DIV Hj + y.,) x₂' = Z_j x (C_j MOD H_j + x₂) y₂' = Zj x (Cj DIV Hj + y₂)

where: x MOD y is the remainder of x/y x DIV y is the integral part x/y

and second by determining the index entries relative to elementary areas of the image Ij.-i as (Fig. 5): y₁'DIV H_M+ y₁'DIV H_M+ y DIV Hi-l÷

÷x^DIV N, +(x₁'+N)DIV N, ... +x₂'DIV N,

(y₁'+M)DIV H_M+ (y₁'+M)DIV H_M+ '+MJDIV H_M+

+x₁'DIV N, +(x₁'+N)DIV N, ... +x₂'DIV N,

y₂'DIV Hi-1+ y₂ ^,DIV H_M+ y₂'DIV H_μι+

+x₁ ^,DIV N, +(x₁'+N)DIV N, ... +x₂T IV N,

that cover the rectangle x-_|', y.,', x₂', y₂' ; these index entries activate the corresponding entries in the position and length files of the image l_μι contained in the retrieval memory that queries the storage apparatus.

The image data in the storage apparatus is accessed in the order of increasing index, reading in the storage apparatus the data file corresponding to the image l at the indexed position and at the indexed length in bytes.

The following operation is the transmission of the elementary areas to the display apparatus.

Method in transmission mode

The method in the transmission mode controls a buffer in which the information relative to the elementary area being transmitted is temporarily stored before accessing the transmission means.

The allocated buffer space corresponds to the sum of all the lengths of the elementary areas that have been requested by the retrieval module and each area and buffer slot are sorted in the order of their increasing index. The transmission operates as follows: the indexed areas enters the transmission buffer in their request order, if storage apparatus is synchronous with the retrieval module, or in a random order, if not, and are put in the corresponding location of the buffer since we know from the index the position and length of all the elementary areas (Fig. 6); when the buffer is filled up and ready to be transmitted, the bytes are then delivered to the transmission means and in turn to the display means. The system stops until the next control signal is received from the display apparatus.

DOMAINS OF APPLICABILITY

The invention has several domains of applicability, generally those in which the inspection of high resolution images is a current practice and in particular when such information have to be stored in a central database for future multiple access, such as:

- database systems for the design of industrial plants and buildings,

- database systems for the design and further inspection of mechanical engines,

- database systems for the computer assisted consultation of cartographic material,

- medical systems to store and access images produced via digital microscopes,

- medical systems to store and access images obtained by radiation,

- medical systems to store and access cardiography films.

The required means are: a computer unit located near the storage apparatus to process digital images in hierarchical format where either original or incremental or coded formats are equally supported, since it is a task of the display apparatus to process the content of the images; storage means to store the information that is processed by the computer unit and accessed later; a computer unit to receive controls from the display apparatus and query data to the storage means; buffer means to structure the information to be sent to the display apparatus; transmission means to carry information from the buffer to the display apparatus and to carry control signals from the display apparatus to a computer unit.

The benefits of using the method of the present invention include:

- user ability to select instantaneously the portion of images to be viewed with magnification,

- system ability to select fast and precisely the information required at a given instant by the user,

- system ability to reduce the amount of information that have to cross the transmission means,

- system ability to co-exist with any present and future method for image processing.

Claims

1. A method for storage, retrieval and transmission of digital images with different levels of resolution, characterised by the steps of storing digital hierarchies of images corresponding to increasing resolution levels in which the whole information content of the image is aggregated as a set of indexes, each of them corresponding to one and only one elementary area and where:

2. A method as in Claim 1 , characterised by storing hierarchies of uncoded images in the form of consecutive elementary areas of M x N pixels, independently from the hierarchical level.

3. A method as in Claim 1 , characterised by storing hierarchies of coded images where elementary areas are obtained through a mask composed of consecutive regular rectangles of M x N pixels.

4. A method as in any one of the preceding Claims, characterised by storing hierarchies of images where the image at level i is stored as a set of consecutive regular elementary areas for which an index is produced in the increasing order from 0 to H_j x V_j , where H₍ and V_j are respectively the number of areas in the horizontal and vertical directions.

5. A method as in Claim 4, characterised by storing a hierarchy of coded or uncoded image data where each image at level i is decomposed in H_s x V_j consecutive elementary areas and in which H_j.^ Zj x Hj and V_j.₁=Z_i x V_s are the number of elementary areas in the upper level i-1 of the hierarchy and Z_j is the zooming factor between image l_j and l^.

6. A method as in Claims 4 or 5, characterised by storing hierarchies of images where an index is associated and used to identify a unique elementary area of the image at level i and in which this index corresponds to Z_j ² indexes of the image at level i-1 .

7. A method as in Claim 6, characterised by storing a hierarchy of images where each image is decomposed as a set of indexed elementary areas and for each index the length in bytes is written in a file at the corresponding index value.

8. A method as in Claim 6, characterised by storing a hierarchy of images where each image is decomposed in a set of indexed elementary areas and for each index the position of the first byte of the information relative the indexed area is written in a file at the corresponding index value.

9. A method as in anyone of the preceding claims, characterised by retrieving steps for stored hierarchies of images where:

10. A method for storage, retrieval and transmission of digital images with different levels of resolution, characterised by retrieving steps for stored hierarchies of images where:

11. A method as in Claim 10, characterised by stored hierarchies of images where the image at the lowest resolution level n is used to define a portion of the image at level n-1 in the form of a rectangular mask defined by the upper-left x_{1 f} y₁ and bottom-right x₂, y₂ corners.

12. A method as in Claim 10, characterised by stored hierarchies of images where at any level i of the hierarchy is defined a portion of the image at level i-1 to be retrieved as a rectangular area x_{1 t} y₁, x₂, y₂ , until the level 1 of the hierarchy is reached.

13. A method as in anyone of the Claims 10, 11 or 12, characterised by stored hierarchies of images where the queries are received from the display apparatus as a query of a rectangular area x-,, y_{1 t} x₂, y₂ of the image at level i that is mapped into a corresponding rectangular area x^, y^, x₂', y₂' of the image i-1 to be retrieved.

14. A method as in Claim 13, characterised by stored hierarchies of images in which a rectangular area x.,', y.,', x₂', y₂' of the image at level i-1 of the hierarchy is mapped into a set of indexed areas for which the indexes are in the increasing order and the set of elementary areas covers the defined rectangle.

15. A method as in Claims 13 or 14, characterised by stored hierarchies of images that queries the database for a portion of an image in the form of a set of rectangular elementary areas and this elementary areas are identified as index numbers in the increasing order and in which the index is associated with the position and length of the elementary area to be retrieved.

16. A method as in Claims 13 or 14, characterised by stored hierarchies of images that queries the database for a portion of an image in the form of a set of elementary areas, where the indexed elementary areas are identified by indexed positions and lengths to be retrieved without any specific ordering.

17. A method as in anyone of the Claims 11, 12, 13, 14, 15 or 16, characterised by stored hierarchies of images that queries a sequence of portions of the same reference image.

18. A method as in anyone of the preceding Claims, where the information relative to the elementary area to be transmitted is temporarily stored in a buffer organised to store the exact amount of information to be transmitted when buffer is filled up.

19. A method as in Claims 18, where the buffer is organised as consecutive indexed slots in the increasing indexing order and each slots corresponds to an indexed length.

20. A method as in Claims 18 or 19, characterised by transmitting portions of images that is organised for sending the data contained in the buffer and in which data is readout in the increasing index order.

21. A method for storage, retrieval and transmission of digital images with different level of resolution, where the information to be transmitted is temporarily stored in a buffer and transmitted when the buffer is filled up and where the buffer is organised to store the exact amount of information to be transmitted.

22. A method as in Claims 21 , where the buffer is organised as consecutive indexed slots in the increasing indexing order and each slot corresponds to an indexed length.

23. A method as in Claims 21 or 22, characterised by transmitting portions of images that is organised for sending the data contained in the buffer and in which data is readout in the increasing index order.

24. A method for storage, retrieval and transmission of digital images comprising: the steps of receiving hierarchical digital image data from the source, where images are analysed and further decomposed as a set of elementary areas in which any such area in one image at a given level of the hierarchy corresponds to a group of elementary areas in the upper level of the hierarchy and in which for each image of the hierarchy, consecutive elementary areas are sorted out in a first file, whereas the position in said first file of each area is indexed in a second file and the length in bytes of each area is stored in a third file; the steps of passing and storing the index file and the length file of the areas in each image of the hierarchy in which these files are further used when retrieving from the storage means a portion of a given image in the hierarchy selected by a control signal issued by the display apparatus and where the selection refers to an images or a portion of an image at a coarser resolution level in the hierarchy; the steps of buffering the requested portion of images at higher resolution where the elementary areas are organised in order to form a sequence of consecutive areas to be transmitted across the transmission means and that are suitable for the display apparatus.