FR3053507A1 - VIEWER MULTIMODALITY - Google Patents

Abstract

The present invention is a method for viewing and manipulating a multi-channel image, comprising the steps of: storing the image in one or more 2D texture units, each 2D texture unit representing a package of four 8-bit tiles corresponding to a or more than one channel at a given position on the image; taking into account a selection of one or more channels of the multichannel image, taking into account transformation and / or projection parameters for the selected channels; applying a transformation on the selected channels in accordance with the received parameters, all the pixels of the channels being calculated at the same time; projecting the selected channels in an RGB color space, in accordance with the received parameters, all the pixels of the selected channels being projected at the same time.

Description

® FRENCH REPUBLIC

NATIONAL INSTITUTE OF INDUSTRIAL PROPERTY © Publication number: 3,053,507 (to be used only for reproduction orders)

©) National registration number: 16 56 262

COURBEVOIE © IntCI ⁸

G 06 T 7/00 (2017.01), H 04 N 19/00

A1 PATENT APPLICATION

©) Date of filing: 30.06.16. © Applicant (s): KEENEYE TECHNOLOGIES Company (© Priority: by simplified shares - FR. @ Inventor (s): BERLEMONT SYLVAIN. ©) Date of public availability of the request: 05.01.18 Bulletin 18/01. ©) List of documents cited in the report preliminary research: Refer to end of present booklet (© References to other national documents ® Holder (s): KEEN EYE TECHNOLOGIES Company related: by simplified actions. ©) Extension request (s): (© Agent (s): ICOSA SELARL.

(□ 4 / MULTIMODALITY VIEWER.

FR 3 053 507 - A1 (ùf) The present invention is a method for viewing and manipulating a multichannel image, comprising the steps of: storing the image in one or more 2D texture units, each 2D texture unit representing a packet four 8-bit tiles corresponding to one or more channels at a given position on the image; taking into account a selection of one or more channels of the multichannel image, taking into account transformation and / or projection parameters for the selected channels; applying a transformation on the selected channels, in accordance with the parameters received, all the pixels of the channels being calculated at the same time; projection of the selected channels in an RGB color space, in accordance with the parameters received, all the pixels of the selected channels being projected at the same time.

Multimodality viewer

The present invention relates to an image transmission system and method, and more particularly to a method of displaying and manipulating multichannel images at very high speed.

Imaging systems are becoming increasingly sophisticated, capturing huge amounts of data at increasing speed and resolution. The main challenge for a successful image analysis is to be able to visualize, manipulate, analyze and share image data in a simple, reliable and fast way.

One issue with multichannel image analysis is having the ability to display a desired selection of channels and apply transformations (e.g. contrast adjustment, color change ...) to each channel separately.

Another challenge is to allow a collaborative interpretation of the results by allowing professionals to access the data displayed at the same time, some evening their geographic locations, and from any communication terminal such as a computer, a tablet or a smartphone.

The invention relates to a system for manipulating a multichannel image stored in a server. This system is accessible from any web browser with no plugin installation required. The method for manipulating a multichannel image, according to the present invention, can be executed on terminals with limited resources such as mobile telephones.

The present invention relates to a method for manipulating and displaying a multichannel image in a very rapid manner allowing users to access data simultaneously from any communication terminal.

The method includes a storage method, to transform the multichannel image into a multitude of 2D texture units. This step corresponds to a preprocessing step performed on a server.

The system (framework in English) for performing the execution of the image manipulation method is accessible by a multiplicity of users from all types of communication terminals such as a computer, a tablet, or a smartphone. For example, when a user chooses a multichannel image stored on the server, and selects several channels, these are displayed on all connected communication terminals.

In general, for a given multichannel image, any user - but one at a time - can choose a selection of channels to display, transformation and projection parameters for each channel, at any time. The transformation and projection steps are performed on the selected channels at very high speed thanks to a parallel implementation of the processes (parallel computing in English) ie all the pixels of all the selected channels are processed at the same time on the graphic processors of the terminals of connected communication.

The invention can be used on any type of multichannel image, whether two-dimensional, three-dimensional, 8-bit or 16-bit format. The different image channels can also come from different imaging modalities.

A method is thus proposed for viewing and manipulating a multichannel image, composed of several channels, the channels of which come from one or more imaging modalities, said method comprising the steps of:

- storage of the multi-channel image in one or more 2D texture units, each texture unit representing a packet of four 8-bit tiles corresponding to one or more channels at a given position on the multi-channel image, each 2D texture unit being a memory storage unit specific to a graphics processor,

- taking into account a selection of one or more channels of the multichannel image,

- taking into account transformation and projection parameters for the selected channels,

- application of the transformation on the selection of channels, in accordance with the parameters received, all the pixels of the selected channels being processed at the same time,

- projection of the selected channels in an RGB color space, in accordance with the parameters received, all the pixels of the selected channels being projected at the same time.

The transformation applied can for example be a change in contrast of the image determined by the input parameters.

The storage step, carried out for an image in 8-bit format, includes the operations of:

- partitioning of each channel of the multichannel image into a regular grid of 8-bit tiles of the same dimensions, and of the same dimensions as a 2D texture unit,

- grouping of 8-bit tiles corresponding to a given position in the grid in one or more 2D texture units.

When the storage step is performed for a 16-bit format image, this step includes the operations of:

- partitioning of each channel of the multichannel image into a regular grid of 16-bit tiles of the same dimension, and of the same dimensions as a 2D texture unit,

- partitioning of each 16-bit tile into a pair of two 8-bit tiles by separating the highest eight bits from the lowest eight bits for each 16-bit pixel value,

- grouping of 8-bit tiles corresponding to a given position in the regular grid in one or more 2D texture units.

The transformation step, carried out for an image of 8bit format, includes the operations of:

- assignment of channel intensity values to the pixels from the values stored in the channels of the 2D texture units,

- calculation of the new pixel values in accordance with the transformation parameters.

The projection step, carried out for an image in 8-bit format, includes the operation of calculating the linear projection of all the pixels in an RGB color space, in accordance with the projection parameters.

In the case of a 16-bit format image, this projection step includes the operations of:

- calculation of the linear projection of all pixels in an RGB color space using the projection parameters,

- conversion of each R, G and B channel obtained into 8bit channel.

The transformation step, carried out for a 16bit image, includes the operations of:

- conversion of each consecutive pair of channels of 2D texture units into 16-bit format values,

- assignment of channel intensity values to pixels from 16-bit format values,

- calculation of new pixel values in accordance with the transformation parameters.

The present invention also provides a system for handling a multi-channel image, said multi-channel image being composed of several channels, said system comprising a server comprising:

- a conversion module to convert multi-channel images into one or more 2D texture units, each 2D texture unit representing a packet of four 8-bit tiles corresponding to one or more channels at a given position in the multi-channel image, a 2D texture unit being a memory storage unit specific to a graphics processor,

- a memory module for the storage of the 2D texture units obtained,

- a connection module to connect one or more communication terminals, such as a personal computer, a digital tablet or a smartphone, to the same account,

a selection module for receiving a selection of one or more channels of a multichannel image stored on the server, from one of the communication terminals connected to the server on the same account,

- an adjustment module to receive transformation and projection parameters as inputs for the channels selected from one of the communication terminals connected to the server on the same account,

- a calculation module to give the instruction to each terminal connected to the server on the same account, to apply - in accordance with the parameters received - a transformation and a projection of the selected channels in an RGB space, directly in their graphics processors .

Other objects and advantages of the invention will emerge in the light of the description below, made with reference to the attached drawings and listings, in which:

FIG.1 is an illustration of the overall communication system. FIG. 2 represents a functional diagram of the transformation and the projection in an RGB color space for an example of image in 8-bit format.

FIG. 3 represents a functional diagram of the transformation

and projection in a space of color RGB for one example 16-bit image format. FIG. 4 is an illustration of the stage of storage for a example image in 8-bit format. FIG. 5 is an illustration of the stage of storage for a

example image in 16-bit format.

FIG. 6 represents a functional diagram of the transformation and the projection in an RGB color space for an example of image in 8-bit format.

FIG. 7 represents a functional diagram of the transformation and the projection in an RGB color space for an example of image in 16-bit format.

LISTING.1 is an example of a sequential implementation of the transformation and projection steps.

LISTING.2 is an example of parallel implementation of the transformation and projection stages.

FIG. 1 represents the global communication system. It includes a server 1 with a memory unit 2 and a central processor 3 (CPU or Central Processing Unit in English). The memory unit 2 makes it possible to store multichannel images in 2D texture units. A 2D texture unit is a memory storage unit whose size is linked to a graphics processor. The step of transforming a multichannel image into several 2D texture units is carried out by the central processor 3 of the server 1.

Several users, regardless of their number, are connected to the server 1 through a network accessible from any web browser, and from any type of communication terminal 4 such as a smartphone, a tablet or a computer, as illustrated in FIG.1.

When one of the connected users sends parameters to the server 1 (arrow A), the latter commands the connected communication terminals 4 to carry out the calculation steps in their graphics processors (arrows B). Any logged in user can enter new settings.

The high speed of the calculation method proposed by the present invention allows a group of users to view the same data at the same time, and the modifications made in real time. In the present description, the term real time refers to an operation whose execution time is less than 40 milliseconds.

The modifications that users can make on multichannel images are:

the selection of one or more channels to display on the screens of the communication terminals, the transformation parameters for each channel, the projection parameters for each channel.

The transformation and projection steps are executed on the graphics processor of each connected terminal. FIG. 2 is a functional diagram of the transformation and projection steps in the case of an example image in 8-bit format. For comparison, FIG.3 illustrates these steps for an example of a 16-bit format image. For a 16-bit format image, there is an additional step of converting 8-bit format data.

The steps illustrated in FIG.2 and FIG.3 are the transformation and projection steps. The transformation operation noted H in FIG.2 and FIG.3 takes two input data for each channel.

This operation can for example be a contrast adjustment. The following description often refers to a particular contrast adjustment operation by way of example, but the present invention is not limited to this type of transformation. Any type of transformation involving a modification of the pixel values can be applied (e.g. deflouting, segmentation, etc.).

A contrast adjustment operation can be expressed mathematically by:

given x an input value in [0.255] or [0.2 ¹⁶ -1], and considering two parameters ThL and ThH, six <ThL maxsi x> ThH

- (x - ThL) otherwise lH — ThL

In the case of an image in 8-bit format, the maximum value is equal to 255, and in the case of an image in 16-bit format, the maximum value is equal to 2 ¹⁶ - 1.

The projection operation f in FIG.2 and FIG.3 is a linear projection of the channels in an RGB color space. It is defined by a matrix M whose values are adjusted by a user. The matrix M has as many rows as there are input channels and three columns.

The projection and transformation steps are performed in real time, which allows users to make changes and view them at the same time.

The input channels of the transformation stage are of particular shape. These are stored on server 1 as 2D texture units. A 2D texture unit is a memory storage unit whose size is fixed and linked to a graphics processor. A 2D texture unit, in accordance with the present invention, is, in particular, a packet of 8-bit format tiles corresponding to one or more channels at a given position in the multichannel image concerned.

Most of the time, a 2D texture unit is used to display an RGBA image i.e. the content of the 2D texture unit is directly displayed on a screen. In the specific use made by the present invention, the 2D texture units are used only for their function of memory storage units in order to facilitate and improve the speed of the whole process.

The operations H and f presented above are executed directly on the 2D texture units.

The step of storing a multichannel image in 8-bit format tile packets, according to the present invention, is pre-calculated on the central processor 3 of the server 1. The storage method is illustrated in FIG. 4 for a 8-bit format image and in FIG.5 for a 16-bit format image.

The storage method consists of two stages: a stage called tilling and another stage called packing. With reference to FIG. 4, the tilling step consists in breaking down a given channel into a regular grid of tiles of the same dimensions. After the tilling step, each channel is a bundle of several tiles, and each tile represents a portion of the multichannel image. For example, tiles 100, 200, 300, 400, and 500 correspond to the same position in the multi-channel image. Each tile has the same dimensions in pixels as a 2D texture unit i.e. 512 x 512 pixels.

The packing step consists of grouping each tile for a given position into bundles of four channels. In the example in FIG. 4, the input image has five channels. The tiles at the top left are grouped into a package (100, 200, 300 and 400) and another package containing tile 500. In this example, the number of packages is two. The number of packets is noted K and the number of files obtained is noted N in FIG. 4.

The number of tiles, packages and files obtained depends on the number of input channels and the dimensions of the input image.

The role of the packing step is to minimize the number of tiles by grouping them into packages. A pack of four tiles is also called a 2D texture unit. In the example in FIG. 4, all the tiles are 8-bit format.

We now refer to the storage step illustrated in FIG. 5 for an example of a 16-bit multi-channel image. It includes the same steps as in the case of an 8bit image, namely a tilling step and a packing step. In addition, it includes an additional operation called S in FIG.5. S is a partitioning operation consisting in separating the highest eight bits from the lowest eight bits for a given value of 16-bit format. This operation is performed after the tilling step, at the end of which the tiles obtained are 16-bit format. For example, the 16-bit format tile located at the top left and numbered 100 is divided into two 8-bit format tiles: 100h containing the highest eight bits for each pixel value and 1001 containing the highest eight bits weak for each pixel value.

FIG.6 shows a functional diagram of the transformation and projection steps in the case of the same 8-bit format image of FIG.4. In particular, it shows the different steps performed for the first file comprising two 2D texture units. The same operations are carried out on all the other texture units illustrated in FIG. 4 but are not shown here.

T1 and T2 are two 2D RGBA texture units. The transformation H is applied to all 2D texture units in parallel, i.e. all the pixels of the 2D texture units are calculated at the same time. After the transformation step, the projection operation f is also applied in parallel to all the pixels, which allows the display of the image in real time (i.e. in less than six milliseconds).

We now refer to FIG.7 representing a functional diagram of the transformation and projection steps in the case of the same 16-bit format image as in FIG.5. In particular, it shows the different operations applied to the first file of two 2D texture units. The same operations are applied to all the other 2D texture units illustrated in FIG. 5 but not shown here.

In this diagram, there are two additional operations denoted Sr and 8-bit conversion. Operation Sr is the inverse function of function S illustrated in FIG.5. Given two 8bit format values, the Sr function recomposes the 16-bit format value. The 16-bit format values of the pixels are transformed and projected in an RGB color space directly before the 8-bit conversion. In the same way as for the 8-bit format, all the pixels are calculated in parallel.

In order to compare a sequential algorithm and a parallel algorithm, we refer to the pseudocodes LISTING.1 and LISTING.2, respectively. These are given by way of example in the case where the transformation operation is a contrast adjustment. According to LISTING.1, the direct approach consists of a sequential implementation of the steps typically at the level of a central processor. With this type of implementation, the pixels are calculated sequentially. From the moment a user modifies the ThLs and ThHs parameters of the transformation vector, the time elapsed before the image is displayed on the screen is too long. For example, the average time to display an image of three 8-bit channels is between two hundred and three hundred milliseconds. Therefore, the direct approach does not allow the parameters entered by the user to be updated in real time.

As illustrated in LISTING.2, the implementation method of the present invention uses the parallel programming capacities (parallel computing in English) of graphics processors to calculate all the pixels p in parallel instead of sequentially. Another difference from the sequential approach is in the input resources. With this method, channels are saved as 2D texture units i.e. in packs of four tiles rather than in the memory system. The input channels refer to the vector T of K 2D texture units. The example of LISTING.2 is for a file. The N files corresponding to the m channels selected are calculated in parallel with the present implementation.

For example, with the present invention, the average time elapsed before displaying five 8-bit channels is between two and six milliseconds. The speed of execution of this method allows the parameters entered by the user to be updated in real time.

Claims (9)

1. Method for viewing and manipulating a multichannel image, the channels of which come from one or more imaging modalities, said method comprising the steps of: - storage of the multichannel image in one or more 2D texture units, each 2D texture unit representing a packet of four 8-bit format tiles corresponding to one or more channels at a given position on the multichannel image, and each 2D texture unit being a memory storage unit whose size is linked to a graphics processor, - taking into account a selection of one or more channels of the multichannel image, - taking into account transformation and / or projection parameters for the selected channels, - application of a transformation on the selected channels, in accordance with the parameters received, all the pixels of the channels being calculated at the same time, - projection of the selected channels in an RGB color space, in accordance with the parameters received, all the pixels of the selected channels being projected at the same time. 2. Method according to claim 1, characterized in that the transformation applied is a change in image contrast. 3. Method according to any one of claims 1 or 2, characterized in that the storage step carried out for an image of 8-bit format, includes the steps of: - partitioning of each channel of the multichannel image into a regular grid of 8-bit tiles of the same dimensions, and of the same dimensions as a 2D texture unit, - grouping of 8-bit tiles corresponding to a given position in the regular grid in one or more 2D texture units. 4. Method according to any one of claims 1 to 3, characterized in that the transformation step, carried out for an image of 8-bit format, comprises the steps of: - assignment of channel intensity values to the pixels from the values stored in the channels (R, G, B, A) of the 2D texture units, - calculation of the new pixel values in accordance with the transformation parameters. 5. Method according to any one of claims 1 to 4, characterized in that the projection step, carried out for an image of 8-bit format, comprises an operation of calculation of the linear projection of all the pixels in a space RGB color, according to the projection parameters. 6. Method according to any one of claims 1 or 2, characterized in that the storage step carried out on a 16-bit format image, comprises the operations of: - partitioning of each channel of the multichannel image into a regular grid of 16-bit tiles of the same dimensions, and of the same dimensions in pixels as a 2D texture unit, - partitioning of each 16-bit tile into a pair of two 8-bit format tiles by separating the highest eight bits from the lowest eight bits for each 16bit pixel value, - grouping of 8-bit tiles corresponding to a given position in the regular grid in one or more 2D texture units. 7. Method according to claim 6, characterized in that the transformation step carried out on a 16-bit format image, comprises the steps of: - conversion of each consecutive pair of channels of 2D texture units in 8-bit format into values of 16-bit format, - assigning channel intensity value to pixels from 16-bit format values, - calculation of the new pixel values in accordance with the transformation parameters. 8. Method according to claim 7, characterized in that the projection step carried out for a 16-bit format image, comprises the steps of: - calculation of the linear projection of all pixels in an RGB color space in accordance with the projection parameters, - conversion of each channel obtained R, G and B into 8-bit format channel. 9. System, for manipulating a multichannel image, said multichannel image being composed of several channels, said system comprising a server (1) comprising: - a module of conversion for convert images multichannel in one or more units of 2D texture representing a pack of four tiles 8-bit format corresponding to one or more channels to a position given in the multichannel image, a 2D texture unit being a memory storage unit whose size is linked to a graphics processor, - a memory module for the storage of the 2D texture units obtained, - a connection module to connect, on the same account, one or more communication terminals (4), such as a personal computer, a digital tablet, or a smartphone, a selection module for receiving a selection of one or more channels of a multichannel image stored on the server (1), from one of the communication terminals (4) connected to the server on the same account, - an adjustment module to receive the transformation and projection parameters as inputs for the channels selected from one of the communication terminals (4) connected to the server on the same account, - a calculation module to instruct each terminal (4) connected to the server (1) on the same account, to apply, in accordance with the parameters received, a transformation and a projection in a color space RGB, directly at the level of their graphics. processors ³ ° S3S ₀₇ V 2/9