EP1963970A2

EP1963970A2 - Method for providing data to a digital processing means

Info

Publication number: EP1963970A2
Application number: EP06847178A
Authority: EP
Inventors: Bruno Liege
Original assignee: Dxo Labs SA
Current assignee: GoPro Technology France SAS
Priority date: 2005-12-19
Filing date: 2006-12-19
Publication date: 2008-09-03
Also published as: JP2009520256A; JP5254030B2; KR20080087123A; FR2895104A1; US20090037877A1; KR101391569B1; US8473720B2; WO2007071882A2; WO2007071882A8; CN101379469A

Abstract

A method for providing generic formatted data (12) to at least one digital data processing means (10) for translating the generic formatted data into specific formatted data (18), wherein the generic formatted data (12) includes data on logic blocks of which at least one relates to an object (22) to be processed directly or indirectly in accordance with the specific formatted data (18) by a processing platform (20) that has one or more processors and memories and is located downstream from or built into the processing means (10), and the object (22) consists of basic information items of the same kind, each of which is represented by at least one numerical value.

Description

METHOD FOR PROVIDING DATA TO A MEANS OF PROCESSING

DIGITAL

The present invention relates to a method for providing generic formatted data to a digital data processing means.

When it is desired to develop new algorithms for one or more object processing platforms with strong constraints in terms of memory consumed and computational power consumed, the development time is very long. Indeed, each platform has its own characteristics, both in terms of hardware (for example number and type of processors or size and type of memory), and in terms of language used (C for a scalar processor and assembler for a vector processor ). Thus, even if an algorithm has already been developed for one platform, its optimized adaptation for another platform requires an almost complete rewrite of the program or code.

For example, a signal processing processor has a small local memory and must copy the data from and to the main memory from this local memory. For example again, a vector processor, in particular of the SIMD (Single Instruction Multiple Data) type, processing data grouped by vectors of different sizes according to the platform and the vectors are not expressible independently of the platform when using a language such as the language C. These adaptations constitute an obstacle to the diffusion of new algorithms since, for example, adapting an image processing software to a type of camera may require several months of development.

However, the market for camera devices, especially cameras equipped with a camera-type camera, is very dynamic, with a renewal of the range every six to twelve months and a very rapid change in functionality. . It is therefore essential to be able to quickly adapt these new features to be able to follow the market.

In addition, the computing power is increasing very quickly and it is therefore possible to take advantage of this performance with more and more powerful algorithms, provided that they can be optimized quickly. On the other hand, the characteristics of the sensors evolve very quickly and it is therefore necessary to adapt the image processing algorithms to the characteristics of these sensors, including noise that increases with miniaturization, as long as we can optimize quickly.

The invention starts from the observation that, in order to overcome these drawbacks, the algorithms should be developed in a language that is independent of the organization of the data and the order of processing, without explicitly coding these informations manually in order to remain independent of the platform and leave room for automatic tools.

Existing languages do not achieve this goal, since they all impose conditions that characterize the algorithms for a given platform. Thus, for example, imperative languages such as the C language require the explicit coding of loops, and thus the definition of a processing order. Table languages such as Matlab, for their part, allow the mixing of table processes with direct processing on the elementary information, these direct processes having the consequence of breaking the regularity of the code and preventing automatic optimizations pushed.

Moreover, with known languages, to perform operations on images with relative displacements or relative scales, in particular a function of the absolute position in the object, it is necessary to use explicit manipulations of the elementary information using access to pixels and loops, which introduces a strong dependence of the code on the platform, in terms, in particular, of memory and parallelism.

Thus, programs written with such languages are dependent on the data processing platform, since they contain instructions directly involving the elementary information.

The purpose of the invention is to provide generic formatted data to overcome at least one of the aforementioned drawbacks and thus to accelerate the placing on the market of data processing hardware and software, particularly images, in particular automatically and quickly obtaining an optimized implementation for various platforms of any algorithm.

In addition, the invention makes it possible to modify the algorithms late, for example for:

- benefit from the performance of processors as soon as they are placed on the market, and / or - benefit from the latest advances in data processing, including images, and / or

- Take into account the particular characteristics of the platform, for example, in the case of an image capture device, quickly adapt an algorithm to the characteristics of a sensor and / or optics.

The invention thus relates to a method for providing generic formatted data to at least one digital data processing means for translating the generic formatted data into specific formatted data. The generic formatted data comprises data relating to logical blocks, at least one of the logical blocks corresponding to an object to be processed based, directly or indirectly, specific formatted data by at least one processing platform with processor (s) and memory (s), located downstream of the processing means or integrated into the processing means. The object to be processed consists of elementary information of the same nature, each of which is represented by at least one numerical value.

The method comprises the following steps:

- Data is provided by digital processing first data, forming part of the generic formatted data, and describing at least one sequence of generic operations to be performed on at least one logic block and / or on at least one parameter.

- Second data is provided by means of digital processing, for generic operations involving multiple logical blocks.

These second data are part of the generic formatted data and relate to the relative position, according to at least one dimension of the object, in particular spatial and / or temporal, of the blocks and / or parameters, some by relative to the others, and / or relating to the relative scale, according to at least one dimension of the object, in particular spatial and / or temporal, logic blocks and / or parameters with respect to each other.

The generic formatted data is digital data describing a processing to be performed on an object in one or more dimensions, included in the group comprising: an image, a digitized sound signal, a sequence of images, a modulated signal, a simulation data item and a digital mesh.

An object to be processed is a physical entity consisting of physical data in the form of elementary information included in the group comprising: the pixels of an image, the samples of a digitized sound signal, the intensity and the phase of a signal modulated, a data characterizing the state of a simulation datum, the temporal and / or spatial information characterizing the points of a digital mesh.

The generic formatted data provided by the processing means excludes the data relating to the identification of the basic information and the order in which the basic information will be processed by the platform, so that the generic formatted data is independent of the platform. treatment used. These generic formatted data are also independent of data relating to the decomposition of the object into sub-objects, the shape, size and overlap of these sub-objects, and the storage of the elementary information constituting these sub-objects in a memory.

These data relating to the order and decomposition into sub-objects are determined by the processing means. In the present description, the generic formatted data is digital data for describing a processing to be performed on an object by a data processing platform, regardless of the platform itself. These generic formatted data are, in the context of the invention, provided to a processing means that may contain the processing platform, or be upstream of it. The treated objects correspond to a set of basic information of the same nature; these objects are for example images, digitized sounds, video or simulation data.

Preferably, the generic sequence of operations does not include a loop.

In one embodiment, the second data are part of the generic formatted data and relate to the relative position, according to at least one dimension of the particular spatial and / or temporal object, of the blocks and / or parameters relative to one another. , and / or relate to the relative scale, according to at least one dimension of the particular spatial and / or temporal object, logic blocks and / or parameters with respect to each other.

In one embodiment, the objects and sub-objects as well as the logical blocks have several dimensions.

The dimensions of the subobjects and logical blocks correspond to all or part of the dimensions of the object. The dimensions can be of various natures, in particular: -spatial, for example a distance, an angle or a course in a mesh, -temporal,

-frequency, for example a color, a frequency, a frequency band -a phase, a decomposition according to another vector space base, for example a wavelet decomposition

-in general, the dimensions of any topology space.

The following non-limiting list gives examples of objects with their dimensions:

a fixed image with 2 dimensions, each corresponding to distances, in particular measured in pixels, a raw fixed image with 2 dimensions, each corresponding to distances, the pixels each being provided with a color, for example red, green or blue,

a fixed image in color, with 2 dimensions, each corresponding to distances, and a dimension corresponding to a frequency representing the color channel, for example red / green / blue,

an animated image with 3 dimensions, corresponding, for 2 of them, to distances, in particular measured in pixels and for one of them to time, a relief image with 3 dimensions, corresponding to distances,

a relief image with 3 dimensions, corresponding, for 2 of them at distances, and for the third at an angle of view, a medical image with distance dimensions and possibly channel dimensions,

-a hologram with angle of view dimensions,

more generally, an image with dimensions of distance and / or of angle and / or temporal, and / or frequency,

a sound with a dimension corresponding to the time,

-a sound with 2 dimensions corresponding to time and a channel, a modulated signal with one or more dimensions corresponding to time and, optionally, a frequency and, optionally, a position in space or an angle,

signal modulation and demodulation, measurement, data analysis, indexing or searching a database, computer vision, graphic processing, simulation represented by elementary information arranged according to one or more dimensions,

more generally an object with one or more dimensions.

In the context of the present invention, the elementary information of an object may have a position and / or an absolute scale, in particular spatial and / or temporal and / or frequency, but also according to at least one dimension of the object as well as in any space, especially a space consisting of wavelets:

- an elementary information of an object "sound" can correspond to an intensity; in this case, the elementary information has an absolute position corresponding to a given instant and, in the case of a multichannel sound, to a given channel.

an elementary information of an "image" object can correspond to a pixel; in this case, the elementary information has an absolute position corresponding to a position in the image and, in the case of a video image, at a given moment.

- An elementary information of a "simulation data" object can correspond to a state; in this case, the elementary information has an absolute position corresponding to a mesh node and at a given moment. an elementary information of a "modulated signal" object may correspond to an intensity and / or a phase; in this case, an elementary information has an absolute position corresponding to a given instant and, possibly, to a given frequency and / or to a given position, if several antennas or transmitters are used. Elementary information is a piece of information to be processed, represented by one or more numerical values. This information can be encoded according to various types of coding such as 8-bit signed coding, 10-bit coding, or signed 16-bit coding. In the case where the object is an image, for example, the basic information will be the pixels of this image.

The objects can be raw images ("raw" type) before demosaicing operation, in which case an elementary information is a pixel represented by a corresponding numerical value, according to the absolute position of the pixel, for example to red, green or blue. The objects may also be visible images, in which case an elementary information is a pixel represented, for example, by three numerical values, each representing a color, for example red, green and blue.

The objects can also be sequences of images, in particular raw or visible, in which case an elementary information is a pixel of an image of the image sequence. Objects, for example, correspond to videos.

In the case where the object is an image, the image may be derived from an image capture apparatus and / or intended for an image reproduction apparatus:

- An image capture device is, for example, a disposable camera, a digital camera, a DSLR (digital or not), a scanner, a fax, an endoscope, a camera, a camcorder, a camera of surveillance, a toy, a camera or a camera integrated or connected to a telephone, a personal assistant or a computer, a thermal camera, an ultrasound machine, an MRI (magnetic resonance) imaging apparatus, a device X-ray radiography. An image rendering apparatus is, for example, a screen, a projector, a television set, virtual reality glasses, or a printer.

A device for capturing and restoring images is, for example, a scanner / fax / printer, a mini lab for printing photos, a video conference device.

The processing platform can take various forms depending on the application. By way of example, in the case where the object is an image, mention will be made in particular of the case where the processing platform is integrated with one of the following devices:

An image capture apparatus that produces processed images, for example a digital camera that integrates a processing platform.

An image rendering apparatus which displays or prints processed images, for example a video projector or a printer including a processing platform.

- A mixed device that corrects the defects of its elements, for example a scanner / printer / fax including a processing platform. - A professional image capture device that produces processed images, for example an endoscope including a processing platform.

The processing platform can be deported in whole or in part to a server.

The processing that will be applied to the object in the platform corresponds to an algorithm, described by one or more sequences of generic operations, which can intervene in various domains such as, for example, image processing, compression and decompression data processing, signal modulation and demodulation, measurement, data analysis, database indexing or search, computer vision, graphics processing, simulation or domain implementing a large amount of data. Generic operations are operations that apply to logical blocks, that is abstract entities, without any notion of size, shape, or time. Generic operations can produce logical blocks. Preferably, at least one logic block corresponds to the object to be processed.

The notion of relative scale and relative position of two logical blocks and / or parameter allows in particular, without the list being limiting, to:

- combine several logical blocks and / or variable parameters according to the absolute position, and / or

- change scale, and / or - make filters on neighborhoods, and / or

determining the value of a parameter when the value depends on the absolute position of the elementary information of the object, and / or

- change of representation.

In the context of the invention, the logical blocks are of several types according to the generic operation, for example, in the case where the object is an image,

at least one logic block is of "raw" type, and / or at least one logical block is of red type, and / or

at least one logical block is of green type, and / or

at least one logic block is of blue type, and / or

at least one logic block can be represented by 8-bit data, and / or at least one logic block can be represented by 16-bit data.

In one embodiment, at least one logic block contains multiscale data, for example scaled data 1, H and 1/8. This allows for generic operations at multiple scales, and then combine the results

At the moment of the processing, the logical blocks may, in one embodiment, correspond, according to the type of the logical block, to physical blocks consisting of elements of the same nature as the elementary information or else constituted by elements of a different nature than the information elements. elementary. The notion of absolute or relative position is independent of the type of the logical block. For the purposes of the invention, the notion of elementary information applies to all elements regardless of the type of the logic block.

An algorithm may, for example, without the list being limiting, correspond, in the case where the object, is an image to:

- a calculation, including statistics, for a white balance, and / or

- a calculation, in particular statistics, for an automatic exposure, and / or

a calculation, in particular statistics, for automatic focusing, and / or

a calculation, in particular statistics, for an automatic contrast enhancement, and / or a raw image to image ("pipe image") conversion, and / or

- correction of optical defects, and / or

- correction of sensor faults, and / or

- a correction of imager defects, and / or - a treatment, in particular improvement of sharpness, and / or

a treatment, in particular for improving the color rendition, and / or

a treatment, in particular for improving the rendering of the contrast, and / or a treatment, in particular for improving the rendering of the details, and / or

- noise reduction, and / or

- a measurement, and / or - a compression, and / or

- decompression, and / or

- interpolation or magnification (zoom), and / or

- a scan, and / or

- A special effect.

The object to be processed is generally, during the processing, broken down into sub-objects, that is to say into groups of elementary information having a determined size and shape, both in space and in time, or other dimensions of the object.

Generally, the generic operations of an algorithm involve, in addition to elementary information, parameters such as, for example, multiplying coefficients.

These parameters may correspond, for example, without the list being limited to:

- filter coefficients,

saturation values, offset values, and

- correspondence tables.

In one embodiment, the value of a parameter, depending on the nature of this parameter, can: be constant and intrinsic to the algorithm; in this case the parameter value may in particular be transmitted to the processing means or to the platform, and / or

- depend on the origin or destination of the object: for example, in the case where the object to be treated is an image from an apparatus with a given optic, the value of the parameter may depend on the type of optics that affects the level of blur in the image; in this case, the parameter value may, in particular, be transmitted to the processing means or to the platform, and / or

depending on the object to be processed: for example, in the case where the object to be processed is an image derived from a sensor, the value of the parameter may depend on the gain of the sensor actually used to capture said object which has an impact on the noise level in the image; in this case the parameter value can in particular be transmitted, chosen or calculated by the platform, and / or - depend on the absolute position of the elementary information in the object; in this case the parameter value may in particular be transmitted, chosen or calculated by the platform, and / or

- not depending on the absolute position of the elementary information in the object.

The parameter value can be determined simultaneously or a posteriori with respect to the definition of the algorithm.

We have seen that the value of some parameters may vary from one object to another, from one sub-object to another, or from one elementary information to another. In this case, in one embodiment, the value of the parameter is calculated at each change. In another embodiment, the possible values of the parameter are calculated a priori, and, at each change, the index or the address making it possible to access the value of the parameter, for example in a table, is determined. In another embodiment, more particularly adapted to parameters whose value varies from one sub-object to another depending on the absolute position of the sub-object and whose number of values is limited, for example the parameters corresponding to characteristics of blur of the optics, a limited number of sets of parameter values are determined, each set is stored and for each sub-object the set to be used is selected, for example by calculating a function of the position giving the address of the game to use.

In the case where the object is an image, in one embodiment the image is decomposed into juxtaposed rectangular sub-objects, the sub-objects being processed, for example, from left to right and then from top to bottom. Depending on the platform, the sub-objects are chosen and stored in one of the following ways, without the list being exhaustive:

in the case of a signal processor, having a small fast memory and a large slow memory, the size of the sub-objects is chosen to be able to process a sub-object without access to slow memory; we can, for example, take sub-objects corresponding to squares of 32x32 pixels: the result of the computation on the preceding sub-object is transferred in slow memory during the computation relating to the current sub-object, and during the transfer of the slow memory to the fast memory of the data necessary for the calculation relating to the following sub-object

in the case of a scalar processor having a small cache memory and a large slow memory, the size of the sub-objects is chosen to be able to process a sub-object using the cache memory as much as possible; we can, for example, take sub-objects corresponding to squares of 32x32 pixels or sub-objects of 1 pixel or sub-objects of 4 pixels, especially in the case of a raw image, tγpe "raw" ,

in the case of a vector processor, the size of the sub-objects is chosen as equal to or multiple of, the size of a vector that the platform knows how to process and store, it will be possible for example to take sub-objects corresponding to 64 horizontal pixels. In the case where the object is of a type other than an image, the decomposition into sub-object can be adapted in a manner similar to the platform.

In languages such as C, this decomposition into sub-objects and the coding of the corresponding loops must be explicit depending on the platform, and performed manually for each platform. There are vector compilers that try to extract from the explicitly coded loops the sub-objects, but their efficiency is limited because they have to extract the concept of the algorithm before automatically coding it, which is very difficult.

The invention thus makes it possible to automate and make more effective the adaptation to any platform.

The following table summarizes the main differences between the invention and the known languages:

In one embodiment, the first and second data and the values of the parameters are sufficient for the platform to process the object.

The first data further comprises information on the number of bits on which are encoded the values representing the logical blocks. The specific formatted data is the data obtained at the output of the processing means. They are, in fact, a translation of the generic formatted data, and in that they represent, in the same way, a processing algorithm intended to be applied to an object to be processed. This specific formatted data contains particular operations, derived from generic operations, and describing the instructions that will be performed on the object itself during processing in the platform. The generic formatted data according to the invention contains enough information to allow the calculation of these specific formatted data, without however being dependent on this platform. Indeed, the operations defined by the generic formatted data are generic operations applying to logical blocks, as defined above. Thus, the first data exclude data describing generic operations involving one or more elementary information independently of the other elementary information belonging to the same logical block (s). Indeed, a logical block being an abstract entity, it is not possible to distinguish and identify the different elementary information that compose it.

Thus, the generic formatted data can be provided to different processing platforms, regardless of the type of architecture of the platform and also regardless of the format of the object to be processed. For example, the generic formatted data will be the same for a 3 million pixel image as for a 5 million pixel image. In this way, when a new algorithm is developed, it is enough to define once only generic formatted data to be able to operate, after a very short delay, this algorithm in an optimized way in terms of performance and memory size on different platforms. .

Depending on the type of generic operation involved in the processing algorithm, the second data may have different characteristics.

According to one embodiment of the invention, at least part of the second data provided to the processing means and / or the value of at least one parameter are, for a given generic operation, common to all the elementary information of the object to be treat. According to another embodiment, at least part of the second data provided to the processing means and / or the value of at least one parameter depend on the absolute position of the elementary information in the object to be processed. For example, using a parameter common to all the basic information for a filter makes it possible to increase the sharpness in a uniform manner.

For example, the use of a parameter dependent on the absolute position of the elementary information in the object to be processed, for a filter, makes it possible to increase the sharpness more significantly at the edge in order to compensate for an optical defect.

For example, the use of a parameter dependent on the absolute position of the elementary information in the object to be processed for a vignetting correction makes it possible to obtain a stronger edge compensation in order to compensate for an optical defect.

For example, the use of a parameter depending on the absolute position of the elementary information in the object to be treated for a demosaicing process makes it possible to treat the "red" pixels, the "green" pixels and the pixels differently. "Blue" of a raw image from a sensor.

For example, the use of second data, in particular a displacement, which depends on the absolute position of the elementary information in the object to be processed for a numerical zooming calculation ("zoom") or a distortion correction, makes it possible to obtain the pixels needed to calculate the interpolation at each point.

For example, the use of second data, in particular a displacement, depending on the absolute position of the elementary information in the object to be processed for a format change, makes it possible to obtain the pixels necessary for the new format. Thus one can, for example, obtain the pixel values for a YUV type color coding from the separate luminance (Y) and red and blue (U and V) chrominance information.

In one embodiment of the invention, the generic operations comprise at least one generic position operation that makes it possible to obtain a logical block consisting of the absolute position along one dimension of the object, as well as a generic indirect operation, which makes it possible to obtain, from a first block, a second block by displacement and / or scaling according to a third block or a parameter.

In another embodiment, the generic operations comprise at least one elementary generic operation included in the group comprising: the addition of logic blocks and / or parameters, the subtraction of logic blocks and / or parameters, the calculation of the absolute value the difference between logical blocks, the multiplication of logical blocks and / or parameters, the maximum among at least two logical blocks and / or parameters, the minimum of at least two logical blocks and / or parameters grouping and unbundling of blocks logic, the calculation of a logic block by applying a parameter, corresponding to a correspondence table, to a logic block, the conditional choice of a logical block among at least two logical blocks and / or parameters, this choice is doing the following: if a> b we choose c, otherwise we choose d, with a, b, c, and d which are logical blocks and / or parameters, the histogram of a logical block, the change d scale of a logic block according to a parameter and / or a logic block, the relative displacement of a logic block according to a parameter and / or a logic block, and an operation producing a block containing at least one coordinate.

Generic operations implementing a logical block and a parameter, such as addition, can be translated into processing in the platform, and correspond, for example when the generic operation is an addition, adding each element or elementary information of the processed physical block, corresponding to the logic block, with the value of the parameter corresponding to the absolute position of the element or of the elementary information processed.

These operations as such are relatively common operations in image processing but also in the processing of other kinds of objects, on the other hand, when they are applied to logic blocks in combination with the second data, these operations solve the problem. These operations serve as a basis for forming all the operations that can be implemented in a data processing algorithm. Thus, in one embodiment, the generic operations include complex generic operations corresponding to groups of generic elementary operations used as such. Among these groups, there may be mentioned in particular: the calculation of the median value of at least three logical blocks and / or parameters, which corresponds to a group of generic operations consisting of calculations of minimum and maximum, the multiplication / accumulation of logical blocks and / or parameters, the convolution of a logical block with a parameter, which corresponds to a group of generic operations consisting of multiplications and additions with several relative positions, the combined addition with a maximum and a minimum , the calculation of a gradient, which corresponds to an absolute value of differences with two relative positions, the scalar product of a parameter consisting of a vector and several logical blocks to produce a logical block, the calculation of a change Interpolated scale scale that corresponds to a group of generic operations consisting of scale and multiplication changes and additions with several relative positions. s, the combination of logical blocks, which corresponds to a group of generic operations consisting of changes of scale with several relative positions.

Some of the operations use multiple logical blocks. It has been seen that, in this case, second data are provided relating to the positions of the logical blocks involved with respect to each other.

Relative positions and relative scales may correspond to various concepts depending on the nature of the object. They apply between any 2 blocks, whatever their type (in the case of an image as described above, a logic block can in particular be raw, red, green, 8 bits ...).

In the case where the object is a fixed image with square pixels, the absolute or relative position and the absolute or relative scale may correspond, in one embodiment, each to 2 values (vertical and horizontal); the pixels in the top line of an object can have as absolute positions (0; 0)

(0; 1) (0; 2) ..., and the pixels of the n th line can have as absolute positions (n; 0) (n; 1) (n; 2); in this case the relative positions can be coded in the following way: (- 1; 0) indicates at the top, (0; 1) indicates at the right and (2; -2) indicates 2 pixels below and 2 at the left; a relative scale of (0.5; 0.5) corresponds to a resolution of half in each direction. More generally, a combination of relative displacement and relative scale can be encoded using 2 functions f and g as follows: (f (x; y); g (x; y))) for each pixel of absolute position x, y. It should be noted that a rounding rule is necessary in order to take, for example, the nearest pixel. So :

a relative position above is coded with f (x; y) = - 1 and g (x; y) = 0

a relative scale of 0.7 is coded with f (x; y) = 0.7 * (x-x0) and g (x; y) = 0.7 * (y-y0); xO and yO corresponding to a parameter relating to an absolute position; - a distortion correction is coded with f and g corresponding to the distortion field

- a YUV 4: 2: 2 format change namely YYUV from the luminance and Y, U and V separated chrominance information can be used to obtain Y1 Y2 xx xx Y3 Y4 xx xx ... from Y1 Y2 Y3 Y4 ... the following functions f (x; y) = (x- xO) * O, 5 if x even, and (x + l-xO) * O, 5 if x-xO odd and f (y) = y-yθ; xO and yO corresponding to a parameter relating to an absolute position; The following list, non-limiting, gives other examples of realization with various types of objects:

in the case where the object is a fixed image with hexagonal pixels arranged per line, two successive lines being offset by one half-pixel, the absolute or relative position and the absolute or relative scale may each correspond to two values ( vertical and horizontal); the pixels in the top line of an object can have as absolute positions (0; 0)

(0; 1) (0; 2) ..., and the pixels of the n th line can have as absolute positions (n; 0,5) (n; 1,5) (n; 2,5). if the line is odd, and (n; 0) (n; 1) (n; 2) ... if the line is even; the relative position can correspond to 2 values

(vertical and horizontal), for example (-0.5, 0.5) indicates in the upper right, (0.1) indicates on the right and (-0.5, 1.5) indicates the pixel on the right of the pixel at the top right; a relative scale of (0.5; 0.5) corresponds to a resolution of half in each direction. Similarly, a combination of relative displacement and relative scale can be coded using 2 functions f and g as follows:

(f (x; y); g (x; y))) for each pixel of absolute position x, y. It should be noted that a rounding rule is necessary in order to take, for example, the nearest pixel.

in the case where the object is an animated image with square pixels, the absolute or relative position may correspond to 3 values (vertical, horizontal and temporal), for example, (- 1; 0; 0) indicates a pixel located in high in the same picture, (0; 0; -1) indicates the pixel having the same position in the previous image and (2; -2, -1) indicates the pixel located 2 pixels below and 2 left in the previous image. Similarly, a combination of relative displacement and relative scale can be encoded using 3 functions f, g, h as follows: (f (x; γ; t); g (x; y; t )); h (x; y; t)) for each pixel of absolute position x, y at time t. It should be noted that a rounding rule is necessary in order to take, for example, the nearest pixel. - in the case where the object is a single-channel sound, the absolute or relative position may correspond to 1 value

(Time); for example (-1) indicates the previous instant, and

(2) indicates 2 instants after; a function f (t) then makes it possible to encode a relative displacement and scale. A rounding rule being used to take the nearest instant.

- in the case where the object is a multi-channel sound, the absolute or relative position may correspond to 2 values

(temporal, channel); for example (-1, 0) indicates the previous instant of the same channel, and (2.1) indicates 2 instants after the next channel, ordered for example spatially in a circular manner. Similarly, a combination of relative displacement and relative scale can be encoded using 2 functions f, g as follows: (f (t; c); g (t; c)) for each sound sample position at time t for channel c. It should be noted that a rounding rule is necessary to take, for example, the instant and the nearest channel.

- In the case where the object is a simulation mesh, the absolute or relative position may correspond to n values, each corresponding to a spatial or temporal dimension depending on the topology of the mesh. Similarly, a combination of relative displacement and relative scale can be encoded using n functions. It should be noted that a rounding rule is necessary in order to take, for example, the node and the nearest instant. in the case where the object is a modulated signal, the absolute or relative position may correspond to n values, respectively corresponding to the time, if appropriate to the frequency channel (transmission or reception on several frequencies) and if necessary (several transmitters or spatially arranged receivers) to a spatial dimension. Similarly, a combination of relative displacement and relative scale can be encoded using n functions, and a rounding rule should be chosen. in the case where the object is a set of measurements, the absolute or relative position may correspond to n values, each corresponding to a dimension of the object which, depending on the case, may be of a temporal, spatial, frequency, phase or other. Similarly, a combination of relative displacement and relative scale can be encoded using n functions, and a rounding rule should be chosen.

in the general case where the object is of dimension n, the absolute or relative position may correspond to n values each corresponding to a dimension of the object which, depending on the case, may be temporal, spatial, frequency, phase Or other. Similarly, a combination of relative displacement and relative scale can be encoded using n functions, and a rounding rule should be chosen.

Thus, in one embodiment of the invention, the relative positions and / or the relative scales of the logical blocks according to at least one dimension of the object, in particular spatial and / or temporal, have integer values or fractional values as described in FIG. the examples above.

In one embodiment, the elementary information is represented by fixed-point numeric values. In this case, it is necessary to take into account this fixed point during the application of the operations, in order not to lose information in the results. For this purpose, in the same Embodiment of the invention, the generic operations comprise offset operations, a saturation operation and / or at least one elementary generic operation combined with this saturation operation.

In one embodiment, the parameters include information representing the type of coding of the elementary information.

For example, in the case where the elementary information is the pixels of an image, they are represented, for example, on 24 bits representing 3 colors in a red-green-blue (RGB) coding, or on N bits representing a color per pixel varying according to the position of the pixel for the raw images from a sensor, or 8 bits corresponding to the luminance information and chrominance Y or U or V for an image coded with a YUV type coding. It is therefore useful to provide the processing means with data enabling it to know the type of coding in order to correctly translate the generic formatted data. Preferably, the generic formatted data represents processing algorithms of objects constituting images, and which are composed of elementary information forming pixels.

Thus, in one embodiment, the processing platform is part of an image capture and / or rendering apparatus, and the values of the parameters are related to the characteristics of the optics and / or the sensor and / or the image. imager and / or electronics and / or software of the capture and / or image recovery apparatus. The characteristics can be, in particular, fixed intrinsic characteristics for all objects or variables depending on the object, for example noise characteristics that vary according to the gain of a sensor.

The characteristics can also be identical for all the elementary or variable information according to the absolute position of the elementary information, for example the fuzziness characteristics of the optics.

However, generic formatted data can represent other types of data processing algorithms. Thus, in one embodiment, the object to be processed is a digitized sound signal, and in this case, the elementary information is the sound samples of this signal. In this case, the relative positions present in the second data will generally be time positions. However, it may happen that these positions are spatial, especially in the case where the object to be treated is a sound present on several channels.

In another embodiment of the invention, particularly in the case of digital simulation, the object to be processed is a digital mesh and the elementary information is the spatial and / or temporal information characterizing each point of the mesh.

Other characteristics and advantages of the invention will become apparent with the nonlimiting description of some of its embodiments, this description being carried out using the figures in which:

FIG. 1 represents a data processing device using a method according to the invention; FIG. 2 represents a device similar to that of FIG. 1, in the case where several processing platforms are provided, and

FIG. 3 represents an example of a sequence of generic operations applied to several logical blocks and to a parameter.

The device shown in FIG. 1 is intended to process an image 22. This image comprises a set of pixels each of which is represented by at least one numerical value. In this device, generic data formatted data 12 is provided to digital data processing means 12. This processing means is a compiler in the example.

Generic formatted data, provided by a method according to the invention, includes first and second data that describe generic operation sequences and that describe the relative positions of the logical blocks involved in these generic operations. These first and second data will be illustrated in Table 1.

From these generic formatted data 12, the processing means 10 provides a processing platform

20, such as an apparatus for capturing or restoring images, directly or indirectly, for example via a compiler, specific formatted data 18.

The specific formatted data contains different types of data, such as data concerning the organization of the pixels in the platform memory, the order in which the pixels are processed by the platform or the specific operations performed by the platform, and their grouping.

The processing platform 20 then uses these specific formatted data 18 to process the image 22 it receives as input.

The system shown in FIG. 2 is similar to that shown in FIG. 1, in the case where two treatment platforms are provided.

In this case, two sets of generic formatted data 32a and 32b are provided to a processing means 30. These generic formatted data 32a and 32b each contain first and second data, respectively 34a for the generic formatted data 32a, and 34b for the generic formatted data 32b. These first and second data have the same characteristics as the first and second data described in connection with FIG.

This processing means 30 provides two platforms

40a and 40b specific formatted data 38a and 38b respectively corresponding to the generic formatted data

32a and 32b. The platform 40a thus processes an object 42a according to the data 32a or according to the data 32b. Likewise, the platform 40b processes an object 42b according to the data 32a or according to the data 32b. Thus, the same processing means 30 is used to provide formatted data specific to two different processing platforms, 40a and

40b.

FIG. 2 thus illustrates several advantages of the invention: in the case where the platforms 40a and 40b are identical, one can rapidly replace the generic formatted data 32a by 32b and translate them into specific formatted data 38b optimized for the platform. This reduces the time to market the platform. in the case where the platforms 40a and 40b are different, and the formatted data 32a and 32b are identical, they can quickly be translated into specific formatted data 38a and 38b optimized for each platform. This also reduces the time to market of multiple platforms.

Table 1 below and FIG. 3 show an example of a sequence of generic operations applied to a logic block B1. This sequence has three generic operations. The columns of the table represent in order: - the rank of the operation in the sequence, the name of the generic operation, the logical block (output) on which is written the result of the generic operation, it is the place where this result would be if we reconstituted the object at the end of each operation, the first input (input 1) of the generic operation, which may be a logic block or a parameter, the relative position of the logic block to be used with respect to the input logic block 1, if any, - the second input (input 2) of the generic operation, which may also be a logic block or a parameter, and the relative position of the logic block to be used with respect to the input logic block 2, if any.

The information in the columns

"Relative position" is the information present in the second data provided to a processing means by a method according to the invention. In this table, this information is in the form of "left" and "right" to be understandable, but in fact, in generic formatted data, it can also be encoded by numerical values such as (0; 1) as described. in the above exemplary embodiments and / or by functions such that f (x; y).

In one embodiment, the generic operations comprise at least one generic position operation which makes it possible to obtain a logical block consisting of the absolute position according to one dimension of the object, as well as a generic indirection operation which makes it possible to obtain , from a first block, a second block by displacement and / or scaling according to a third block or a parameter. The relative position and / or relative scale functions can then be calculated, for example, 0.5 * (x-100) using generic block operations, and then use the generic operation of indirection to effect relative displacement and / or relative relative scaling.

Table 1 is only an example of coding, the first data and second data can be coded in various ways in tabular form, but also in symbolic form, in graphic form or in any other form. Of in addition, additional information relating to data types, offsets and saturations are not shown for the sake of simplification of the example.

TABLE 1

The first logic block used in this sequence of operations is a logic block B1 (51). The first generic operation is an addition (52) between the logic block B1 shifted to the left (51g), and the logic block B1 shifted to the right (5Id). The result of this addition is written in block B2 (53): B2 = Lefthand + Bldroite.

The second operation (54) is a transformation of the block B2 (53) with respect to a table. This operation therefore has the block B2 (53) and a Param1 parameter (55) which represents the modification table. The result of this operation is written in block B3 (56): B3 = LUT (Param1, B2). The third and last operation (57) of this sequence is a multiplication of logical blocks. This operation has as inputs the logic block B3 (56) and the logic block B1 (51): B4 = B3 * B1.

The logic block B4 (58) is thus the block obtained at the end of the sequence of generic operations.

The generic formatted data in the example in Table 1 are platform-independent, from the object's decomposition to subobjects, from the way in which the object's basic information is traced, from the order in which the elementary information will be processed in the platform, as well as the organization in memory. Indeed, the data Table 1 generic formats can be translated in various ways into specific formatted data or code for the platform, for example, without the list being limiting, according to the following translations.

A first example of a translation, although not optimal in terms of memory and computation time, makes it possible to illustrate a simple translation without going through a decomposition into sub-objects:

For each pixel of the input object BP1 (corresponding to the logic block B1) excluding the two columns of left and right, the pixels being traversed from left to right then from top to bottom: Add the pixel located to the left of the current pixel and the pixel located on the right, store the result in a physical block BP2 (corresponding to the logic block B2).

For each pixel of BP2 traversed from left to right then from top to bottom: apply the table to the current pixel, and store the result in a physical block BP3 (corresponding to logic block B3)

For each BP3 pixel traveled from left to right and from top to bottom:

Multiply the current pixel by the corresponding pixel of BP1, and store the result in the output physical block BP4 (corresponding to the logic block B4).

A second example of translation shows that the size of the memory used can be decreased without changing the generic formatted data. Indeed, in the first example, 4 physical blocks of size close to the image are used. We can only use 2 physical blocks using the same memory for BP2, BP3 and BP4. We obtain the following translation:

For each pixel of the input object BP1 (corresponding to the logic block B1) excluding the two left and right columns, the pixels being traversed from left to right then from top to bottom:

Add the pixel to the left of the current pixel and the pixel to the right, store the result in a physical block BP2 (corresponding to the logic block B2).

For each pixel of BP2 traversed from left to right then from top to bottom: apply the table to the current pixel, and store the result in a physical block BP2 (now corresponding to logic block B3)

For each BP2 pixel traveled from left to right and then from top to bottom:

Multiply the current pixel by the corresponding pixel of BP1 and store the result in the output physical block BP2 (now corresponding to logic block B4).

A third example of translation shows that one can reduce the computation time without changing the generic formatted data. In fact, in the second example, two physical blocks of size close to the image are used, but the physical block BP2 is written 3 times entirely, the physical block BP1 is read out twice, and the physical block BP2 is read out 2 times entirely. . We can limit ourselves to reading and writing only with a different course mode and different blocks. This reduces the number of necessary instructions, but also the accesses to the memory. We obtain the following translation: For each pixel of the input object BP1 (corresponding to the logic block B1) excluding the two columns of left and right, the pixels being traversed from left to right then from top to bottom: Add the pixel located to the left of the current pixel and the pixel located on the right, apply the table to the result and multiply the output of the table by the current pixel, store the result in the physical output block BP2 (corresponding to logic block B4)

In a fourth example, more particularly adapted to a scalar processor with cache, the result is written in the same memory area as the input. This makes it possible to further reduce the size of the memory and to make local memory accesses, which is very favorable in the case of a cache memory or a paged memory. We thus obtain the following translation:

Add the pixel to the left of the current pixel and the pixel to the right, apply the table to the result and multiply the output of the table by the current pixel, store the result in the current output physical block BP1 instead of the pixel located to the left of the current pixel (the left pixel is no longer used later unlike the current pixel which will become the left pixel for the next iteration, BPl partially corresponds to logic block B4 and partially to logic block B1)

A fifth example of a translation is particularly suitable for a signal processing processor with a small fast memory and a large slow memory, each sub-object is a rectangle for example 32x32 or any other value maximizing the use of the fast memory, the rectangles being joined. We thus obtain the following translation:

For each sub-object, the sub-objects are traversed from left to right then from top to bottom:

Start a transfer by a direct memory access (DMA) mechanism of the next input physical block from the slow memory to the fast memory, corresponding to the next extended subobject of a column on the left and right, ie 32x34

Initiate a DMA (direct memory access) mechanism from the previous output physical block from fast memory to slow memory

We take as input the physical block corresponding to the extended current sub-object on an additional left and right column, 32x34, and obtained at the end of the DMA. from previous iteration

For each pixel of the input physical block (corresponding to the logical block B1) excluding the two columns of left and right traveled from left to right then from top to bottom: Add the pixel located to the left of the current pixel and the pixel located on the right, apply the table to the result and multiply the output of the table by the current pixel of the block, store the result in the current output physical block (corresponding to logic block B4).

A sixth example of a translation is particularly suitable for a vector processor capable of applying the same calculation to the various pixels of the vector, each sub-object is a rectangle for example 64 horizontal pixels or any other value equal to the size of a vector that the platform knows process and store. This translation does not require any memory because a vector is processed at a time. We thus obtain the following translation:

For each subobject Vl of the BPl entry object

(corresponding to the logic block B1) excluding the two columns on the left, the sub-objects being traversed from left to right then from top to bottom:

At the beginning of each line create a vector VO containing on the right the 2 pixels of left of the line

Extract from VO and Vl, the vector V2 corresponding to the two right-hand pixels of VO and the left-hand pixels of Vl excluding the 2 right-hand pixels of VO; add V1 and V2 to obtain V2, apply the table to each pixel of V2 to obtain V2, Extract from VO and V1, the vector V3 corresponding to the right pixel of VO and the left pixels of V1 excluding the pixel from the right of VO; copy Vl into VO for the next iteration; multiply V2 by V3 to obtain V2, store the result V2 in the current output physical block.

The examples of translations above show that it is possible, from the same generic formatted data, to translate in several ways with memory structures, loops and a degree of parallelism adapted to very diverse platforms. In particular we can: reduce the size of the code using only one loop, and / or reduce the memory size, to 0 in the example, but in the more general case where there are vertical filters, it you still need some lines of memory if you want to avoid redoing calculations for input data of vertical filters, and / or

- reduce the number of instructions needed, including grouping loops, and / or adapt to any vector size, and / or adapt to any memory architecture.

For simplicity, the examples produce a smaller image than the input image. It is easy, if necessary, to obtain an output image of the same size as the input image by adding code at the beginning and end of each line to duplicate the edge pixel.

Claims

REVENDIC-VTIONS

1. Method for providing generic formatted data

(12) describing a treatment to be performed on an object of one or more dimensions (22), included in the group consisting of: an image, a digitized sound signal, an image sequence, a modulated signal, a simulation data item and a digital mesh; the generic formatted data being provided to at least one digital data processing means (10) for translating the generic formatted data into specific formatted data (18), the generic formatted data (12) including data relating to logical blocks at least one of the logical blocks corresponding to an object (22) to be processed based, directly or indirectly, specific formatted data (18) by at least one processing platform (20) with processor (s) and memory ( s), located downstream of the processing means (10) or integrated in the processing means (10), the object (22) being constituted by elementary information of the same nature, each of which is represented by at least one a numerical value, this elementary information being included in the group comprising: the pixels of an image, the samples of a digitized sound signal, the intensity and the phase of a modulated signal, a d characterized by the state of a simulation datum, the temporal and / or spatial information characterizing the points of a digital mesh, this method comprising the following steps: the first data (14) is provided by means of digital processing, forming part of the generic formatted data, describing at least one sequence of generic operations to be performed on at least one logical block and / or on at least one parameter, and - providing digital processing means, for the generic operations involving several blocks logic, second data (14), forming part of the generic formatted data and relating to the relative position, according to at least one dimension of the object, especially spatial and / or temporal, logic blocks and / or parameters with respect to each other and / or relating to the relative scale, in particular spatial and / or temporal, of the logic blocks and / or parameters, some by in relation to others, the generic formatted data (12) provided by digital processing (10) excluding data relating to the identification of basic information and the order in which the basic information will be processed by the platform, so that the generic formatted data (12) is independent of the processing platform (20) used, and the provided generic formatted data (12) being independent of data relating to the decomposition of the object into sub-objects, to the shape, size and recovering these sub-objects, and storing the elementary information in a memory, these data relating to the order and those relating to the decomposition e n sub-objects being determinable by the processing means (10).

2. Method according to claim 1, in which at least part of the second data provided to the processing means and / or the value of at least one parameter are, for a given generic operation, common to all the elementary information of the object. treat.

3. Method according to claim 1 or 2 wherein the second data provided to the processing means and / or the value of the parameter (s) depends (s), for a given generic operation, on the absolute position. basic information in the object to be processed.

Method according to one of the preceding claims, in which the generic operations comprise at least one generic position operation which makes it possible to obtain a block logic consisting of the absolute position according to one dimension of the object, as well as a generic indirection operation which makes it possible to obtain from a first block, a second block by displacement and / or scaling function d a third block or parameter.

5. Method according to one of the preceding claims wherein the generic operations comprise at least one elementary generic operation included in the group comprising: the addition of logic blocks and / or parameters, the subtraction of logic blocks and / or parameters , calculating the absolute value of the difference between logic blocks, the multiplication of logic blocks and / or parameters, the maximum among at least two logic blocks and / or parameters, the minimum of at least two logic blocks and / or parameters, the grouping and the unbundling of logical blocks, the calculation of a logical block by applying a parameter corresponding to a correspondence table to a logic block, the conditional choice of a logical block among at least two logical blocks and / or parameters, this choice being made in the following way: if a> b one chooses c, if one chooses d, with a, b, c, and d which are logical blocks and / or parameters, the h istogram of a logic block, the scaling of a logic block according to a parameter and / or a logic block, the relative displacement of a logic block according to a parameter and / or a logic block and an operation producing a block containing at least one coordinate.

6. Method according to claim 5, in which the generic operations comprise complex generic operations corresponding to groups of elementary generic operations used as such, these groups comprising in particular: calculating the median value of at least three logical blocks and / or parameters that corresponds to a group of generic operations consisting of minimum and maximum, the multiplication-accumulation of logical blocks and / or parameters, the convolution of a logical block with a parameter that corresponds to a group of generic operations consisting of multiplications and additions with several relative positions, the combined addition with a maximum and a minimum, the combined subtraction with a maximum and a minimum, the calculation of a gradient that corresponds to an absolute value of difference with two relative positions, the scalar product of a parameter consisting of a vector and a plurality of logic blocks for generating a logic block, calculating a scale change with interpolation that corresponds to a generic operation group consisting of scaling and multiplying and adding with several relative positions, the logical block combination that corresponds to a group of generic operations consisting of scaling with several relative positions ,.

7. Method according to one of the preceding claims wherein the relative positions and / or relative scales, according to at least one dimension of the object, in particular spatial and / or temporal, logic blocks have integer values or fractional values. .

Method according to one of the preceding claims, in which the elementary information is represented by fixed-point numerical values, and wherein the generic operations comprise offset operations, a saturation operation and / or at least one generic generic operation. combined with this saturation operation.

9. Method according to one of the preceding claims wherein the object to be treated is an image and the elementary information is the pixels of this image.

The method of claim 9 wherein the processing platform is part of an image capture and / or rendering apparatus, and wherein the values of the parameters are related to the characteristics of the optics and / or the sensor. and / or the imager and / or the electronics and / or the software of the image capture and / or reproduction device.

11. Method according to one of claims 1 to 8 wherein the object to be processed is a digitized sound signal and the elementary information is the sound samples of this signal.

12. Method according to one of claims 1 to 8 wherein the object to be treated is a digital mesh and the elementary information is the spatial and / or temporal information characterizing each point of the mesh.