FR2861938A1 - Video image streams and graphic objects combining process for automobile, involves combining values of image planes pixels and determined transparency ratio to determine value of each pixel of resulting video image streams - Google Patents

Abstract

The process involves determining a value of each of several image planes according to position and size of each clip art or video image streams. A transparency ratio for each pixel is determined based on a specific transparency ratio. The values of the image planes pixels and the determined transparency ratio are combined by a video/graphics accelerator (12) for determining value of each pixel of resulting video image streams. An independent claim is also included for a device to combine resulting video image stream, input video image stream and a clip art.

Description

The present invention relates to the field of processing and display

  video signals, in particular for combining video images and fixed or animated graphic objects.

  It applies in particular, but not exclusively, to the field of on-board display systems, in particular adapted to the automobile.

  In the automotive field in particular, there is a growing need for integration of multimedia applications, in order to improve the safety, comfort and information of the driver, and to provide entertainment for the passengers. Thus, several image sources (video and graphic objects) can be coexisted. As sources of images, there can be mentioned a retro-vision camera, a DVD player, a game console, a navigation system. For reasons of cost, space and power consumption, it seeks in return to reduce the number of displays required.

  But to combine in real time video images and graphic objects, it is necessary to implement powerful computing means. It turns out that such means are totally unsuited to the world of the automobile. Indeed, these means generally have a cost too high, too much space and power consumption. They require a very stable environment in terms of temperature and vibration, which is unfeasible on board an automobile. They must also have a very high reliability and availability comparable to those of radio receivers designed for the automobile, which is far from the case of current computers.

  The present invention aims to eliminate these disadvantages. This objective is achieved by means of a method for combining in real time, in a single resulting video image stream, at least one input video image stream and at least one graphic object, said method comprising steps of defining a plurality of image planes to be superimposed in a resulting video image stream, and having dimensions equal to that of the images of the resulting video image stream, respectively assigning each graphic object and video image stream to inputting one of the image planes, and a display position in the image plane, determining the value of each pixel of each image plane, according to the position and size of each graphic object or a video image stream assigned to the image plane, determining for each pixel of each image plane a resultant transparency coefficient as a function of a specific transparency coefficient and respective transparency coefficients of corresponding pixels. , having the same position in possible upper image planes, so that each pixel of the image plane is seen through the corresponding pixels of the possible upper image planes taking into account the respective transparency coefficients of the latter and determining the value of each pixel of the resulting video image stream by a linear combination of the pixel values of the corresponding image planes having the same position, and respective resultant transparency coefficients calculated for each of the planar pixels. corresponding image.

  Advantageously, the method further comprises steps of assigning a global coefficient of transparency to each graphic object and / or each video image stream, and determining a new coefficient of transparency for each pixel by multiplying the coefficient of transparency of its own. of the pixel by the overall transparency coefficient assigned to the object.

  According to the method of the invention, each pixel value comprises a color component for each primary color, red, green, blue, the determination of each pixel of applying the linear combination to each color component.

  According to a particular feature of the method, it further comprises a step of reducing the size of the images of an incoming video image stream and inserting the reduced image size video image stream into a picture plan.

  According to another characteristic of said method, a graphic object inserted in an image plane is an animated graphic object comprising several images which are successively inserted in the image plane and at the position assigned to the graphic object, to each image of the image plane. resulting video image stream.

  The invention also proposes a device for combining in real time into a single stream of resulting video images, at least one input video image stream and at least one graphic object, comprising a processing unit comprising a processor and a processor. a graphics / video accelerator, the latter comprising: É wired logic circuits for inserting graphic objects and / or digitized video image streams in separate image planes, each image plane comprising one pixel per pixel image of the resulting video stream, associated with a coefficient of transparency, É of the wired logic circuits for determining for each pixel of each image plane a resultant transparency coefficient, as a function of a coefficient of transparency and coefficients respective pixels, having the same position in any upper image planes, so that each pixel of the image plane is seen through the corresponding pixels of any higher image planes taking into account the respective transparency coefficients of the latter, and wired logic circuits for determining the value of each pixel of the resulting video image stream, by a linear combination of the pixel values of the corresponding image planes having the same position, and respective resultant transparency coefficients calculated for each of the corresponding image plane pixels.

  The device is also designed such that the accelerator includes wired logic circuits for applying an overall transparency coefficient to a graphic object inserted in an image plane.

  According to a remarkable characteristic of the device according to the invention, the processor and the accelerator communicate with each other by means of an exchange structure comprising a data block for each of the graphic objects and / or video image stream. an input capable of being inserted into an image plane, each data block defining the image plane where the graphic object is to be inserted, the position and the dimensions of the graphic object in the image plane, an overall transparency coefficient to be applied to the graphic object and indicators indicating whether the graphic object should be erased or updated in the image plane.

  According to another characteristic of the device according to the invention, the accelerator comprises wired logic circuits for converting a stream of scanned video images YUV into a digitized video stream RGB.

  Advantageously, the processing unit of the device according to the invention is implemented using a programmable gate array (Field Programmable Gate Array or FPGA).

  According to another remarkable feature of the device according to the invention, it further comprises at least one analog / digital converter for converting an analog video stream into a digital video stream which is input to the processing unit and a digital converter. / analog to convert the resulting video image stream into an analog video stream that can be displayed using a display screen.

  A preferred embodiment of the invention will be described hereinafter, by way of non-limiting example, with reference to the accompanying drawings in which: Figure 1 shows schematically an image processing device according to the invention; Figure 2 shows in more detail the processing unit of the image processing device shown in Figure 1; Figure 3 shows in greater detail the main functions performed by the video accelerator of the image processing device shown in Figure 2; Figure 4 shows the exchange structure between the processor and the video accelerator; Figure 5 shows the organization of the RAM used by the processing unit.

  FIG. 1 represents a device for combining in the same image several video sources and graphic objects. This device comprises a processing unit 1 connected to memories, such as a non-volatile memory 3 for example of the Flash memory type and a random access memory 4, for example of the SDRAM type. The processing unit is also connected to one or more digital video sources. In order to be able to process analog video sources, the device also comprises one or more analog / digital converters 5. The video signal that is generated by the processing unit is applied to the input of a digital-to-analog converter 6 to then be able to be applied to the input of a display screen 7.

  The processing unit 1 advantageously consists of a processor 11 coupled to a graphics / video accelerator 12, consisting of hardwired logic circuits capable of carrying out several processes in parallel.

  The non-volatile memory 3 stores the program executed by the processor 12, and various graphic objects that can be displayed.

  The RAM 4 is used by the processing unit 1 as a working memory for executing a program, and storing the graphic objects to be displayed, as well as various image planes to be displayed.

  The video signals applied at the input of the processing unit 1 are preferably signals in the YUV (for example YUV422) or RGB format, in which each image pixel is represented by a word containing the three primary color components, red, green and blue of the pixel, and a coefficient of transparency, these four pieces of information being coded for example on 8 bits respectively. The YUV format collects for each pixel luminance information (Y), and chrominance and saturation information (U, V), and can be converted to RGB format using known conversion formulas.

  FIG. 2 shows the main functions provided by the graphics / video accelerator 12. In this figure, the graphics / video accelerator 12 comprises: a graphic object management module 13 designed to communicate with the processor 11 via a structure exchange 22, and to coordinate the processing performed by the other functions of the accelerator.

  É a clock module 14 which synchronizes all the functions of the accelerator, É a unit DMA (Direct Memory Access) 18 which allows direct access to the memory without intervention of the processor to perform block transfers of data from one memory address to another, É a set of line registers 19 in which are stored rows of image pixels to be displayed, and É a fusion module 20 designed to combine the pixels stored in the different registers 19 to get the pixels of an image line to display.

  All the treatments performed by the accelerator 12 use the same memory and are performed in parallel. The management module 13 therefore has the role of responding to requests for access to memory, issued by the various processes, while managing access priorities.

  The DMA unit 18 is activated by the management module 13 to transfer video streams or objects stored in the non-volatile memory 3 to the RAM 4, and to fill the registers 19 from the latter.

  The purpose of the clock module 14 is to provide the other functions of the accelerator and the display screen 7 with all the necessary synchronization signals, these signals being extracted from the video signals applied at the input of the processing unit. It sequences the other treatments of the accelerator 12 so that the registers 19 are refreshed at the rate of the synchronization signals applied to the screen 7. The clock module comprises an RGB signal generator 15 and a YUV signal generator 16, designed to clock the analog-to-digital converter 5 according to the format of the input video signal, and output line synchronization signals to indicate the beginning and the end of each image line.

  In addition, the YUV generator 16 converts the inputted YUV video stream into an RGB video stream.

  The clock module 14 may furthermore comprise a pattern generator 17 designed to send a pattern to the display screen 7 when the processor 11 is stopped or when the memory 4 does not contain any image to be displayed.

  The set of line registers 19 comprises several line registers each intended to receive an image line to be displayed, each line register corresponding to an image plane. As shown in FIG. 3, it is thus possible to provide an image background line register 31, an image foreground line register 33 and an intermediate image plane line register 32. Other line registers 34, 35 may be provided to directly receive a video signal image line input to the processing unit, for example a video image line register 34 from the RGB signal input. of the processing unit and a video image line register from the RGB converter 21 provided in the YUV generator 16.

  It is also possible to provide registers 36, 37 for receiving an image line respectively from the RGB and YUV video sources, and having undergone prior reduction processing so that the video image is displayed only in part of the image. display screen, for example on a quarter of the screen (incrustation of a video image animated in another). In a variant, these images of reduced size can be registered directly in an image plane area provided in the random access memory 4.

  The merge module 20 is designed to merge the image lines stored in several of the registers of the set 19 in order to obtain a single image line which is applied at the output of the processing unit 1 to the converter digital / analog 6.

  As represented in FIG. 3, the fusion module comprises as many multiplexers 42, 43, 44 as there are image planes, these multiplexers being controlled by the module 13 and making it possible to select an image line per image plane. to merge among the different registers of the set 19, and to apply the selected image lines for the merger at the input of the merge processing 41 itself. Thus, in the example of FIG. 3, the multiplexer 42 makes it possible to select the line to be displayed in the background among the background line registers 31, the video image line registers 34, 35, and the registers lines 36, 37 of reduced video image. The multiplexer 43 makes it possible to select the line to be displayed in the intermediate plane among the intermediate plane line registers 32, and the reduced video image line registers 36, 37. The multiplexer 44 makes it possible to select the line to be displayed in the foreground among the line registers 33 of the foreground, and the video line registers 36, 37 reduced.

  The reduced video images can thus be displayed in any of the three image planes, while the unreduced video images can only be displayed in the background of the image.

  To merge several image lines into one, the merge processing 41 exploits the transparency coefficient associated with each image pixel, which makes it possible to obtain highly evolved transparency effects. In order to be able to carry out real-time fusion processing, the fusion module has a "pipeline" architecture capable of performing several operations simultaneously.

  The processing carried out by the fusing module 20 is designed on the one hand, so that a pixel of the background image is seen with its own coefficient of transparency and through the corresponding pixels of the intermediate image plane and foreground, taking into account the respective coefficients of transparency of the latter, and secondly for a pixel of the intermediate plane to be seen with its own coefficient of transparency through the corresponding pixel of the foreground. plan taking into account the coefficient of transparency of the plan. For this purpose, the following formulas (for three image planes) are applied: X2 = a2x2 (1) X, = (1 a2) a, x, (2) X0 = (1 a2) (1 a1) aoxo ( 3) in which: the indices 0, 1 and 2 refer respectively to the background, the intermediate plane and the foreground of the image, x; (i = 0, 1 or 2) represents the value of a component R, G or B of a pixel of the image plane i to be displayed,, a; (i = 0, 1 or 2) represents the eigenvector coefficient of the pixel x ;, and X; (i = 0, 1 or 2) corresponds to the value of the image pixel calculated for the image plane i.

  In these formulas, each transparency coefficient is between 0 and 1, and is 0 for full transparency and 1 for full opacity. The X value of a pixel to be displayed of the resulting image being obtained by summing the X values; obtained for each of the planes: X = Xo + X, + X2 (4) Providing to apply a transparency coefficient to the objects displayed in the background makes it possible to display a background color, which allows to achieve transitions effects.

  It should be noted that the number of superimposed planes that can be managed by the processing unit is fixed because it is defined by the architecture of the accelerator 12 and in particular by the choice of the number of line registers. and the architecture of the fusion module 20. To have fewer or more superimposed image planes, it is therefore sufficient to adapt the number of line registers 19 and the fusion module accordingly.

  The processor 11 and the accelerator 12 communicate with each other via an exchange structure 22 comprising several data blocks 50, one block per graphic or video object likely to be simultaneously displayed by the accelerator 12. This exchange structure is shown in FIG. 4. In this figure, each block 50 comprises: E a memory address field 51 defining the first location in the memory, where the data pixels of the memory are located. the object, for example on 32 bits, É a field size 52 of the object, intended to receive two numbers, for example on 16 bits, defining the number of pixels in width and height of the object, É a field position 53 of the object in the image, intended to receive two numbers, for example on 16 bits, defining the coordinates of a pixel of the display screen from which the object is to be displayed, for example to from its upper left corner, and É an ensem display parameters 54 of the object, ie a global transparency coefficient to be applied to the object, a refresh indicator indicating whether or not the object must be updated in the images sent to the screen display, an erase flag indicating whether or not the object should be erased from the images sent to the display screen, and an image plane number where the object is to be displayed.

  It can be provided that the set of parameters 54 further comprises a field intended to receive a number of images to be displayed for the object, in order to produce an animation effect of the object. If this number indicates that multiple images are to be displayed, the accelerator 12 changes the image of the object to each frame of image to be displayed, and loops around the number of images of the object. In this case, all the images of the object are located one after the other in the random access memory 4, starting from the address defined by the address field 51, each image of the object having the size defined by the field size 52.

  In the exchange structure 22, it is thus possible to provide for example 16 blocks of data, 14 of which are used to display graphic objects and 2 video objects. In the blocks 50 of exchange data relating to the video objects, the address field 51 is not used and is replaced by a field defining the video source to use if the device is designed to handle several video sources, and a field indicating whether the video image should be displayed in full screen or should be reduced in size to be embedded in another image.

  The processing unit 1 is advantageously carried out using an FPGA 5 (Field Programmable Gate Array), for example the CYCLONETM component of the ALTERA company.

  In FIG. 5, the random access memory 4 may for example be used in such a way as to include: E a block 61 of pixel data of graphical objects to be displayed, E blocks 62 of pixel data of reduced video images to be embedded in the an image displayed on the screen, comprising for example a block for the RGB source and a block for the source YUV, É a block 63, 64, 65 of pixel data for each of the image planes to be displayed and intended to receive the value of all the pixels of the image plane to be displayed, and É a memory zone 66 intended to receive the program to be executed by the processor 11 and usable for storing data of intermediate processing.

  The processor 11 is programmed to provide a set of predefined functions accessible through what is called an API 20 (Application Program Interface).

  The API thus combines administration functions for initializing and configuring the processing unit 1, reserving memory space and obtaining information on the hardware configuration of the accelerator, as well as functions managing the display of graphic or video objects.

  Among the administration functions, there is a configuration function of the background color receiving as parameter the respective values of each RGB component of the desired background color.

  The object management functions include functions for creating and deleting a graphic object, the creation function receiving as parameter the size of the graphic object to be created, and being designed to reserve and assign the object to displaying a memory space in the area 61 of the graphic objects to be displayed, as well as a block in the exchange structure 22, and for returning an identifier of the object, the size of the object to be created being written in the field 51 of the block 50 assigned to the object in the exchange structure.

  A function of loading an object in the memory zone 61 makes it possible to copy the pixel data of an object to the memory zone assigned to the object, this function receiving in parameter an identifier of the object and an address where finds the object to be copied to memory 4. If the image of the object to be copied is larger than the memory space allocated to the object, only the upper-left corner of the image is copied.

  The object management functions also include functions receiving as parameter an object identifier and making it possible to update the data stored in the block 50 assigned to the object in the exchange structure 22, namely: the refresh or erase flag in the field 54 to trigger the display or the erasure of the object, the coordinates in the image of the upper left corner of the object to be displayed, stored in the field 53 , É the overall transparency coefficient of the object, stored in the field 54, É the display plane number of the object, stored in the field 54, and 15 É the number of images of a graphic object animated, stored in field 54.

  The object management functions further include functions for managing video objects, including a function for creating a video object by specifying its dimensions, and a function for assigning a video source to a previously created video object.

  Periodically, at the display frequency of the images defined by the clock module 14, the management module 13 of the accelerator scans the exchange structure 22, to determine whether objects are to be displayed or erased, thanks to the indicators of refresh and erase provided in the field 54 of each block 50.

  If the refresh flag of a block 50 signals that the corresponding object needs to be refreshed, the management module 13 triggers the copying of the corresponding graphic object, taking into account the information provided in the data block, and particular: E the address of the pixel data of the object, provided by the field 51, to determine where are the pixel data of the object to be copied, É the plane number provided by the field 54, to determine to which image plane 63, 64, 65 the object must be copied, É the coefficient of transparency provided by the field 54, to determine which global transparency coefficient to apply to the object, the position and the dimensions of the object provided by the fields 52, 53, to determine the location in the image plane where the pixel data of the object is to be copied, and E the number of frames in the sequence provided by the field 54 if the object is animated.

  To apply an overall transparency coefficient, the accelerator calculates for each pixel of the object a new transparency coefficient by multiplying the transparency coefficient of the pixel by the overall transparency coefficient. For this purpose, the accelerator uses an internal register to temporarily store the value of the pixel and its transparency coefficient.

  If the erase flag indicates that an object must be erased, the management module 13 controls the erasure of the memory area occupied by the object, defined by the field 54 to determine in which image plane 63, 64, 65 is the object, and the fields 52 and 53 to determine what are the pixels to be erased in the image plane. Erasing the object consists in setting to 0 the transparency coefficient of each pixel of the object in the image plane, so as to make it completely transparent.

  In parallel with these processes, the fusion module 20 calculates, at the refresh rate of each line of the image, the value of each pixel X of the line to be displayed as a function of the respective values of the pixels of the corresponding lines in each image plane. 63, 64, 65.

  In order to generate reduced incrusted images, the accelerator processes the video stream to be embedded by inserting the pixels of an image line to be displayed with their respective transparency coefficients, in one of the registers 36, 37, these pixels coming from the memory zone 62 provided for this purpose in the RAM 4 which is read asynchronously by the display process.

  It should be noted that the memory zone 62 can be deleted, the images to be displayed being able to be written directly into the memory plane 63, 64, 65 chosen.

Claims (12)

CLAIMS,   A method for combining in real time, in a single resulting video image stream, at least one input video image stream and at least one graphic object, the video images and graphic objects being pixels, characterized in that the method comprises the steps of: E defining a plurality of image planes to be superimposed in a resulting video image stream, and having dimensions equal to that of the images of the resulting video image stream, E assign respectively to each graphic object and input video image stream one of the image planes, and a display position in the image plane, E determining the value of each pixel of each image plane, according to the position and the size of each graphic object or video image flow attributed to the image plane, É determine for each pixel of each image plane a coefficient of transparency resulting as a function of a coefficient of transparency e t corresponding respective transparency coefficients of corresponding pixels, having the same position in any upper image planes, so that each pixel of the image plane is seen through the corresponding pixels of any upper image planes taking into account counting the respective transparency coefficients of the latter, and É determining the value of each pixel of the resulting video image stream, by a linear combination of the pixel values of the corresponding image planes having the same position, and transparency coefficients respective resultants calculated for each of the corresponding image plane pixels.   2. Method according to claim 1, characterized in that it further comprises the steps of assigning a global transparency coefficient to each graphic object, and determining a new coefficient of eigenvice for each graphical object pixel by multiplying the own transparency coefficient of the graphic object pixel with the overall transparency coefficient assigned to the object.   3. Method according to claim 1 or 2, characterized in that it further comprises the steps of: assigning a global transparency coefficient to a stream of video images, and determining for each pixel of the image plane allocated to a resultant transparency coefficient, as a function of the overall transparency coefficient assigned to the video stream and the respective transparency coefficients of corresponding pixels having the same position in any higher image planes, so that each pixel of the image plane is seen through the corresponding pixels of the possible upper image planes taking into account the respective transparency coefficients of the latter.   4. Method according to one of claims 1 to 3, characterized in that each pixel value comprises a color component for each primary color, red, green, blue, the determination of each pixel of applying the linear combination to each component color.   5. Method according to one of claims 1 to 4, characterized in that it further comprises a step of reducing the size of the images of an incoming video image stream and the insertion of the video image stream reduced image size in an image plane.   6. Method according to one of claims 1 to 5, characterized in that a graphic object inserted in an image plane is an animated graphic object comprising a plurality of images which are inserted successively in the image plane and at the position assigned to the graphic object, to each image of the resulting video image stream.   A device for combining in real time into a single resulting video image stream, at least one input video image stream and at least one graphic object, the video images and graphic objects being pixels, characterized in that it comprises a processing unit (1) comprising a processor (11) and a graphics / video accelerator (12), the graphics / video accelerator comprising: É hardwired logic circuits (18) for inserting in planes d distinct images of the graphic objects and / or digitized video image streams, each image plane comprising an image pixel per image pixel of the resulting video stream, associated with a transparency coefficient, É of the wired logic circuits (41) for determining for each pixel of each image plane a resultant transparency coefficient, as a function of a proper transparency coefficient and the respective transparency coefficients of corresponding pixels, having the same position in possible upper image planes, so that each pixel of the image plane is seen through the corresponding pixels of the possible upper image planes taking into account the respective transparency coefficients of the latter, and É circuits wired logic (41) for determining the value of each pixel of the resulting video image stream by a linear combination of the pixel values of the corresponding image planes having the same position, and respective resultant transparency coefficients calculated for each of the corresponding image plane pixels.   8. Device according to claim 7, characterized in that the accelerator (12) comprises wired logic circuits for applying a global transparency coefficient to a graphic object inserted in an image plane.   9. Device according to claim 7 or 8, characterized in that the processor (11) and the accelerator (12) communicate with each other by means of an exchange structure (22) comprising a block of data for each graphic objects and / or input video image stream capable of being inserted into an image plane, each data block defining the image plane where the graphic object is to be inserted, the position and the dimensions from the graphic object in the image plane, a global transparency coefficient to be applied to the graphic object and indicators indicating whether the graphic object should be erased or updated in the image plane.   10. Device according to one of claims 7 to 9, characterized in that the accelerator (12) comprises wired logic circuits for converting a digitized video image stream YUV into a digitized RGB video stream.   11. Device according to one of claims 7 to 10, characterized in that the processing unit (1) is made using an FPGA.   12. Device according to one of claims 7 to 11, characterized in that it comprises at least one analog / digital converter (5) for converting an analog video stream into a digital video stream that is applied to the input of the unit for processing (1), and a digital-to-analog converter (6) for converting the resulting video image stream into an analog video stream that can be displayed using a display screen (7).