LU100596B1 - METHOD AND DEVICE FOR LOW-LATENCY DISPLAY AND HIGH-REFRESH RATE FOR HIGH-DEFINITION MIXED REALITY DISPLAYS - Google Patents

Abstract

The subject of the invention is in particular a display image processing device (220) comprising a separate computer (230) and a logic circuit (235) connected to each other by a communication bus. computer being configured to fragmentably transmit a synthesis image to the logic circuit and to trigger the successive transmission of image fragments from an external image source to the logic circuit, the logic circuit being configured to combine a fragment of a computer image received from the computer and a fragment of an image received from the external image source to create a fragment of a mixed image, the logic circuit being further configured to transmit the created fragment of a mixed image to said image at least one calculator.

Description

METHOD AND DEVICE FOR LOW-LATENCY DISPLAY AND HIGH-REFRESH RATE FOR HIGH-DEFINITION MIXED REALITY DISPLAYS

The present invention relates to the processing and display of images and more particularly to a method and a display device with low latency and high refresh rate for high definition mixed reality displays, for example for augmented reality headphones.

Because of the many advantages they provide, especially in terms of ergonomics and immersion, the headphones used for virtual reality or augmented reality applications, also called augmented reality headsets (for augmented reality applications), immersive helmets and HMD (abbreviation of head-mounted display in English terminology), have experienced a particularly important development in recent years.

Augmented reality applications are intended to add virtual elements to real images, producing augmented images. These can result from the addition of virtual elements on a view of a real scene, such a process is known as optical see-through in English terminology, or a combination of several streams of images, typically a stream of images from one or more cameras and a stream of synthetic images, this process being known as video see-through in English terminology.

Because of the movements generally authorized by wearing a headset, it is necessary, in practice, to calculate in real time the flow of computer images according to the position of the user and the observed direction (or line of sight) . The calculation volume is all the more important as the resolution of the display screens and the refresh rate are high. There are, for example, augmented reality headsets whose display screens have a resolution of 2240 x 2160 pixels, for each eye, with a refresh rate of 90 Hz.

Moreover, the latency time between the user's movements and the display of the corresponding images (ie corrected images according to the change of pose and direction of the line of sight) must be very low for better immersion and to avoid disturbing the inner ear of the user (making the use of the headset very uncomfortable or impossible after a relatively short period of time).

To meet these computational performance requirements, some solutions use one or more graphic processing units (called GPUs, acronym for graphs processing unit in English terminology) to parallelize certain elementary tasks optimally.

FIG. 1 illustrates an example of an augmented reality system mixing a stream of images representing a real scene with a flow of synthetic images, in particular an augmented reality helmet viewed from above schematically.

As illustrated, the augmented reality headset 100 is here connected to a computer 105 via a communication cable 110 allowing the transmission of a high volume of data.

The augmented reality headset 100 here comprises two video cameras 115-1 and 115-2, one associated with the right eye and the other with the left eye, two screens 120-1 and 120-2 also associated with right eye and left eye, respectively, as well as two 125-1 and 125-2 loudspeakers located near the user's ears when wearing the augmented reality headset (the head of the user is here schematically represented with an oval shape in dashed lines). The screens 120-1 and 120-2 are for example LCD type screens (liquid crystal display acronym in English terminology) or OLED (acronym for organic light-emitting diode in English terminology) having a resolution of 2 240 x 2160 pixels.

The augmented reality headset 100 further comprises a sensor 130, for example a sensor with 6 degrees of freedom for obtaining a position (abscissa, ordinate and height) in a 3-dimensional space and an orientation (roll , pitch, yaw) in this same 3-dimensional space.

The data from the sensor 130 as well as each of the image streams from the cameras 115-1 and 115-2 are transmitted to the computer 105. These data make it possible to generate two streams of virtual images (one for each eye) linked at the user's position and the line of sight. The image streams from the cameras 115-1 and 115-2 are then combined with the virtual image streams generated and retransmitted to the augmented reality headset 100 to be displayed on the screens 120-1 and 120-2. The computer 105 may comprise one or more GPUs for combining the real-time image streams, for example by a keying technique also known as chromakey.

However, despite the power of recent GPUs, the increasing resolution of screens used in head-mounted displays increases latency. In addition, the computing power required does not generally allow the use of a miniaturized computer, which requires, because of the amount of data exchanged between the head and the computer, the use of a communication cable . The latter is particularly uncomfortable for the user.

There is therefore a need to improve the processing and display of images in augmented reality headsets and more generally for mixed reality displays. The invention solves at least one of the problems discussed above. The invention thus relates to an image processing device for display, the device comprising at least one computer and at least one logic circuit, separate and connected to each other by a communication bus, said at least one a calculator being configured to fragmentably transmit at least one synthesis image to said at least one logic circuit and to trigger the successive transmission of image fragments from at least one external image source to said at least one logic circuit, said at least one logic circuit being configured to combine at least one fragment of a synthetic image received from said at least one computer and at least one fragment of an image received from said at least one external image source to create at least one a fragment of a mixed image, said at least one logic circuit being further configured to transmit said at least one created fragment of a mixed image to said at least one calculator tor.

The device according to the invention thus makes it possible to display mixed images of high resolution with a high refresh rate and a low latency time. The invention may further be implemented with standard elements.

According to a particular embodiment, said at least one logic circuit comprises at least two distinct buffers of first-in / first-out type, one of said at least two buffers being configured to store image fragments received from said at least one a computer and the other of said at least two buffers being configured to store image fragments received from said at least one external image source.

Still according to a particular embodiment, said at least one computer is configured to receive position and / or orientation data associated with images of said at least one external image source and to generate image fragments of synthesis according to received position and / or orientation data.

Still according to a particular embodiment, said at least one computer comprises at least one microprocessor and at least one graphic coprocessor, said at least one graphic coprocessor being configured to generate fragments of synthesis images.

Still according to a particular embodiment, said at least one microprocessor is a multi-core microprocessor, at least one core being associated with a function for receiving position and / or orientation data.

Still according to a particular embodiment, the device comprises said at least one external image source, said at least one external image source comprising at least one video sensor and at least one elementary logic circuit, the elementary logic circuit being configured for transmitting, in successive fragments, at least one image from said at least one video sensor upon receipt of a predetermined event.

Still according to a particular embodiment, the device further comprises a communication interface implementing a SDI type communication protocol and allowing a data transfer between said at least one external image source and said at least one computer.

Still according to a particular embodiment, the communication bus is PCI Express type.

Still according to a particular embodiment, the device comprises at least two computers and at least two logic circuits forming at least two pairs each comprising at least one computer and at least one logic circuit, each pair being associated with at least one computer source. external images and at least one screen for performing image processing associated with said at least one external image source and auditing at least one associated screen.

Still according to a particular embodiment, said at least two logic circuits are formed in the same component of the FPGA type. The invention also relates to an image processing method for a display device, the device comprising at least one computer and at least one logic circuit, separate and connected to each other by a communication bus, said method being executed by said at least one computer and comprising the following steps: - generating a new image processing event and transmitting said event to at least one external image source to trigger transmission of fragments of image from said at least one external image source to said at least one logic circuit; following the generation of said event, generating at least one fragment of at least one synthesis image; transmitting said at least one fragment generated to said logic circuit; and - receiving said logic circuit of at least one fragment of a mixed image, said at least received fragment resulting from a combination of said at least one transmitted generated fragment and at least one image fragment transmitted from said at least one an external image source to said at least one logic circuit.

According to a particular embodiment, the method further comprises a step of receiving at least one position and / or orientation data, said at least one generated fragment of at least one synthesis image being generated as a function of said at least one position and / or orientation data received.

The method according to the invention thus makes it possible to display mixed images of high resolution with a high refresh rate and a low latency time. The invention may further be implemented with standard elements. The invention also relates to a computer program comprising instructions adapted to the implementation of each of the steps of the method described above when said program is executed on a computer and a means for storing information, removable or no, partially or completely readable by a computer or a microprocessor comprising code instructions of a computer program for the execution of each of the steps of the method described above. Other advantages, aims and features of the present invention will become apparent from the detailed description which follows, given by way of non-limiting example, with reference to the appended drawings in which: FIG. 1 illustrates an example of a reality system augmented mixing a stream of images representing a real scene to a flow of synthetic images, including an augmented reality helmet seen from above schematically; FIG. 2 schematically illustrates an exemplary architecture of a system according to the invention for processing and displaying high definition images in an augmented reality headset, with low latency and a high refresh rate; FIG. 3 illustrates certain phases of an exemplary implementation of the invention; FIG. 4 schematically illustrates a memory management mechanism used for temporarily storing generated fragments of a synthetic image and fragments of a mixed image; FIG. 5 schematically illustrates the architecture of a portion of the system illustrated in FIG. 2, making it possible to process and display high definition images in an augmented reality headset, for an eye, with low latency and a high rate refreshment; FIGS. 6a, 6b, 6c and 6d illustrate an exemplary content of buffers of a computer and of an associated programmable logic circuit during processing for the purpose of displaying a new mixed image; FIG. 7 illustrates an example of steps implemented in the computer 230-i; FIG. 8 illustrates an example of steps implemented in the elementary programmable logic circuit 505-i of the video camera 215-i; and FIG. 9 illustrates an example of steps implemented in the programmable logic circuit 235-i.

According to embodiments of the invention, the image processing and display device is based on a modular architecture making it possible to process, in a distinct or almost distinct manner, images intended for several screens, for example for two screens corresponding to each of the eyes. It is preferably based on programmable standard elements. These elements may in particular include computers, for example miniature computers incorporating multi-core microprocessors, and programmable logic circuits. As an illustration, the calculators can be in credit card format to be integrated into the augmented reality headset or be placed nearby, for example in a bag, typically a backpack, including a part of the necessary electronics the operation of augmented reality headphones and electrical sources, for example one or more batteries. The proximity of the various elements used in particular makes it possible to increase the bit rate of the data exchanged while reducing the latency.

It is observed here that the invention can be implemented for processing different types of image streams, in particular for augmented reality applications in which virtual elements are added to representations of real scenes, applications of augmented virtuality according to which real elements are added to virtual scene representations or virtual reality applications according to which virtual elements are added to virtual scene representations. These different applications combining the real world and the virtual world or different virtual worlds are subsequently called mixed applications, the corresponding images being called mixed images. The object of the invention is therefore to mix in real time a first image stream coming from an external source, for example a video sensor or a synthesis image generator, with a second stream of images generated in real time. according to characteristics of the first stream of images such as the point of view.

FIG. 2 schematically illustrates an exemplary architecture of a system according to the invention for processing and displaying high definition images in an augmented reality headset, with low latency and a high refresh rate.

The augmented reality system 200 includes an augmented reality headset 205 and a computing device 210 that may be distinct from or integrated with the augmented reality headset 205.

The augmented reality headset 205 includes in particular two video cameras 215-1 and 215-2, two screens 220-2 and 220-2 and a position and / or orientation sensor 225. It is observed that the video cameras 215- 1 and 215-2 may be distant from the augmented reality headset 205. In this case, they may be moving slaved to the augmented reality headset 205 (to aim towards a point "observed" by the user) or the displayed images may be be determined by the position and / or orientation of the video cameras 215-1 and 215-2.

Each of the video cameras 215-1 and 215-2 comprises a video sensor, for example a sensor of the CMOS type (acronym for complementary metal oxide semiconductor in English terminology) whose resolution can be, by way of illustration, 3 840 x 2160 pixels (called 4K resolution), as well as a data preprocessing circuit for transferring data from the video sensors via a communication interface, for example a communication interface implementing a communication protocol of the type SDI (acronym for serial digital interface in English terminology).

According to embodiments, the video cameras 215-1 and 215-2 are used in a mode called slave in which the camera transmits (or makes available) an image, typically element by element, for example line by line, following the receiving a predetermined event, for example receiving an event named frame_event.

The screens 220-1 and 220-2 are standard screens, typically OLED type screens for augmented reality headphones, for example screens with a resolution of 3840 x 2160 pixels and a refresh rate of at least 90 Hz. It is observed here that the images acquired being processed to be displayed, the resolution of these screens may be different from that of the video sensors used.

According to embodiments, the position and / or orientation sensor 225 is a sensor with 6 degrees of freedom for obtaining a position (abscissa, ordinate and height) in a 3-dimensional space and an orientation ( roll, pitch, yaw) in this same 3-dimensional space.

As illustrated in FIG. 2, the computing device 210 comprises in particular two computers 230-1 and 230-2 each integrating, in particular, a microprocessor or CPU (acronym for central processing unit in English terminology), for example a microprocessor. multicore. These calculators can be integrated in the same component or in separate components. The computing device 210 further comprises two programmable logic circuits 235-1 and 235-2, which can also be integrated in the same component or in separate components. By way of illustration, the programmable logic circuits 235-1 and 235-2 may be of the FPGA (field-programmable gate array in English terminology), PLD (logic programmable logic in English terminology) , EPLD (acronym of erasable programmable logic device in English terminology), CPLD (acronym of complex programmable logic device in English terminology), PAL (acronym of programmable array logic in English terminology) or PLA (programmable acronym logic array in English terminology). For purposes of illustration, it is here considered that the programmable logic circuits 235-1 and 235-2 are FPGA type. These programmable logic circuits can also be ASIC (acronym for application specific integrated circuit in English terminology), that is to say specific components preprogrammed.

As illustrated, the image streams from the video cameras 215-1 and 215-2 are transmitted to the programmable logic circuits 235-1 and 235-2, respectively, via communication buses allowing a rate sufficient to minimize latency. In embodiments according to which the refresh rate of the screens of the augmented reality headset is 90 Hz and their resolution is 3840 x 2160, the rate of this bus is preferably about 2.24 GigaBytes / s. As previously described, the communication interface may be of the SDI type.

The computers 230-1 and 230-2 and the programmable logic circuits 235-1 and 235-2 are here connected to each other by a high-speed bus, notably allowing the transfer of image streams of real scenes and streams. mixed images as well as the exchange of control data, for example a bus type PCI Express (PCI being the acronym for peripheral component interconnect in English terminology), allowing for example data rates of 16 GigaBytes / s.

The 230-1 and 230-2 computers can be connected to the 220-1 and 220-2 screens using standard video connections, such as HDMI 2.0 or Display Port 1.2 compliant cables, to transmit mixed images. and allow their display.

Figure 3 illustrates certain phases of an exemplary implementation of the invention. According to this example, the images are processed in fragments, each fragment can notably correspond to a portion of a line or a column of the processed image, a line or a column of the processed image or a plurality of lines or columns of the processed image. For purposes of illustration, it is considered here that each image is treated by line.

An event called frame_event is generated by one of the computers or, synchronously, by each of the two computers. This event, referenced 300 in FIG. 3, indicates the beginning of generation of a new computer image in the computers. According to a particular embodiment, this event is transmitted via the bus connecting the computers and the programmable logic circuits and is relayed by them to the video cameras.

It is observed here that the mechanism described with reference to FIG. 3 is reproduced at least twice in a device such as that described with reference to FIG. 2, once to process the images to be displayed in front of the left eye and a times to process the images to be displayed in front of the right eye.

The reception of the frame_event event by each video camera triggers the successive transmission of fragments of an acquired image to a buffer of the associated programmable logic circuit. The buffer memory is preferably a first-in / first-out memory or FIFO (acronym for first-in first-out in English terminology), organized in fragments. The transfer of an image from a video camera to the associated programmable logic circuit, fragment by fragment (for example line by line), is performed during the period 305 following the event 300.

Following the frame_event event, the summary image part of which must be integrated into an image of a real scene is generated, in fragments, for example by lines. Each fragment is thus generated by a computer, typically by a graphic coprocessor of the latter, one after the other, during the periods 310-1, 310-2, 310-3, etc. The frequency with which these fragments are generated is determined according to the number of fragments to be generated and the processing time of each fragment.

After a fragment has been generated, it is transmitted from the computer that generated it to the programmable logic circuit associated therewith, as illustrated with references 315-1, 315-2 and 315-3. It is for example stored in a FIFO buffer, organized by fragments (for example by lines).

A fragment generated and received by the programmable logic circuit is then processed by the latter with the corresponding fragment of the image of the real scene received from the sensor, as illustrated with references 320-1, 320-2, 320-3, etc. to create a fragment of a mixed image It is observed that the fragments used for these purposes are here stored in FIFO type memories, the computer generated image fragments are synchronized, when they are accessed. by the programmable logic circuit with those of the image of the real scene. The processing performed by the programmable logic circuit consists, for example, in the insertion of elements of a generated fragment into an image fragment of a real scene, for example according to a chromakey method.

It is observed here that the programmable logic circuits can be configured to generate a mixed image fragment with different numbers of fragments of a real image and an image of synthesis.

After obtaining, the fragment resulting from the processing (or combination) of a fragment generated with a fragment of the image of a real scene is transmitted by the programmable logic circuit to the associated computer, as illustrated with the references 325-1,325 -2 and 325-3. The image reconstituted in the computer, typically in a buffer memory thereof, is then transmitted to the associated screen to be displayed (reference 330).

It is observed here that the buffer memory used to store fragments of a synthesis image may be the same as that used to store the mixed image (resulting from the combination of a synthetic image and an image of a synthesis image. a real scene) because of fragment processing. Indeed, after being transmitted, a generated fragment of the synthesis image can be replaced by a fragment of the mixed image.

FIG. 4 illustrates such a memory management mechanism, according to which a generated fragment of the synthesis image is transmitted to the programmable logic circuit, for example of the FPGA type, to be combined with the corresponding fragment of an image of a real scene, the result of this combination, that is to say a fragment of the mixed image, then being stored in place of the previously generated generated fragment. The arrow on the representation of the image in memory indicates the order of processing of the fragments (for example lines).

FIG. 5 schematically illustrates the architecture of a part of the system illustrated in FIG. 2, making it possible to process and display high definition images in an augmented reality headset, for an eye, with a low latency and a high rate of refreshment.

It is observed here that the architecture illustrated in FIG. 5 making it possible to process images for a display on a screen can be used without being duplicated if the mixed reality system uses only one screen. Likewise, it can be replicated as many times as the mixed reality system includes screens.

The elements making it possible to process and display high definition images in augmented reality headphones, for one eye, are here in particular the video camera 215-i (the index i being here and thereafter equal to 1 or 2 depending on whether the system concerns the left eye or the right eye), the position and / or orientation sensor 225, the programmable logic circuit 235-i and the computer 230-i.

It is observed here that the video camera 215-i could be replaced by another source of image flow from which each image could be obtained on requests (for example an interruption or upon reception of an event of the frame_event type), for example a source of computer graphics or a storage device storing one or more image streams. According to embodiments, position and / or orientation information can be associated with each of these images (or groups of images) to enable the generation of a corresponding image stream in the computer 230. i.

As illustrated, the video camera 215-i comprises a high definition video sensor 500-i and an elementary programmable logic circuit 505-i, for example of the FPGA type, making it possible to transmit data from the video sensor 500-i to a remote device according to a predetermined format. As described above, the video camera 215-i is, according to embodiments, used in a slave mode in which the data from the video sensor 500-i is transmitted via a communication interface following the reception of a particular event , for example the event named frame_event.

According to the illustrated example, the data coming from the video sensor 500-i is transmitted to a first buffer memory 510-i of the programmable logic circuit 235-i, for example via a communication interface implementing a communication protocol of SDI type. . The communication interface used is a high-speed interface.

The position and / or orientation sensor 225, preferably mechanically secured to the video camera 215-i or a carrier thereof, transmits position and / or orientation data to the computer 230-i, preferably according to 6 degrees of freedom as previously described. This position and / or orientation information may, for example, enable the latter to generate computer images according to a point of view adjusted in real time.

As illustrated, in addition to the first buffer 510-i configured to receive image fragments of the video camera 215-i, the programmable logic circuit 235-i includes a second buffer memory 515-i configured to receive image fragments. synthesis generated by the computer 230-i.

Buffers 510-i and 515-i are preferably similar. This is for example FIFO-type memory whose inputs and outputs are fragments of images, for example lines of images. Therefore, the same location in these memories corresponds to the same location in the images whose fragments are stored (by considering images of the same dimensions). In a particular embodiment, each of the buffers 510-i and 515-i makes it possible to store a few image fragments (ie a few fragments of a synthesis image or of an image received from the video camera 215-i, for example between 2 and 10 lines of an image.

The programmable logic circuit 235-i further comprises a calculation module 520-i for combining image fragments belonging to at least two different images into fragments of a single image. According to particular embodiments, the calculation module 520-i is configured to allow different modes of combining fragments, the choice of mode being for example parameterized by the computer 230-i. One of these modes may be chromakey. There are other modes of combination of images, for example the fusion (or blending in English terminology). As previously described, the number of fragments obtained from the buffer 510-i and the number of fragments obtained from the buffer memory 515-i to produce one or more fragments of a mixed image may be different. These numbers are typically managed by the calculation module 520-i according to the combination mode used.

Moreover, the programmable logic circuit 235-i comprises a communication module 525-i, for example of the PCI Express type, for exchanging data with, in particular, the computer 230-i. The communication module 525-i makes it possible, in particular, to transfer data received from the computer 230-i into the buffer memory 515-i and to transfer to the computer 230-i data resulting from a processing carried out in the data module. calculation 520-i.

It is observed here that to process data relating to the two eyes of a user, data can be exchanged between the computers 230-1 and 230-2 and / or the programmable logic circuits 235-1 and 235-2, for example to take into account the common data between the images viewed by each eye or between the images used to generate them. For the sake of clarity, these connections are not shown in FIG.

Like the programmable logic circuit 235-i, the computer 230-i comprises a communication module 530-i, for example PCI Express type. It is similar or compatible with that of the programmable logic circuit 235-i to allow data exchange between these entities.

The computer 230-i further comprises at least one buffer memory 535-i for storing synthesis image fragments generated by the computer 230-i as a function of position and / or orientation data received from the sensor 225 and for storing mixed image fragments received from the programmable logic circuit 235-i. Transmission of image fragments stored in this buffer to the display screen 220-i may be performed, for example, by the output video controller 540-i. By way of illustration, the buffer memory 535-i can store image fragments such as those described with reference to FIG. 4.

The computer 230-i comprises one or more microprocessors 545-i, for example one or more multi-core microprocessors, in which are run one or more operating systems or OS (acronym for operating system in English terminology) with their 550-i kernel and 555-i drivers (commonly known as drivers) that support data transfers to other modules, allowing the execution of 560-i applications.

One or more applications are notably executed by at least one core to control the generation of computer images, at least part of the calculations necessary for the generation of these images being advantageously carried out in one or more graphic coprocessors 565-i as illustrated on FIG. 5 is the link between the microprocessor (s) 545-i and the graphic coprocessor (s) 565-i.

The computer 230-i may also comprise an input video controller 570-i for receiving image fragments, for example fragments of computer generated images generated by another source. These fragments may come in addition to or in place of those generated by the computer 230-i. They can be stored in the buffer 535-i in order to be treated in the same way as the fragments generated by the computer 230-i.

According to particular embodiments, particular processes are assigned to one or more cores of the microprocessor (s) of the computer 230-i so that these processes are not disturbed by interrupts for other functions. Such processes are, for example, related to the reception of position and / or orientation data, so that this reception is not disturbed by other processes, in particular the generation of computer-generated images, and that data recent position and / or orientation can be used.

Still according to particular embodiments, the processes for processing and displaying mixed images are initiated by the computers, that is to say by a computer for one eye and by another computer for the other eye, these processors are synchronized and running the same processes in parallel.

As described with reference to FIG. 3, when an image has to be processed and displayed, the computer 230-i generates an event of the frame_event type which is transmitted to the programmable logic circuit 235-i, the latter relaying it, for example in the form of an interruption, to the video camera 215-i, as shown with the references ® and © '.

Following this event, the computer 230-i generates fragments of a synthetic image, for example image lines that are stored in the buffer memory 535-i (reference ®). Meanwhile, the event frame_event triggers, in the video camera 215-i, the transmission of fragments of an image obtained by the video sensor 500-i to the buffer 510-i of the programmable logic circuit 235-i (reference ® ' ).

Such a transfer can advantageously be performed according to a data transfer mode known as the DMA (acronym for direct memory access), allowing a fast transfer of data. According to this transmission mode, the data is transferred directly by a memory controller to a target memory, without the intervention of a microprocessor.

When at least one fragment of the synthesis image is stored in the buffer memory 535-i of the computer 230-i, an image fragment is transmitted from this buffer memory to the buffer memory 515-i of the programmable logic circuit 235. i, as shown with the reference ©. Again, this transfer advantageously uses the DMA mode.

When image fragments are available in the buffers 510-i and 515-i, at least one image fragment of each of these buffers is transmitted to the calculation module 520-i to generate at least one fragment of mixed image (reference ®). As previously described, the number of image fragments obtained from the buffers 510-i and 515-i for generating one or more mixed image fragments depends on the processing performed. Again, the use of FIFO buffers for buffers 510-i and 515-i makes it very easy to synchronize image fragments.

The one or more mixed image fragments obtained in the calculation module 520-i are then transmitted to the buffer memory 535-i of the computer 230-i (reference)), for example at the location of the synthetic image fragment. generated, used to obtain the mixed image fragment to be stored, as described with reference to Figure 4. Again, a DMA data transfer mode is preferably used.

When all the fragments of the synthesis image generated in the computer 230-i have been processed (ie transmitted to the programmable logic circuit 235-i) and all the mixed image fragments have been received by the computer 230-i, the mixed image can be displayed (reference ©). According to a particular embodiment, the display on the screen 220-i can be managed, conventionally, by the output video controller 540-i.

As previously described, the architecture illustrated in FIG. 5 may be duplicated to process and display left eye and right eye images or may be replicated as many times as the system comprises screens.

FIGS. 6a, 6b, 6c and 6d illustrate an exemplary content of buffers of a computer and of an associated programmable logic circuit during processing for the purpose of displaying a new mixed image. By way of illustration, the buffer memory 535-i of the computer 230-i and the buffers 510-i and 515-i of the programmable logic circuit 235-i are here considered.

Figures 6a, 6b, 6c and 6d correspond to four successive instants. Thus, for example, FIG. 6a represents the contents of the buffers at a time t, FIG. 6b represents the contents of these buffers at a time t + 1, FIG. 6c represents the contents of these buffers at a time t +2 and FIG. 6d represents the contents of these buffers at a time t + 3.

For the sake of clarity, it is considered here that the images are processed by lines.

According to this example, the buffer memory 535-i can store at least one complete image (i.e. a virtual image or a mixed image) while the buffers 510-i and 515-i can store only a few lines of images.

As illustrated in FIG. 6a, after an event of the frame_event type has been generated, lines of a synthesis image are generated and stored in the buffer memory 535-i of the computer 230-i. Three lines generated for the synthetic image referenced imag_S (k), numbered 1 to 3, are here stored.

In parallel, lines of a real image, obtained from a camera and corresponding to the generated synthesis image (sharing here the same index k), are transferred to the buffer memory 510-i of the programmable logic circuit 235-i . In the illustrated example, two lines numbered 1 and 2 of the real image referenced imag_R (k) are stored in the buffer memory 510-i. At this time, no line corresponding to the synthetic image imag_S (k) has yet been transferred to the buffer memory 515-i of the programmable logic circuit 235-i. The buffer memory 515-i is therefore empty. At a next instant, as illustrated in FIG. 6b, a fourth line of the synthetic image imag_S (k) has been generated and stored in the buffer memory 535-i of the computer 230-i which then stores four lines of the synthetic image imag_S (k).

Similarly, two new lines of the real image imag_R (k) have been transferred from the video camera to the buffer 510-i which then contains four lines of the real image imag_R (k).

Moreover, the first line of the synthesis image imag_S (k) was transferred from the buffer memory 535-i of the computer 230-i to the buffer memory 515-i of the programmable logic circuit 235-i.

Since the buffers 510-i and 515-i contain equivalent lines of the real image and the synthesis image (i.e., line 1), these lines can be processed to create a line of a mixed image.

As illustrated in FIG. 6c, the created line of the mixed image, referenced imag_M (k), is stored in the buffer memory 535-i of the computer 230-i, instead of the line of the synthesis image which was used for these purposes (ie line 1).

The lines of the synthesis image and the real image used to create this line of the mixed image have disappeared from the buffers 510-i and 515-i (according to the principle of FIFO memories). In addition, new lines of the synthesis image and the real image have been transferred to the buffers 510-i and 515-i, respectively.

In this example, no new row of the synthesis image has been stored in the buffer 535-i of the computer 230-i. Again, the buffers 510-i and 515-i containing equivalent lines of the real image and the synthesis image, respectively, these lines can be processed to create one or more lines of a mixed image.

Thus, as illustrated in FIG. 6d, the second created line of the mixed image is stored in the buffer memory 535-i of the computer 230-i, in place of the line of the synthesis image that has been used in FIG. these purposes. The lines of the synthesis image and the actual image used to create this line of the mixed image disappear from the buffers 510-i and 515-i and new lines of the synthesis image and the image actual were transferred to buffers 510-i and 515-i, respectively.

In this example, a new line of the synthesis image has been stored in the buffer memory 535-i of the computer 230-i.

The process continues until all the lines of the mixed image have been created.

FIG. 7 illustrates an example of steps implemented in the computer 230-i.

As illustrated, a first step (step 700) is intended to initialize the variables r and n to the value of the variable k which represents the index of the last generated synthesis image. The variable n represents an index of synthesis images whose fragments must be transmitted to the programmable logic circuit and the variable r represents a composite image index of which fragments must be stored in the computer.

When an event of type frame_event is detected (step 705), the variable I is initialized to the value one and the variable k is incremented by one (step 710). The variable I represents an index of fragments in the synthesis image that is generated.

The fragment / of the synthetic image imag_S (k) is then generated (step 715) then stored in the buffer memory 535-i of the computer 230-i (step 720). In a next step, a test is performed to determine if the index / has reached its maximum value (maxL), that is to say to determine if the index / corresponds to the last fragment of the computer image that is generated (step 725).

If the index / has not reached its maximum value (maxL), the index / is incremented by one (step 730) and steps 715 and 720 are repeated to generate and store a new synthesis image fragment.

In the opposite case, if the index / has reached its maximum value (maxL), a test is performed to determine whether the buffer 535-i of the computer 230-i contains a free space (step 735) for generating and storing new fragments of computer-generated images.

In parallel, when at least one fragment of a synthesis image has been generated and stored (steps 715 and 720), the variable m is initialized to the value one and the variable n is incremented by one (step 740). The variable m represents a fragment index in the generated synthesis image that is transmitted to the programmable logic circuit 235-i.

In a next step, the fragment m of the image imag_S (n) is transmitted to the programmable logic circuit 235-i (step 745) to be stored in the buffer memory 515-i.

A test is then performed to determine if the index ma reaches its maximum value (maxL), that is to say to determine if the index m corresponds to the last fragment of the summary image to be transmitted (step 750) .

If the index m has not reached its maximum value (maxL), the index m is incremented by one (step 755) and step 745 is repeated to transmit the next fragment.

If, on the contrary, the index m has reached its maximum value (maxL), a test is performed to determine whether there are fragments of a next synthesis image to be transmitted (step 760). If there are fragments of a next synthetic image to be transmitted, steps 740 to 755 are repeated.

At the same time, if at least one fragment of a synthesis image has been transferred to the buffer memory 515-i of the logic circuit 235-i, the variable q is initialized to the value one and the variable r is incremented by one (step 765). The variable q represents a fragment index in a mixed image to be received from the programmable logic circuit 235-i.

In a next step, when the fragment q of the mixed image imag_M (r) is received from the programmable logic circuit 235-i, it is stored in the buffer memory 535-i of the computer 230-i at the location defined by the q and r variables (step 770).

A test is then performed to determine if the index q has reached its maximum value (maxL), that is to say to determine if the index q corresponds to the last fragment of the mixed image to receive (step 775).

If the index q has not reached its maximum value (maxL), the index q is incremented by one (step 780) and step 770 is repeated to receive the next fragment.

If, on the contrary, the index qa reaches its maximum value (maxL), an event is generated to display the received mixed image (step 785) and a test is performed to determine if there are fragments of a mixed image next to receive (step 790). If there are fragments of a next mixed image to receive, steps 765 to 785 are repeated.

FIG. 8 illustrates an example of steps implemented in the elementary programmable logic circuit 505-i of the video camera 215-i.

When a frame_event event is detected (step 800), the fragments of the last image acquired by the video sensor 500-1, representing a real scene, are successively transmitted to the programmable logic circuit 235-i to be stored in the buffer 510-i (step 805).

FIG. 9 illustrates an example of steps implemented in the programmable logic circuit 235-i.

As illustrated, as soon as it is determined that a fragment of a synthesis image is stored in the buffer memory 515-i and that a fragment of a real image is stored in the buffer memory 510-i (step 900 ), at least one fragment of each of these images, corresponding to each other, is used by a calculation module to combine them and create a fragment of a mixed image (step 905). This mixed image fragment is then transmitted to the computer 230-i as previously described (step 910).

Naturally, to meet specific needs, a person skilled in the field of the invention may apply modifications in the foregoing description. The present invention is not limited to the described embodiments, other variations and combinations of features are possible.

In particular, while the invention has been described with reference to the use of a computer and an associated programmable logic circuit, other equivalent configurations can be implemented.

The present invention has been described and illustrated in the present detailed description with reference to the accompanying figures. However, the present invention is not limited to the embodiments presented. Other variants and embodiments can be deduced and implemented by the person skilled in the field of the invention upon reading this description and the accompanying figures.

In the claims, the term "include" does not exclude other elements or other steps. The indefinite article "one" does not exclude the plural. A single processor or several other units may be used to implement the invention. The various features presented and / or claimed can be advantageously combined. Their presence in the description or in different dependent claims does not exclude the possibility of combining them. The reference signs can not be understood as limiting the scope of the invention.

Claims (12)

An image processing device for a display (220), the device comprising at least one computer (230) and at least one logic circuit (235), separate and connected to each other by a communication bus, said at least one computer being configured to fragmentably transmit at least one synthesis image to said at least one logic circuit and to trigger successive transmission of image fragments from at least one external image source to said at least one logic circuit, said at least one logic circuit being configured to combine at least one fragment of a synthetic image received from said at least one computer and at least one fragment of an image received from said at least one external image source for creating at least one fragment of a mixed image, said at least one logic circuit being further configured to transmit said at least one created fragment of a mixed image to said at least one computer. 2. Device according to claim 1 wherein said at least one logic circuit comprises at least two distinct buffers of first-in / first-out type, one of said at least two buffers being configured to store image fragments received from said at least one computer and the other of said at least two buffers being configured to store image fragments received from said at least one external image source. Apparatus according to claim 1 or claim 2 wherein said at least one computer is configured to receive position and / or orientation data associated with images of said at least one external image source and to generate synthetic image fragments according to received position and / or orientation data. 4. Device according to any one of claims 1 to 3 wherein said at least one computer comprises at least one microprocessor and at least one graphics coprocessor, said at least one graphics coprocessor being configured to generate synthesis image fragments. 5. Device according to claim 4 dependent on claim 3 wherein said at least one microprocessor is a multi-core microprocessor, at least one heart being associated with a function of receiving position data and / or orientation. 6. Device according to any one of claims 1 to 5 comprising said at least one external image source, said at least one external image source comprising at least one video sensor (500) and at least one elementary logic circuit ( 505), the elementary logic circuit being configured to transmit, in successive fragments, at least one image from said at least one video sensor upon receipt of a predetermined event. 7. Device according to claim 6 further comprising a communication interface implementing a SDI type communication protocol and allowing a data transfer between said at least one external image source and said at least one computer. 8. Device according to any one of claims 1 to 7 according to which the communication bus is PCI Express type. 9. Device according to any one of claims 1 to 8 comprising at least two computers (230-1, 230-2) and at least two logic circuits (235-1, 235-2), forming at least two pairs each comprising at least one computer and at least one logic circuit, each pair being associated with at least one external image source and at least one screen for performing image processing associated with said at least one external image source and auditing at least one associated screen. 10. Device according to claim 9 wherein said at least two logic circuits are formed in the same component type FPGA. 11. An image processing method for a display device (220), the device comprising at least one computer (230) and at least one logic circuit (235), separate and connected to each other by a bus communication method, said method being executed by said at least one computer and comprising the following steps: generating a new image processing event and transmitting said event to at least one external image source to trigger the transmission of image fragments from said at least one external image source to said at least one logic circuit; following the generation of said event, generating at least one fragment of at least one synthesis image; transmitting said at least one fragment generated to said logic circuit; and - receiving said logic circuit of at least one fragment of a mixed image, said at least received fragment resulting from a combination of said at least one transmitted generated fragment and at least one image fragment transmitted from said at least one an external image source to said at least one logic circuit. The method of claim 11, the method further comprising a step of receiving at least one position and / or orientation data, said at least one generated fragment of at least one synthesis image being generated based on of said at least one position and / or orientation data received.