WO2012130208A1 - Video arrangement for providing hdr video streams - Google Patents

Abstract

The invention relates to a video arrangement for providing high-dynamic-range video streams, based on progressively recorded frames. In this context, a means for automatically providing a high-dynamic-range frame stream comprising frame exposure series elements, recorded at a frame exposure series element rate which at least approximately corresponds to the frame rate, in a manner that spans frame exposure series and a means for synthesizing a motion blur signature are provided.

Description

 VIDEO ARRANGEMENT FOR PROVIDING HDR VIDEO STREAMS

description

The present invention relates to the oberbegrifflich Bean ^¬ spruchte and thus relates to video streams high Dyna ^¬ mik.

Digital image and video technology has made significant progress in recent years. First, the spatial resolution of the images has been increased, that is, the number of pixels has been increased. In principle, the resolution achievable today would be sufficient for many purposes, if appropriate also for cinema screens to be displayed on large screens. In the area of video streams, for example, high-definition television technology, or HDTV for short, has already become established in the consumer sector. However, if it is required that in a scene both very bright areas and very dark areas should be displayed clearly recognizable, ie a high (brightness) dynamic is required, the conventional technology still reaches its limits. In the present case, the term "dynamics" refers to differences in brightness or light intensity, not particularly short movements.

In order to be able to provide images of high dynamics, photography has established the technology of single image bracketing (the so-called "bracketing"), in which one and the same view is briefly followed by different ones

- 1 - CONFIRMATION COPY Taken exposures. This can, for example, by Varia ^¬ tion exposure durations occur with about one and the same ^¬ view with a 1/60 second exposure, a 1/125 second and 1/250 second exposure duration of exposure is taken. Thus, one obtains an exposure series of individual images which, although conventionally viewed as overexposed, normally exposed and underexposed, are those in which, however, the dark areas, the "normal" areas and the particularly bright areas are particularly well-finished with particularly high quality, ie in particular unaffected by sensor linearity, sensor noise, and / or sensor saturation.

There are a variety of still photography image processing programs that combine the individual elements of a single frame of exposure into a single, high-dynamic view. These programs also provide the so-called tone-mapping, which is required to render the resulting view with high dynamics on a monitor or in print on paper. This is merely mentioned in order to state that in the present case it is primarily not the tonemapping that matters, but the question of how frame streams for highly dynamic video per se can be provided in a suitable manner at all.

It has already been suggested to play those composed of Einzelbildbe ^¬ clearing rows photographs in time-lapse photography as a movie. Examples of such time-lapse recordings can be found, for example, on YOÜ TUBE.

In addition, it has already been proposed to simultaneously take a picture with several cameras or to do it with a camera. jektiv captured image via a suitable Strahlteileranord ^¬ tion simultaneously multiple sensors one and the same camera to feed, for example, by Davorsetzen different neutral density filters or the like different sensor sensitivities are realized. The individual images simultaneously recorded with such a multi-sensor system can be combined in a suitable manner to a frame ^'high dynamics, in turn, it can be a Tone mapping are made, etc. is particularly problematic in the already sold by the applicant Multisensorlö- sung, however, that the images the respective sensors must be made pixel-precise to coincide. This requires an extremely high adjustment effort, which makes the arrangement significantly more expensive. Thus, the multi-sensor arrangement, assuming sufficiently high data storage rates, while real-time capable, but suffers from high costs, while the time-lapse method is completely unsuitable for the recording of movies, for example. These problems lead to a lot of the movies are still shot on analogue film material, ge ^¬ if necessary, even if a digital imaging requires a digitizing scanning the analog film material for intermediate steps.

It is desirable to be able to provide video streams of high dynamics without the problems mentioned above having a serious effect. Reference is otherwise made to the U S 2005/0275747 AI, DE 10 2009 029 868 B4, WO 2010/082692 AI and the

US 2006/0133688 Al. The object of the present invention is to provide new products for commercial use. The solution to this problem is claimed in an independent form. Preferred embodiments can be found in the dependent claims.

The present invention thus proposes, in a first aspect, a video arrangement for providing high-dynamic-time video streams based on successively acquired individual images, wherein a means for automatically providing a high-dynamics framestrora comprises individual images recorded at at least approximately the frame rate of a corresponding single-frame exposure element rate. bildbelichtungsreihenelementen is provided in Einzelbildbelichtungsrei- ^' henübergreifender way.

Accordingly, one basic idea of the invention is that the determination of a frame stream is based on consecutive, ie successively recorded, individual images, with a plurality of individual images always forming an exposure series. The frames are then provided with reference to these frames by not looking en bloc one frame exposure row, then independently contemplating a single frame exposure row en bloc, then looking at the next frame exposure row en bloc, etc., but instead overlapping the single frame exposure rows is working. Thus, significant improvements can be achieved because the invention makes it possible, despite the generation of a highly dynamic frame current, to lower the element rate to the extent that it corresponds approximately to the desired frame rate. It should be mentioned, however, that he is doing so, especially by Application of conventional Retiiningtechni k may also differ from the desired frame rate, it being preferred that the ratio of element rate to frame rate between 0.5 and less than 2, preferably between 0.8 and 1.4. Higher ratios lead to higher data rates during recording; if the conditions are too low, a continuous flow of motion is impaired. The fact that, if necessary, for sequences with slow motion shots faster recording is also possible in the short term, but should be noted.

It is explained in this context, moreover, that in the present case, as a frame as usual, a part of the video stream is understood, in which case it is often added in a precise manner that the generated frame stream is a high-dynamic frame stream, that is the amount of brightness values is greater than that which can be achieved with a single shot on a given sensor. Accordingly, it will be appreciated that the term "high dynamics" can not refer to a fixed number of detected aperture values or a fixed number of (intensity-related) bits, but varies from sensor to sensor It should be estimated that where sensors with a lower dynamic range per se have four or five frames for each frame exposure row, sensors of later generations may possibly accommodate three, possibly even only two frames per frame exposure row; is readily anticipated by the present invention, although this is not mentioned below as advantageous for all peculiarities, and may be particularly advantageous for single image exposure series having more than two frame exposure row elements and procedures are possible, without being each se ^¬ ready discussed. It should also be mentioned that the single image exposure rows can "cling" to different levels of exposure as the prior art progresses, and that, if appropriate, the exposure stages used need not be equidistant; so it was possible for example Belichtungsrei ^¬ hen instead of 1/50, 1/100 and 1/200 sec also bracketed with 1/50, 1/400 and 1/4000 seconds. It should be noted that the actual supply Stel ^¬ development of video streams to happen based on successive frame bracketing, both itself by means of that camera, which also receives the frame bracketing in an appropriate manner, as arrangement by a data processing to the previously recorded bracketing Frames are summarized in accordance with the invention.

It should, however, be noted in particular that where data processing is still limited on the filing date in real time in video cameras with reasonable energy expenditure, integration into cameras is readily anticipated by future developments in microelectronics; It may be mentioned that, if necessary, in order to enable all subsequent subsequent -perse automatic, but if necessary, manually parameterizable-processing, individual frame exposure row elements can be stored instead of and / or preferably in addition to high-dynamic frame streams. The single image exposure series can be generated in different ways and in particular by the gain on the sensor, the aperture and / or the exposure time of is riiert; Particularly simple and therefore preferred is the variation of the exposure time at electronically controlled exposure times. Wherever below is the exposure levels of the frame exposure series, reference can be made to each of these possible variations or combinations thereof, although the particular electronically controlled exposure time is the preferred option. Unlike the variation of the aperture values, this does not lead to changes in the depth of field.

It should be noted that linking to a variety of below of ^¬ fenbarten methods of Einzelbildbelichtungsreiheneleraente- to frame streams high dynamics is useful to determine the capture time much more accurately than for photographic cameras currently standard; it will be especially anticipated that preferably the accuracy of the time signature will be better than the shortest intended exposure time. The video camera of the present invention will thus differ from a conventional bracket-operated and optionally fast-acting camera in that the individual image exposure row elements are preferably provided with a high-precision time signature for each element, namely in particular with a time signature which is at least as accurate or more accurate than the shortest possible exposure time; The time of the exposure is thus recorded, for example, to 1/2000 seconds exactly and erfoderlichen if written away with the element together. This is advantageous where, according to the invention, retiming techniques are to be applied later, and therefore applies in particular to those where the single-frame exposure sequences are recorded as possible and preferably with non-rigid exposure conditions. For example, in non-rigid exposure conditions of the elements of an exposure series, it is possible not to equally divide the available recording time for one frame every frame, for example, 1/30 second, 1/60 second, 1/125 second, but approximately in the case of predominantly bright images with nevertheless very dark parts, to choose as exposure times a 1/30 second, a 1/500 second and a 1/1000 second. A suitable non-rigid ratio may preferably be determined by reference to histograms, that is, to the frequency distribution of pixel brightness values of the images. When speaking of brightnesses in the present case, the term "luminance" will generally be used, but it is also readily possible to consider the bit value distribution in individual color channels instead and / or additionally.

It is possible and preferred to create tables or the like in order to be able to generate optimum single-frame exposure ranges for particular histogram ratios. ^'Thus, in very dark scenes, that is, a large proportion of low brightness values in the histogram, optionally provided with a particularly long exposure for the lowest exposure level, and the like. It should be mentioned that it is possible to determine the ratios of the single-frame exposure in a single-frame exposure row on a sensor, which optionally only approximately accommodates the actual scene, that is, for example, with separate optics and comparable focal length in approximately the same direction. This can be advantageous if sensors of high dynamics but low resolution are provided for the histogram determination. the should, a particularly fast response to .. changing light conditions is desired and the like.

It is possible and preferred that the high dynamic frames from the elements of Einze-lbildbelichtungsreihen be generated by the individual image elements are grouped together bracketing moving ^¬ tend. If successive Einzelbildbe ^¬ lichtungsreihen be designated by the letters A, B, C .... of the alphabet and the individual elements within a frame exposure series are characterized by trailing Roman numerals, so that for example a first frame exposure series with elements AI, All, A1II, AIV then a second frame exposure row with frame exposure frame elements BI, BII, BIII, BIV, then the subsequent frame exposure row with the frame exposure row elements CI, CII, CHI, CIV, where AI, BI, CI are particularly long and AIV, BIV, CIV are respectively very short exposures be a sequence of frames FRAM1, FRAM2, FRAM3 ... can be determined as follows:

FRAM1 = (AI, AH, AIII, AIV)

 FRAM2 - (AH, AIII, AIV, BI)

 FRAM3 = (AIII, AIV, BI, BII)

 FRAM4 = (AIV, BI, BII, BIII)

 FRAM5 = (BI, BII, BIII, BIV)

 FRAM6 = (BII, BIII, BIV, CI)

 etc.

All these frames have an element of each exposure level, even if these elements do not belong to such an exposure series. The use of the sliding summary of the frame exposure row elements makes it possible to significantly reduce the frame exposure rate in comparison to a situation in which. Single-frame bracketing can only be viewed in blocks.

When frame row elements are slidably combined to generate the frame streams and the frame row elements are taken by exposure rows with different exposure durations, it is not guaranteed from the outset that a video stream with temporally equidistant frames or a video stream can be made by sliding summary without any further measures is generated, the time-equidistant information is based. However, this can be ensured in several ways. On the one hand, the aperture value could be changed to record the single-image exposure series; but this is less preferred because the aperture value change usually leads to a change in depth of field, etc., which can have a negative effect on the aesthetic impression when viewing a video stream generated according to the invention. Further, and preferably, the time available for taking a still frame can be divided into equal blocks, for example, four equal parts for frame row elements AI-AIV. Neglecting the time to read the sensor and write off the read sensor data, the longest exposure, for example, fully exploit its allocated quarter of the total time available, the slightly shorter second exposure level will only half utilize its assigned quarter, while the other half passes unused; the exposure time is preferably in the middle of the exposure time. duration and thus also in the middle of the second Be ^¬ Lichtungsstufe associated block; The third stage is again exposed in such a way that, although it is also in the middle of its block, it spends an even longer time idle. Thus, a considerable amount of time, which would be available for the exposure, is spent waiting. Although time-dependent individual image row elements are generated in this way, and with a rolling summary it is possible to dispense with retiming if necessary; However, the disadvantage of this technically very simple procedure lies in the fact that in this way the total exposure time available to the single-image exposure row is only partially utilized, which may result in particularly dark parts being still slightly noisy.

One way around this problem is to immediately start the next, shorter exposure during a frame exposure series without pauses, and then provide a timing method with respect to the non-equidistant elements. Accordingly, it is also possible to start recording the first element of the next exposure row immediately after the last element of a single-frame exposure row has been recorded. Optionally, various methods may be combined to spend only a minor portion of the time waiting for the exposure of the next element of the exposure series.

It should be pointed out that, where appropriate, especially where the exposure times for varying the exposure levels are varied as preferred, recourse can be made to per se known retiming techniques. In principle, with retiming techniques it is attempted to make use of a sequence. in the case of existing recordings, the passage of time of movements and the like recorded therewith. and then interpolating how that affects an image sequence whose frames are at the desired (and different from the real recording times) times.

It is therefore preferable to provide suitable retiming means. For example, initially a sequence of successive images HDR effected by sliding Summary and then Retiming- (interpolation) of the thus generated are highly dynamic (intermediate) frames an equidistant frame sequence gene ^¬ riert. Alternatively and preferred, however, is another approach to retiming, which not only applies per se known reti- ming techniques to highly dynamic frames.

Thus, a retiming preferably takes place per exposure step, which is even particularly preferred in the case of rigid exposure state ratios; to the possibility, both variants of the Retiming, thus the Einzelbildreihenelementbetrachtung for the timing of highly dynamic (intermediate) frames is pointed out. Further, it should be noted that in floating element aggregation and retiming to obtain equidistant frames, those frame row elements that are approximately more accurate for a respective frame may be given a higher weighting with respect to the location of identified moving objects, or with respect to those image portions included in FIG a frame row overlap area of the intensities. Alternatively and / or additionally, it would also be possible to generate specific information from the single-image exposure row elements of a comparable or identical level, and mes high dynamic turn in response to generie out ^¬ ren. This will be further discussed in the following. Incidentally, it should be pointed out at this point that where specific steps are taken to manipulate data processing steps or individual image exposure row elements, this also applies regularly, even if not explicitly discussed, to automated processes on corresponding automated arrangements which, in turn, make unnecessary the intervention of human spirit in carrying out the procedure. The suitable data processing means can be realized by dedicated circuits such as ASICS and / or by programmable data processing equipment. In a particularly preferred variant, the means for providing high-dynamics frames of the video arrangement of the present invention comprise means for determining the high-dynamics frames in response to optical flow information obtained from different intensity levels of successive frames of exposure. The optical

River is known per se from image analysis. In this case, displacement vectors are determined for each pixel. For example, edges can be identified in a first image and edges can also be identified in the subsequent image. Edge identification methods are known per se. Since objects in an image are delimited by such edges, the associated edges will move as an object moves in the image. If corresponding (edge) pixels of successive images are assigned to one another, information about the movement results. The assignment of certain pixels in an image to the corresponding pixels in a subsequent image yields displacement vectors. The displacement vectors of the pixels together form a field of vectors, which is referred to as optical flow.

It is already known for conventional video recording, agree to the optical flow to account for movements to be ^¬. However, this is not always possible for all points. The present invention now proposes to search for a given optical feature, for example an edge, not only in the same exposure steps on successive frame exposure rows, ie, for example, not only those edges which are exclusively found in the frame exposure group elements AI, BI, CI, DI; Rather, it is proposed according to the invention to track an optical feature, for example an edge, even beyond the limits of the exposure stages. A Traj ktorienfeld can thus be constructed, for example, by evaluating the frame exposure sequence elements AI, BI, CII, Dil, EIII, FIV. In this way, objects can be tracked well, even then. when they move into shaded areas or when, for example, self-luminous objects, the brightness changes, so that in a darkening object this no longer or no longer exactly in the Einzelbildbelichtungsreihenelements highest exposure level can be found. It exploits the fact that, due to the overlap of the sensitivities, a feature that is just disappearing in one exposure stage will appear in the neighboring one if it is still in the field of view. It is otherwise possible to construct overall trajectory fields. For this purpose, the trajectories of all objects which can be found in any of the exposure levels or in several overlappingly evaluated exposure levels can be considered and summarized. This may be advantageous and important in particular for retiming applications.

It is particularly preferred if a motion blur signature is synthesized. Motion blur occurs because a finite exposure time is required to capture a scene and some objects in a scene move during the exposure time. An example is the traces of light from the vehicle lights in nocturnal street scenes. The presence of such motion blur designs, that is, of "motion blur", is perceived as being particularly aesthetic, and to provide an aesthetically pleasing video stream, it is preferable to provide a means of synthesizing the motion blur signature in real scene scenes It is particularly preferred if the (automatic) synthesizing of the motion blur signature occurs in response to the trajectories of optical features It is possible to determine a separate trajectory field for each exposure step and then to add the individual information obtained for each exposure step. the trajectories given optical characteristics in a ent ^¬ speaking exposure step can here in turn, as described above, by Mitauswertung of the total flow, as it results from different exposure levels can be determined. it can also Plaus ibilitätsbetrachtungen and the like done. In this way, aesthetically pleasing results can be obtained. An example may illustrate this:

If, above the camera looking upwards, a light bulb suspended from a thread describes a circular movement, then it will be clear that even the blurred illumination should lead to a luminous arc. The particularly bright glowing filament is then added with particularly short Belich ^¬ obligations, the interconnected might look more like a polygon than a circular path, the prolonged exposure of the individual exposure rows indicate the position of the incandescent filament radiated over a tracer; if necessary, this tracer will be much wider than the actual filament. It will be appreciated that an aesthetically pleasing tracer light tracing arc must move in this broad area overexposed in other exposure levels, even though the direct connection of the frame row elements captured with correct exposure for the filament provides a polygon along the circumference. In order to obtain a particularly appealing motion blur signature, a movement for the pixels representing the incandescent filament, such as spline curves and the like, is now determined by evaluating the total trajectory fields and, using known etching techniques, a corresponding trajectory of the bright luminous spot certainly. Such a determination is also carried out for all other brightness levels and only then is the addition of the corresponding data to the background carried out. This results in aesthetically pleasing motion-sharpness signatures. It is also preferable if an image quality enhancer is provided which is adapted to identify and, if necessary, correct for sites of pronounced sensor linearity effects. This takes into account the fact that an image sensor typically does not have a completely linear response, but deviations from the linear behavior in particular in the range of very high brightness, ie shortly before saturation, and in the range of very low brightness, where noise also has a stronger effect occur. In a related scene, that is ei ^¬ nem still image, it would in principle be possible to correct itself in a highly dynamic image or frame resulting impact. It could be exploited per se that all objects remain in the same place in the image. However, such a linearization is not readily vornehmbar when moving objects, since then the movement makes it difficult to identify particularly affected areas. The invention now proposes to take advantage of the fact that intensity jumps occur at locations of pronounced sensor linearity in a highly dynamic image that does not take account of them, without these having corresponding edges of real objects being imaged in the individual images. In this way, a position of pronounced sensor linearity effect can be determined by comparing the intensity distribution in a highly dynamic image with the intensity distributions in the individual images; It is more preferable to look for edges that occur in the highly dynamic image, but not in the single-frame exposure row elements in which they actually have to be found. Such edges found only in the HDR composite image or others Optical features indicate sensor linearity. Because the eye is particularly sensitive to such edges, a correction can be made by spatial filtering eliminating such artifact edges. Preferably, the larger the gradient of the artifact edge, the greater the corrections made by the filter. Thus, there is a quantization of the sensor linearity effect and an impact quantification-dependent correction. It should be mentioned that it may not be necessary to use the single-image ^illumination stages themselves for the artifact edge identification, but optionally instead also and / or additional information obtained by Retirning can be obtained from individual image row elements via real-object edges. It should be further noted that the Arte ^¬ fakt-edge identification by the gradient analysis of a high dynamic image and comparing the edge position in Hochdyna ^¬ mikbildern with edge positions in Einzelbildbelichtungsreihenele- instruments, that is capable of real object edges, is interesting not only for video technology, but is regarded as inventive by itself in the processing of conventional high-dynamic range photos and thereby considerable advantages in the correction of Sensororalineaitätsausgaben even with conventional still photos with high dynamics may result, which are considered to be patentable and particularly justify a divisional application without intending to adversely affect the ability to separately protect other aspects of the present invention.

Protection is also claimed for a method of providing video streams in which the frame streams are high Dynamics from the successively recorded single-frame exposure sequences of successively recorded individual images by automatic combination of single-image exposure-row elements in single-frame exposure arrays overarching manner erf.olgt. It should be noted that such automatic methods are executable on dedicated circuits, dedicated or general purpose data processing equipment, etc. Automatically means that no human or human intervention is required to generate the high dynamic frame streams through the inventive combination. However, certain parameters may optionally be settable by a user, for example, the resolution of a generated high-dynamic frame rate, dynamic range, high dynamic frame rate, exact frame rate high dynamic range, destination for the high dynamic range generated frame stream etc. are given. Such specifications will be made before the actual, automatic combination. Further protection is claimed for a method in which single-image exposure trainers are automatically slidably linked. The floating link allows the frame rate frame rate to be virtually matched to the frame rate. Thus, if appropriate, the amount of data can be significantly reduced, for example (approximately) by a factor of three or more, in comparison with multi-sensor applications in which all single-image exposure row elements are simultaneously written off. The method can evaluate flow information to determine the high dynamic frames. In particular, the flow information can be used for interpolation, so that a individual exposure bracketing elements can be displayed in real time. It is particularly preferred also when a "motion blur" that is a motion blur signature is synthesized, which in turn provided automatically in real time while recording, a sufficiently powerful computing, or below the Aufnahrae, that is offline, happened in automatable ^¬ ter way can. in a particularly preferred variant of the method the effect of a per se unknown Sensoralinearität in the perception of frames high dynamics is corrected automatically based on edges of the spatial brightness curves are determined in the frames high dynamics and / or Pla ^¬ teaus which no equivalent in Preferably, a correction of such locations is made automatically, and instead such locations are merely marked, for example, by altering the chrominance values of affected areas to indicate problematic locations to a user, and to support such a manual correction by automated processing is nevertheless disclosed as a possibility.

The invention will now be described by way of example only with reference to the figures. In these is shown by

1 shows a scheme for recording individual image row elements which allow a particularly simple generation of a high-dynamic-framefrom successively recorded individual images; 2 shows a further scheme for the successive acquisition of individual image rows, wherein four individual image rows A, B, C, D are illustrated with elements I, II, III, IV;

FIG. 3 shows the intensity distribution from pixel to pixel in a given line of an image and the intensities, which are recorded in steps with individual image row elements, for this line; FIG.

4 shows a {symmetrically imaginary) curve in which the actual intensity distribution of an object in an image line is shown on the left and in the right half of the curve the corresponding intensity distribution caused by artifacts which results from the combination of a plurality of (still standing) single image row elements I to IV results;

Fig. 5 is an illustration illustrating general artifact edge formation in HDR composites due to sensoralinearity;

FIG. 6 shows an illustration of the trajectory formation that spans individual image row elements or the determination of the optical flow in single image row elements in a cross-over manner; FIG.

 Fig. 7 is an illustration of a preferred method according to the invention for producing a

Motion blur signature ("rnotion blur"). In accordance with the present invention, a video array for providing high-speed video streams based on successively captured still images comprises means for automatically providing the high-dynamics frame stream from single frame exposure row elements taken in frame exposing rows at a frame rate corresponding approximately to the frame rate. In the following it is assumed that the video arrangement of the present invention is a fully integrated video camera which automatically executes all technical steps between taking a frame row element and generating the high dynamic frame stream inside the camera body. It will be appreciated by those skilled in the art that the camera per se will have conventional elements that no longer need to be described, such as camera body, power supply, viewfinder, lens, etc. In this respect, such details are not explained, but assumed to be known. Also, the detailed description, for example, of electronic circuits such as implementing the invention serving ASICS is omitted because about their structure is not only highly variable, but based on the information disclosed here, a corresponding design readily, possibly with the involvement of appropriate hardware designers will be possible ,

It is further pointed out that, although for reasons of illustration a camera completely integrating the complete generation of the high-dynamic frame current is described, it is readily possible, for example with the camera, initially only a sequence of inputs. receiving zelbildbelichtungsreihenelementen, namely tenrate with a desired later frame rate at least approximately corresponding Einzelbildbelichtungsreiheneleme- and series elements, these successive series of Einzelbildbelichtungs- that are approximately aufgenoirunen with the frame rate corresponding individual bracket elements rate outside the Kameragehauses according optionally he ^¬ folgter caching to a frame power high dynamic to link. In such a case, both the corresponding system with which the stored Einzelbildbe- series rows elements are linked to the frame stream of high dynamics, an inventive means for automatically providing the high-speed frame stream from recorded at least approximately the frame rate corresponding Einzelbildbelichtungsreihenelementenrate recorded individual image exposure row elements in frame exposure modes, namely, a corresponding frame exposure element linkage stage, which in turn may be implemented in the form of suitable dedicated hardware, for example by ASICS and / or in an alternative manner; the same will happen. A means for automatically providing a high-dynamic-frameframe comprising at least approximately frame rate element frame rate applied Einzelbildbelichtungsreihenelementen in Einzelbildberichtungsreihen übergrei ender manner, namely insofar as the camera receiving means are operated so that they particularly well the automatic provision of high-dynamic frame current are adjusted; it will be understood from the foregoing that in such a case where the camera stores the still image exposure line items and the stored frame exposing row items follow each other. Are linked to separate hardware, the respective means will be different and yet both devices individually and for themselves as a commercial usable elements each embody the invention.

By way of explanation, it should be further noted that the present text is concerned with frames, single-frame exposure rows and single-frame exposure row elements. In this case, frames are taken as individual parts of a video stream, whereby the video stream is given by the sufficiently fast sequence of the individual frames; For conventional HDR still photography, where the combination of individual photographs of an exposure series taken to form a single image, that is, a frame exposure series, is referred to herein, reference is made to "elements" of the exposure series to make it clear that not only Stand photography techniques must be meant, although from the invention as disclosed but in some places certainly also benefits for still photography arise.

The frame rate will be about 24 to 30 frames per second, corresponding to the temporal resolution; However, it should be mentioned that possibly higher temporal resolutions, for example to enable better slow-motion shots, are conceivable.

It is further pointed out that the term "high dynamics" refers to the fact that the dynamics of the frame stream is greater than the dynamics that can be achieved with a single exposure on the sensor used, the term "high dynamics "therefore comes no fixed Importance to; nevertheless will be appreciate that, as early as the requirements about the cinema and film industry to ge ^¬ nügen can be understood and readily typically a dynamic range of, for example, 20 bits to "high dynamic range".

Given all this, a single image exposure row according to FIG. 1 is now taken with a camera in a first operating mode such that four elements AI to AIV are recorded per exposure row. For each element of the single-image exposure series, the time dt is available, which should last exactly as long as one frame; the times needed to read out a sensor are otherwise neglected for the following consideration; the trade ^¬ man but readily recognize how he may take into consideration ^■ term.

It is first selected a gain for the sensor, which is so large that when exposure over the entire frame time, so for example, with 1/30 second, the dark areas of the scene to be recorded are still well lit; then, for the first element of the single-image exposure row, the - preferably electronic - shutter of the camera is opened and exposed for the frame duration. Subsequently, the second element of the single-image exposure series is recorded. For purposes of illustration, it is to be understood without limitation to the generality or the disclosure that this element is exposed to me only one exposure step difference, that is, by varying the exposure time, with 1/60 second and equal gain of the sensor. It is then waited a quarter of the frame duration, the electronic shutter of the camera open for the intended time, so that after closing the shutter a quarter of the frame will continue until the end of the frame duration. Thus, the middle of the exposure period is exactly at the middle of the frame duration.

5 Now, the third element of the frame exposure range is received, and it is assumed that for the duration Belich ^¬ tung again half of the previous one, so should be a 1/120 second. This is 1/4 of the frame duration. It is now waiting for a 3/8 duration of the frame, then for the

10 provided exposure time, which is a quarter of the frame duration, opened the electronic shutter of the camera and then waited for a 3/8 frame duration, until the time available for recording the third element of the frame exposure series has elapsed.

 '5

 Then, the fourth element of the frame exposure series is taken, in the present case it is assumed that this is exposed with 1/240 second exposure time. To record this element, a 14/32 frame period is first waited for after the beginning of the respective frame duration 20, then the exposure takes place and then again a 14/32 frame period is awaited. Again, the middle of the exposure interval is exactly at the middle of the frame period.

25 This results in a single image exposure series with four elements whose exposure time centers are exactly in the middle of the frame period, regardless of the actual exposure time. It should be noted that compliance with this exact match of frame periods

30 mid and exposure time center is advantageous, but not mandatory and quite deviations may occur, so that about the element 2 slightly before the middle of the frame period center is centered and element 3 is something afterward. It is easily possible to subtract from the frame duration the time required to write off the respective element data from the sensor. Also in this case, the elements are not exactly in time centered. Nevertheless, the elements can be centered approximately equidistant temporarily, with deviations of, for example, 10 to 20%, possibly not perceived by the human eye. However, the exact ^'compliance is preferred tion for a particularly simple imple-.

It should be noted that the frame exposure series can be easily picked up by providing a corresponding "interval timer", the intervals will change from time to time with the successive progress of the frame sets, but typically repeat periodically to record a first frame exposure row immediately thereafter to capture a second frame exposure row in the same manner, etc. Such automatic interval generation is considered new. It may be in response to the absolute exposure time and / or the frame rate and / or the number of frame exposure line items in the frame exposure row.

This results in a frame exposure series AI to AIV, BI to BIV, CI to CIV, wherein each element of a frame exposure series of the subsequent element is offset in time by a frame duration.

Frames are now generated in a sliding manner by combining the frame exposure elements in order to to provide high dynamics. For this purpose, the elements (AI to AIV) are for the first frame Fl summarized, for the second frame F2, the elements (All to AIV, BI) are interconnected ^¬ quantity sums for the third frame F3, the elements (AI II, AIV , BI, BII), for the fourth frame F4 the elements (AIV, BI, BII, BIII), and so on. Under summarization, the generation of an HDR composite is obvious. There is thus produced sliding over the Einzelbildbe ^¬ clearing rows across the frame current high dynamics, with those HDR techniques could be used for the combination of frames per se in a special ^¬ DERS simple implementation of the invention, which is also from the stand, photography per se are known, but typically OF INVENTION ^¬ dung as just no removal of so-called "ghost" is ER- follow.

In this way, simple retiming techniques can be implemented on the most minimal restrictive HDR camera simply by providing the timer for providing suitable intervals between the frame row elements. It should be noted that it is not necessary to use the frame duration completely. It should also be noted that it is not necessary to combine four elements into a single frame exposure series. Optionally, for example, three shots could be sufficient or five shots can be used. It will be seen from the above explicit example, Rauss like, carried out depending on the number of shots and the Ver ^¬ ratio of the exposure times in an exposure series, the exposure time control, that is, how the inventions dungsmgemäße interval timer means has to control the intervals of exposure.

- 2 B - It could be and / or it would be possible to vary the time ^{interval of} the element ^recording in a single-frame exposure series comprising two elements as a function of the actual exposure time.

While the recording scheme described above permits be ^¬ Sonder simple frame power generation, located disadvantages to the extent that in many exposure levels elapses a great time in which no exposure. This

 i

is disadvantageous in particular if the longest Belich ^¬ tung duration should be increased at the expense of the other elements in order to reduce the noise in the image; This can be achieved, although then additional techniques make sense to generate a jerk-free seeming frame stream can. This will be explained below.

In Fig. 2, an alternative sequence is proposed, wherein the recording of four frames A, B, C, D is illustrated. Plotted are the time blocks in which the frame exposure line elements I, II, III, IV are recorded, and it can be seen that the individual centers of the frame exposure line element durations do not have the same time distance from each other; that between about the shortest block of the frame exposure series A and the longest block of the subsequent exposure series B is a recording-free break, incidentally, only the higher over ^¬ clarity of the drawing owed and must not be technically the case.

It is possible to capture the frames at frame rate row rate, such as the frame rate is shorter or shorter, but leaves more time for the dark spots. It is assumed, for example, that the duration of the block A, the block B, the block C and the block D each correspond to four frame durations. Obviously enough, could instead, which would lead to fewer problems when retiming, as will be nachfol ^¬ quietly appreciated only twice or 1.5 times the amount of time can be selected. It can be seen from the figure that in any case more of the total time available for a single-frame exposure row is used to take the darkest picture, which reduces the noise. The sliding summarizing about the elements (AI to AIV) to the first frame, the elements {All, AIII, AIV, BI} to the second frame, the elements (AIII, AIV, BI, BII) to the third frame, etc., that the generated (interim)

) Frames of high dynamics are not equidistant. Retiming techniques are therefore preferably used.

Per se, this would be possible with known retiming techniques using only the frames of high dynamics. Although this possibility for protection is not excluded who should ^¬, is revealed that this is not the cheapest Me ^¬ Thode to produce a particularly high-quality frame stream; especially fertilize for high-quality applications such as video streams other techniques USAGE ^¬ det therefore that not only use the combined information, but a direct link to the existing in the frame number information as elements such reference. Therefore, it is sensible, especially where particularly high-quality frame streams of high dynamics are desired, to write off the single-frame row elements per se as such and, if appropriate, to either improve on a simultaneous recording. have been able to enable high-dynamics frame streams, or to provide high-dynamic-frame generation in a separate post-capture phase from the outset. To understand how a more aesthetically pleasing frame current high dynamics can be erf ndungsgemäß produces particularly good, will first be generally the HDR technique beschrie ^¬ ben. Based on this and the knowledge that the human eye is particularly sensitive to the edges, then the suggestions are explained how can be provided by the present invention, the aesthetic feeling of the viewer particularly ^¬ speaking framestream.

Reference is first made to Fig. 3, which is intended to generally illustrate the HDR technique as it belongs to the state of the art per se, for example in still photography. For the purpose of a basic understanding of artifacts occurring in the HDR technique, there is initially no movement.

For example, if a scene is taken in which - as shown - a glowing light bulb is to be detected in an otherwise empty room, so typically the Glühwen ^¬ del extremely bright and the areas around the filament, such as the holder of the same or a teilmattierter Glass bulbs of the bulb will also be very bright, while the socket and its shadowed transition to the cable, where the bulb down from a ceiling, are dark.

In Fig. 3 this scene is shown and it is shown how the intensity distribution on a pixel comprises Represents the line of an image sensor. The intensity assumes extremely high values, which the sensor can no longer comprehend holistically in a single image. Therefore, the HDR technique proposes the recording of the frame exposure elements with different sensitivity, so that the intensities I, II, III, IV result. This is shown on the right in FIG. Ideally, can thus determine the correct intensity of the individual image ^¬ series elements. FIG. 4 illustrates, however, that deviations from the ideal curve occur due to sensor linearities in the respective lower and upper sensor range. There are steps and steps in the intensity distribution that are not present in reality. These levels in the intensity distribution, which is obtained over a line of the sensor, when viewed flatly in the element or photo results in a linear intensity error, that is, an artifact. This is illustrated by the four frame exposure row elements in Figure 5 and the HDR composite. In it, artifact lines are found around the filament in the HDR photo, next to a real line corresponding to the glass bulb of the pictured bulb.

The human eye detects such edges particularly sensitively; Therefore, the treatment of these edges is of particular importance, because the artifact edges are easily recognized and disturb accordingly.

At the same time, not only artefact edges are important, but also the movement of the edges from frame to frame in the video stream; this is already common in retiming techniques considered; but in the present invention there are advantages over conventional retiming techniques.

Retiming techniques work per se and in a known manner in such a way that edges with corresponding optical features are identified in images using methods known per se which can also be used both for the frames of high dynamics and for the individual image exposure row elements. Edges identified in successive images are then assigned to each other.

For example, if all the edges of an object move from image to image by the same distance and in the same direction, the object has moved as a whole. If the movement of the object per se is known, for example as a uniform movement over several images, it can be interpolated where a particular object would be located between frames at any given time and a corresponding frame can be generated for this arbitrary point in time by retiming techniques known per se without being exactly at this frame time, an image capture.

With this understanding, it will be seen below how improvements in retiming are possible to achieve a more aesthetically pleasing frame stream.

The accuracy of such retiming techniques depends on the fact that enough features are identified, that is, enough trajectories of corresponding features can be determined. As explained with reference to FIG. 5, not all features can be found in all images. Fig. 5 veranschualicht first of all that edge structures in the HDR composite occur ktkanten one hand as ArteF that no equivalent in the frames or elements be ^¬ sit, and that while corresponding the other hand, real objects edge structures as those of the glass bulb occur in specific frame bracket elements , but can not be found in all frame exposure line items, for example because they are not bright enough (IV), or completely outshone (I).

The invention now intervenes to eliminate on the one hand artifact edges and on the other hand - per se, regardless of the artifact edge elimination - to improve the identification of real edges.

A typical case is considered in which a brightness varies and a movement takes place. This can happen in real scenes when a short reflex is detected by an object moved by a light beam. In the present case, for purposes of illustration, a moving lamp is considered, which is dimmed periodically simultaneously with its movement, so that the associated edge features no longer remain in exposure levels of the same intensity and accordingly can no longer be found there.

Illustrates this with Fig. 6 by an over the camera when recording around its connection to the power cable circular light bulb, the intensity of which intensity is also changed periodically. The light bulb thus describes a circular path; the corresponding characteristics of the light bulb, for example the edges belonging to the glass bulb, can however, depending on the current brightness of the light bulb, be reduced to different levels. find different frames, as indicated by the caption along the trajectory.

The invention now proposes to generate trajectories by tracing features that disappear at one exposure level in other exposure levels. Typically, this is unproblematic insofar as the exposure levels are indeed chosen so that there is an overlap. When the lamp is extinguished, therefore, the glass bulb edge will initially no longer be found in the element recorded with a particularly short exposure time, but in that with the next longer exposure time; then, if the corresponding edge is no longer detected there, it is due to the overlap in the following again. This is used to piece together the trajectories.

It is now proposed to use such, generated in the sensitive ^¬ keitsstufen cross manner trajectories for Retimingtechniken. This is specifically advantageous for the generation of high-dynamic-time frame streams.

Furthermore, the particular trajectories, that is, the optical flow, are preferably used not only for retiming techniques, but also for creating a motion-sharpness signature, which is also considered to be particularly aesthetically pleasing.

In principle, it would be possible to generate the motion blur signature only from the composite images, that is, the high dynamic frames. It is to be appreciated, however, that it will be considered more aesthetic if first of all light trails for each feature, that in a given exposure level can be determined using retiming techniques known per se. This results in four motion unsharp part signatures or traces of light, each showing different optical characteristics. _Sun For example, it is possible to calculate the light track which the extremely bright filament travels in the time in which the comparatively long exposure takes place, which captures the shadowed connection between the cable and the bulb socket. By separately for each exposure step the respective ge light trace is determined and then the thus obtained by retiming Einzeltra ^¬ techniques are added ektorien, results in a particularly appealing Bewegungsunscharfe- signatu. It is further suggested, and independently of this, that artifact edges present in the composite which are not found in associated frame-set elements are eliminated, in particular by spatially averaging.

Thus, an image analysis is performed on edges in the composite image and an image analysis on edges in the underlying individual image elements. Then an identification of edges takes place in the composite image without correspondence in at least one single picture element or one single picture element pair. These are preferably eliminated or partially suppressed.

For this purpose, the intensity values at an artifact edge location can be corrected by offsetting with intensity values from the environment of the artifact edge in order to make the artifact edge appear less hard and thus to disappear to the human eye. The way in which the can resort to techniques as they are known per se to make a picture "softer"appear; but it was mentioned that it is advantageous to treat the be ^¬ Sonder strong artifact edges particularly strong, so apply a greater degree of spatial filtering than just weak artifact edges. Quantification per se is known per se.

This is also disclosed as advantageous for stationary still photography.

Claims

claims

1. Video arrangement for the provision of

 Video streaming high dynamics

 based on

 successively recorded

 Frames,

 being an agent

 for the automatic provision of a high-speed frame stream

 out

 With

 one

 at least approximately the framerate corresponding

 Frame exposure row element rate recorded

 Still image exposure row elements in

Frame-by-frame exposure series and ^■ a means of synthesizing a texture blur design

is provided. Video camera according to one of the preceding claims, characterized in that the means for synthesizing a Bewegungsunschärf signature is adapted to the Bewegungsunschärfesignatur

 in response to interpolated trajectories

 optical characteristics

 in successive frames of exposure.

Video camera according to the preceding claim, characterized in that the means for synthesizing a motion blur signature is adapted to the Bewegungsunschärfesignatur

 in response to

 interpolated trajectories

 optical characteristics

in exposure frame exposure elements corresponding exposure level to synthesize. Video camera according to one of the preceding claims, characterized in that the means for synthesizing a Bewegungsunschärfe- signature is designed to the Bewegungsunschärfesignatur

 in response to

 separate for each exposure level

 certain trajectories

 optical characteristics

 to synthesize.

Video camera according to one of the preceding claims, characterized in that the means for synthesizing a Bewegungsunschärfe- signature is adapted to the Bewegungsunschär esignatur by addition of

 obtained in response to trajectories

 dates

 to a picture background,

 especially

 by addition of

 to a

to synthesize for each given exposure level specifically determined Bewegungsunschärfesignatur. Video camera according to one of the preceding claims, characterized in that

the means for providing video streams of high dynamics

a means for capturing the frame exposure series by varying at least one of

 gain

 Cover,

 Exposure time,

 in particular electronically controlled exposure time,

is designed for the Einzelbildbelichtungsreihenelemente,

 especially for recording on a number of sensors smaller than the number of elements in a single image exposure series,

 in particular on a single sensor, in particular in such a way that it is designed for single image exposure row recording with non-rigid, in particular histogram-related conditions, the camera being provided with a different sensor for histogram determination, in particular, and / or for using a previously determined histogram is.

Vide ^'oanordnung according to any one of the preceding claims, characterized in that - the means for providing video streams with a high dynamic means for generating frames of high dynamics of the single- exposure series is provided, in particular under He ^¬ ranziehung sliding summarized Einzelbildbelich- series elements.

A video device as claimed in the preceding claim, characterized in that the means for providing high dynamic frames comprises means for determining the high-dynamics frames in response to optical flow information obtained from different intensity levels of consecutive framesets, in particular in response to optical flow information. which is obtained to image features that can be found in successive frames of exposure in different intensity levels.

Video arrangement according to one of the preceding claims, characterized in that the means for providing a frame of high dynamics comprises means to encourage improvements ^¬ tion of the image quality, which is adapted to locations marked Sensoralinearitätsauswirkung in a frame of high dynamics by reference to exposure gradually recovered image information identify and if necessary to correct, in particular under impact quantification and quantification-dependent correction.

Method for providing video streams of high dynamics, based on successively recorded individual images,

in which

a frame stream of high dynamics from at least approximately the frame rate corresponding Einzelbildbelich frame row element rate recorded Einzelbildbelich- series elements in single image exposure rows is combined automatically over a gripping way

 And thereby

 a motion blur signature in response to

 interpolated trajectories of optical features, preferred

 interpolated trajectories of optical features in frame exposure frame elements of corresponding exposure level,

 especially preferred

 for each exposure level

 in successive frames of exposure

 certain trajectories,

 especially preferred

 by adding data obtained in response to trajectories to an image background,

 is synthesized, particularly preferred

is synthesized by addition of a particular given Be ^¬ lichtungsstufe specific specific motion blurred fesignatur.

11. The method according to the preceding method claim, characterized in that the Einzelbildbelichtungsrei hen elements are automatically slidably linked to provide successive frames of high dynamics. Method according to one of the preceding Verfahrensansprü che, characterized in that points of pronounced sensor linearity impact in a frame of high Dynami automatically identified by reference to exposure level obtained image information and require chenfalls be corrected, in particular under automatic shear effect quantification and quantification or classification-dependent correction.

Method according to one of the preceding method claims, characterized in that the motion blur signature

 under evaluation of

 light trails

 for respective characteristics,

 the

 can be found in a given exposure level,

 and the

 by retiming techniques for it

 be determined

 and addition

 of the

 thus obtained

 Motion blur partial signatures

 is synthesized.

Method according to one of the preceding claims, characterized in that features which disappear in a Beiich processing stage, be followed up in other exposure levels.