CA2386250A1 - Method for image data processing and camera recording system - Google Patents

Method for image data processing and camera recording system Download PDF

Info

Publication number
CA2386250A1
CA2386250A1 CA002386250A CA2386250A CA2386250A1 CA 2386250 A1 CA2386250 A1 CA 2386250A1 CA 002386250 A CA002386250 A CA 002386250A CA 2386250 A CA2386250 A CA 2386250A CA 2386250 A1 CA2386250 A1 CA 2386250A1
Authority
CA
Canada
Prior art keywords
background
camera
recording
frequency
data set
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
CA002386250A
Other languages
French (fr)
Inventor
Steffen Setzer
Dittmar Wright
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SCANNOVA GmbH
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Publication of CA2386250A1 publication Critical patent/CA2386250A1/en
Abandoned legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/262Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272Means for inserting a foreground image in a background image, i.e. inlay, outlay

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Television Signal Processing For Recording (AREA)
  • Studio Devices (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to a method for image data processing, whereby object data of an image data set representing an image foreground (object data set) is separated from data set representing an image background (background data set), whereby a background to be recorded by a camera is changed into a frequency matching the recording frequency of the recording camera. The background is changed, for instance, by switching between two states or by gradually switching between two states having a plurality of intermediate states. The invention also relates to a camera recording system, especially for implementing the inventive method, comprising at least one camera for recording an object located in front of a background, whereby the background is changed into a frequency matching the recording frequency of the recording camera.

Description

METHOD FOR IMAGE DATA PROCESSING AND CAMERA RECORDING SYSTEM
The present invention relates to a method for image data processing in which object data of an image data set representing an image foreground is separated from a data set representing an image background, as well as a camera recording system, which is particularly suitable for performing a method of this type.
The blue screen method is known from television technology, in which a surface forming the background of the setting to be recorded is implemented as a monochrome, particularly blue, background. The recording of an object in front of this background is first recorded by a television camera at an image frequency of, normally, 50 to 60 half-frames per second, while in parallel to this a background replacing the blue background of the first image, for example a weather map, is recorded by a second television camera. The two image sequences recorded in parallel are superimposed in synchronization on one another in a mixer and the blue background area is replaced by the image recorded with the second television camera.
In the blue screen method, two principle methods are differentiated, the first of which is based on a definition of the chromaticity (the color representing the background is defined as transparent), while in the second method, which was commercially exploited particularly by the firm Ultimatte, the background is defined via the consistency of the brightness and/or saturation of the color space of the background.
The advantage of the known methods is that the processing of the analog image data is relatively simple and occurs in real time.
It is a disadvantage that the typically blue, but also frequently green, background has to be very well illuminated, since the color or brightness and/or saturation of the color, respectively, must be very uniform and of high quality in the image sequence WO 01!06766 PCT/EP00/06827 recorded, which may only be achieved by excellent illumination of the colored surface representing the background. This illumination is, however, connected with a very high outlay, since the typical lighting time for a recording studio is approximately one week, or often even more. Not considering the man time connected with this, a large amount of power is consumed and unpleasantly large amounts of heat are produced for those working in the vicinity of the recording. Furthermore, it is disadvantageous that shadows and/or reflections must be avoided, which makes the lighting even more difficult. Reflections may not be corrected with the known methods, and shadows only with difficulty.
However, according to the present invention a method for image data processing having the features of Claim 1 and a camera recording system having the features of Claim 8 is suggested.
The present invention is based on the idea that the background is altered in a defined way at a frequency tuned to the recording frequency of the recording camera. Tuning is understood here to mean that the frequency of the background alteration has a known ratio to the recording frequency. This may be an integer multiple of the recording frequency, but could also be any desired multiple. The recording frequency is preferably a multiple of the playback frequency, for example double the typical television image frequency, i.e., approximately 100 to 120 half-frames per second. In principle, the method according to the present invention functions, as will be described in the following, at any desired recording frequency up to several thousand images per second, with the alteration frequency of the background tuned to this. A requirement is that the content (foreground) of two images recorded in sequence does not alter or at least alters only insignificantly, since every second image of the image sequence is used as a reference image.
' WO 01/06766 ~ PCT/EP00/06827 The method according to the present invention opens up the possibility of using digital cameras and of the utilization of their technical advantages. The recording of an image sequence at, for example, double frequency with a background changing in a defined way corresponding to this frequency leads to a sequence of image pairs, with each two images forming an image pair (at the typical movement speeds of recorded objects) not or barely able to be differentiated from one another except for the background, which alters in a defined way from one recording to the next. One of the two images of an image pair may thus be used as a reference image for the respective other image.
Using the advantages of digital recording technology-each recorded pixel of an image may be directly compared with the corresponding pixel of the following image-the two images of an image pair are compared with one another, in that, for example, each pixel of the reference image is subtracted from the corresponding pixel of the other image. While, as a result of the comparison, a value of zero or almost zero results for the foreground of the object forming the recording if there is no alteration (non-moving or slowly moving object) and an arbitrary value not equal to zero results for a slight alteration (rapidly moving object), the comparison of the background surfaces results in a predetermined defined value, since the background was purposely altered in a defined way from the recording of the one image to the next image. The alteration of the background may be performed by switching between two states or by gradual continuous oscillation, for example sinusoid, between the two states.
In order to now separate the foreground of the object forming the recording from the background, to "clip it out," all pixels to which a defined content was assigned during the comparison of the two images forming an image pair are suppressed and possibly replaced by the content of an image recorded in parallel (for example the weather map). This second image may have any desired ' WO 01/06766 ~ PCT/EP00/06827 source. Instead of the image recorded in parallel as described, a single image (fixed image) may also be used as a background or, for example, even a computer-generated image and/or image sequence.
In a preferred embodiment of the invention, to determine the assignment of an element of a camera image data set to the object data set or the background data set, a comparison of the data sets of camera images recorded at a different instant within a playback frequency is performed. If the recording frequency is, for example, three times the later playback frequency, then during one playback frequency, i.e., within a period corresponding to the playback frequency, three camera images are recorded. These three images are compared with one another to identify any desired element in the camera image as an object or background. The comparison preferably occurs in pairs. Thus, for example, the first camera image of this sequence of three images is first compared with the second image and then with the third image, and finally the second image is compared with the third image. In order to design the method more economically, the comparison may be performed in steps, with sequential comparison steps only performed if the prior method steps have not yet provided an unequivocal result, i.e., have not yet allowed unequivocal assignment of the camera image element as object or background.
In a particularly preferred embodiment, the comparison is a subtraction of elements of identical coordinates or regions of the camera image data sets, with the elements of the camera image data sets to be compared with one another able to be single pixels or even pixel regions. A comparison of pixel regions has been shown to be particularly advantageous, since the method may then be performed more rapidly. It suggests itself in this case to first select larger pixel regions, in order possibly to subdivide these into subregions step-by-step. The method according to the present invention also opens up the possibility WO 01/06766 ~ PCT/EP00/06827 of comparing not only two-dimensional pixel regions with one another, but also n-dimensional pixel regions.
As described above, the method according to the present invention may be performed at any desired recording frequency up to several thousand images per second. The higher the recording frequency is, the larger the number of images having different background states (particularly background brightnesses) recorded within one playback period, which causes the comparison possibilities, i.e., the permutations of comparison pairs, to increase significantly.
In this way, a better foundation for the assignment of a camera image element to the background data set or the object data set is given.
In addition, the present invention opens up the possibility of evaluating the reflection behavior of objects in the camera image at a high recording frequency and gradual background alteration and drawing conclusions about the refractive index and the material of the object from this. This, in turn, allows the adjustment of the relevant data set elements of the object data set to the new replacement background, which replaces the original recording background. In contrast to the methods known from the related art, not only is correction of reflections (and shadows) possible according to the present invention, but also their evaluation, in order to allow the recognition of the affiliation as object or background of an image element or the adjustment of the image and/or the object to a background used.
In this way, shadows and/or reflections on the object and/or the background may be recognized according to the present invention and, depending on the application or the desire of the user, suppressed or adjusted and processed in another way.
The method according to the present invention is preferably performed using a neuronal net or other methods of artificial intelligence (AI). In this way, particularly good recognition and wo 01/06766 pcT/EP00/06827 assignment of the image elements as object, background, reflection, shadow, etc. is achieved.
Finally, the present invention opens up the possibility of investigating the camera image data sets not only for an assignment in the two-dimensional range (object and background affiliation), but also to obtain further information about the object and/or the background, such as the speed, among other things. This in turn allows conclusions about three-dimensional data of the camera image data set (distance from the camera etc.), so that the present invention is suitable not only for two-dimensional, but also for multidimensional image data processing.
Furthermore, it has been shown to be advantageous to alter the recording background only partially during the recording.
To perform the method according to the present invention, film-like or sheet-like elements are preferably used, which may be optically influenced very rapidly in a targeted way, i.e., their color and/or brightness and/or saturation may be switched from a first preset value to a second preset value in a very short time.
For this purpose, electroluminescent or electrochromatic flat elements are particularly suitable, as they are already known from the related art.
Further advantages and embodiments of the present invention result from the description and the attached drawing.
It is obvious that the features previously mentioned and described in the following are usable not only in the respective combination indicated, but also in other combinations or alone, without leaving the framework of the present invention.

WO 01/06766 ~ PCT/EP00/06827 The present invention is schematically illustrated in the drawing with reference to an exemplary embodiment and is described in detail in the following with reference to the drawing.
Fig. 1 shows a block diagram to illustrate the present invention.
Fig. 2 shows a block diagram to illustrate an evaluation unit to be used in the present invention.
The central component of the invention is the recording of one or more objects (objects 1 to m) by one or more cameras (cameras 1 to n), the objects being located in front of a background. The background is, according to the present invention, influenceable and/or alterable at a high frequency. The alteration frequency is tuned to a recording frequency of the recording cameras) in this case. The alteration of the background particularly occurs in regard to its color and/or brightness and/or saturation.
The influenceable parameters of the background are thus, besides the alteration frequency, the color, the brightness, and/or the saturation. In the camera(s), the influenceable parameter is, besides the typical camera parameters, the recording frequency, possibly restricted to a definable image region x*y. These parameters are controlled by the evaluation unit, with direct control, particularly for adjustment of the cameras) to the background, also being possible.
Objects 1 to m included by the background, i.e., surrounded by the background, are recorded with cameras 1 to n, the recording frequency of the cameras being a multiple of the later playback frequency. As a result, Az camera images from n cameras are provided per time unit z, which are supplied to the evaluation unit illustrated in more detail in Fig. 2.

' WO 01/06766 ~ PCT/EP00/06827 The evaluation unit illustrated in Fig. 2 is, for each of the 1 to n cameras, its own camera-specific evaluation unit is provided, which are each supplied the data of the Az camera images. In consideration of the additional parameters, as they were described previously, also supplied to each camera-specific evaluation unit, in each camera-specific evaluation unit the objects) is/are separated from a background, in that an identification and an assignment of individual camera image elements (pixels or pixel regions) to "object" or "background"
occurs. After the assignment is performed, the image elements assigned to the background are removed and replaced by image elements of a replacement background. In simple applications, this replacement background is a static, unchanging image, for example the weather map. However, more complex applications are also possible and are made easier by the present invention, such as the overlay of separately recorded images or moving images generated by a computer as a background during studio recordings.
The cycle of the identification was already described above.
The image data produced by the individual camera-specific evaluation units is then supplied to a central recording-instant-specific evaluation unit, which includes a buffer for evaluating multiple time units and is used for synchronization and tuning of the data supplied by the camera-specific evaluation units. The usage of multiple cameras allows the generation of image data which is suitable for playback observable by an observer from different perspectives as "three-dimensional." A recording and playback method of this type is, for example, described in German Patent Application 199 13 853.2.
The data produced by the central recording-instant-specific evaluation unit is output from the evaluation unit, so that a single image (frame) is present as a result (cf. Fig. 1), in which the background and the object are separated from one another. A typical playback frequency of the frame produced is approximately 25 to 30 sec-1 (television image frequency).

Claims (15)

Claims:
1. A method for image data processing, wherein object data of an image data set (object data set) representing an image foreground is separated from a data set (background data set) representing an image background, wherein a background to be recorded by the camera is altered at a frequency tuned to a recording frequency of the recording camera.
2. The method according to Claim 1, wherein the alteration of the background occurs by switching between two states.
3. The method according to Claim 1, wherein the alteration of the background occurs gradually between two states with multiple intermediate states.
4. The method according to one of Claims 1 to 3, wherein the color and/or brightness and/or saturation of the background is altered.
5. The method according to one of Claims 1 to 4, wherein the recording frequency of the camera is a multiple of the playback frequency.
6. The method according to Claim 5, wherein the recording frequency is double the typical television image frequency.
7. The method according to one of Claims 1 to 6, wherein a comparison of the data sets of camera images recorded at a different instant within a replay frequency is performed to establish the assignment of an element of a camera image data set to the object data set or the background data set.
8. The method according to Claim 7, wherein the comparison includes a subtraction of elements of identical coordinates or regions of the camera image data sets.
9. The method according to Claim 7 or 8, wherein the elements of the camera image data sets compared with one another are pixels.
10. The method according to Claim 8 or 9, wherein the elements of the camera image data sets compared with one another are pixel regions.
11. The method according to one of Claims 1 to 6, wherein methods of artificial intelligence, such as neuronal nets, are used for recognition and assignment of the elements of the camera image data sets.
12. The method according to one of Claims 6 to 11, wherein the elements of the camera image data set identified as associated with the background data set are replaced by elements of corresponding coordinates or regions of a data set of a replacement background.
13. A camera recording system, particularly for performing the method according to one of Claims 1 to 12, having at least one camera for recording an object located in front of a background, with an alteration of the background occurring at a frequency tuned to a recording frequency of the recording camera.
14. The camera recording system according to Claim 13, wherein a film-like or sheet-like element influenceable in an optically defined way is provided as a background.
15. The camera recording system according to Claim 14, wherein the background element is an electroluminescent or electrochromatic flat element.
CA002386250A 1999-07-15 2000-07-17 Method for image data processing and camera recording system Abandoned CA2386250A1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
DE19932663 1999-07-15
DE19932663.0 1999-07-15
PCT/EP2000/006827 WO2001006766A1 (en) 1999-07-15 2000-07-17 Method for image data processing and camera recording system

Publications (1)

Publication Number Publication Date
CA2386250A1 true CA2386250A1 (en) 2001-01-25

Family

ID=7914585

Family Applications (1)

Application Number Title Priority Date Filing Date
CA002386250A Abandoned CA2386250A1 (en) 1999-07-15 2000-07-17 Method for image data processing and camera recording system

Country Status (7)

Country Link
EP (1) EP1197069B1 (en)
CN (1) CN1372761A (en)
AT (1) ATE256363T1 (en)
CA (1) CA2386250A1 (en)
DE (1) DE50004739D1 (en)
IL (1) IL147606A0 (en)
WO (1) WO2001006766A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9256793B2 (en) 2008-09-05 2016-02-09 Fujitsu Limited Apparatus and method for extracting object image
US10313607B2 (en) 2010-09-20 2019-06-04 Fraunhofer-Gesellschaft-Zur Foerderung Der Angewandten Forschung E.V. Method for differentiating between background and foreground of scenery and also method for replacing a background in images of a scenery

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101094768B1 (en) * 2004-06-15 2011-12-16 엘지전자 주식회사 Methods and a apparatus of making background photo on sticker photo for mobile phone
DE102021215048A1 (en) 2021-12-27 2023-06-29 Friedrich-Schiller-Universität Jena Arrangement and method for distinguishing between a background and a foreground object of a scene

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2321814B (en) * 1997-01-28 2001-02-14 British Broadcasting Corp Video imaging
US6191812B1 (en) * 1997-04-01 2001-02-20 Rt-Set Ltd. Method of providing background patterns for camera tracking
JP3241327B2 (en) * 1998-08-22 2001-12-25 大聖電機有限会社 Chroma key system

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9256793B2 (en) 2008-09-05 2016-02-09 Fujitsu Limited Apparatus and method for extracting object image
US10313607B2 (en) 2010-09-20 2019-06-04 Fraunhofer-Gesellschaft-Zur Foerderung Der Angewandten Forschung E.V. Method for differentiating between background and foreground of scenery and also method for replacing a background in images of a scenery
US11165973B2 (en) 2010-09-20 2021-11-02 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Method for differentiating between background and foreground of scenery and also method for replacing a background in images of a scenery
US11950014B2 (en) 2010-09-20 2024-04-02 Fraunhofer-Gesellschaft Zur Foerderungder Angewandten Forschung E.V Method for differentiating between background and foreground of scenery and also method for replacing a background in images of a scenery

Also Published As

Publication number Publication date
CN1372761A (en) 2002-10-02
ATE256363T1 (en) 2003-12-15
IL147606A0 (en) 2002-08-14
DE50004739D1 (en) 2004-01-22
EP1197069B1 (en) 2003-12-10
WO2001006766A1 (en) 2001-01-25
EP1197069A1 (en) 2002-04-17

Similar Documents

Publication Publication Date Title
JP6749953B2 (en) Displaying DCI and other content on extended dynamic range projectors
CA2284032C (en) Real-time method of digitally altering a video data stream to remove portions of the original image and substitute elements to create a new image
CN103891294B (en) The apparatus and method coded and decoded for HDR image
CN106782429B (en) Eliminate the method and device of screen inhomogeneities
Ancuti et al. Enhancing by saliency-guided decolorization
EP0868089B1 (en) An image processing system
CN101273624A (en) Compensating for improperly exposed areas in digital images
CN101523990A (en) Method for color transition for ambient or general illumination system
US9100530B2 (en) Image processing apparatus, image processing method and storage medium
RU2611599C1 (en) Image colour adjustment device, image display device and chromaticity control method
KR20070090224A (en) Method of electronic color image saturation processing
EP1720127B1 (en) Image creation device and image creation method for yarn images
CN110333924A (en) A kind of image morphing method of adjustment, device, equipment and storage medium
JP6771977B2 (en) Image processing equipment and image processing methods, programs
EP1590955A2 (en) Method and apparatus for combining video signals to one comprehensive video signal
WO2023097109A1 (en) Increasing resolution and luminance of a display
JPH0350685A (en) Method of automatically adjusting luminance and contrast and image fetching system
CA2386250A1 (en) Method for image data processing and camera recording system
JPH0721367A (en) Picture recognition device
DE102007041719A1 (en) Augmented reality producing method for use in e.g. TV studio, involves synchronizing camera with video projectors, and modulating images in luminance and/or chrominance in pixelwise, and integrating coded structure into projected images
EP0268360A2 (en) System for processing video image signals
CN109040598A (en) Image processing method, device, computer readable storage medium and electronic equipment
KR100661528B1 (en) Adjustive chroma key composition apparatus and method
Kartalov et al. Fully automated exposure fusion algorithm for mobile platforms
Reiter With j: image processing 2: color spaces

Legal Events

Date Code Title Description
FZDE Discontinued