WO2002035843A1 - Commande de focalisation de mouvement video - Google Patents

Commande de focalisation de mouvement video Download PDF

Info

Publication number
WO2002035843A1
WO2002035843A1 PCT/US2001/042808 US0142808W WO0235843A1 WO 2002035843 A1 WO2002035843 A1 WO 2002035843A1 US 0142808 W US0142808 W US 0142808W WO 0235843 A1 WO0235843 A1 WO 0235843A1
Authority
WO
WIPO (PCT)
Prior art keywords
motion
stationary
recited
video
image components
Prior art date
Application number
PCT/US2001/042808
Other languages
English (en)
Inventor
Steven D. Edelson
Original Assignee
Dynapel Systems, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dynapel Systems, Inc. filed Critical Dynapel Systems, Inc.
Publication of WO2002035843A1 publication Critical patent/WO2002035843A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/144—Movement detection
    • H04N5/145—Movement estimation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Definitions

  • This invention relates to the processing of video data and more particularly to a system wherein the video data is processed to show objects in motion in focus and to show stationary objects or background unfocussed.
  • the lens aperture and focal length can be chosen to focus on the subject of the photograph with the remainder of the photograph shown unfocussed. This technique is used particularly in portrait photography. It also is desirable to be able to show the subject of a motion picture in sharp focus while leaving the background or unimportant details in the motion picture unfocussed in order to draw attention to the subject of the motion picture.
  • this effect is not easily accomplished with the focal length and aperture of a camera lens because the position of the subj ect of a motion picture often changes from frame to frame and sometimes there will be more than one object in a motion picture to which it is desired to draw the attention of the viewer. Accordingly, there is a need for a system for processing video data which will automatically make the subject or subjects of a motion picture be in focus while defocusing the remainder of the motion picture.
  • a system which processes the video data to automatically show in focus the objects depicted to be in motion while showing stationary objects or slowly moving objects or other image components to be unfocussed.
  • the subject will be in motion. Accordingly, by processing the video data to automatically show the objects in motion to be in focus while showing the stationary or slowly moving objects and background to be defocused, a system is achieved which automatically focuses the subject of the motion picture while defocusing the other parts of the depicted scene.
  • the video data representing the motion picture is processed to generate dense motion vector fields which represent the motion of pixel sized image elements from frame to frame.
  • image element refers to pixel sized bits of objects, scenery or background depicted in the motion picture. When the depicted objects are shown as moving, the image elements move with the objects. Thus, image elements are distinct from pixels, which are stationary with respect to a display screen on which a motion picture is displayed.
  • the vectors of a dense motion field represent the direction and magnitude of the motion of the image elements from frame to frame. In a dense motion vector field, a vector will be generated for each pixel and will indicate the motion of the corresponding image element from frame to frame in the motion picture.
  • the pixel data representing the pixels in the frames are processed by a filter whereby image elements identified by the dense motion vector fields as being more rapidly moving are shown in focus and more slowly moving objects or stationary objects are shown out of focus.
  • the pixels corresponding to stationary or slowly moving image elements are defocused by replacing the pixels with averages of the pixels around the pixels being defocused.
  • the averages are weighted and the weights of the weighted averages are in accordance with a gaussian distribution. In this manner stationary objects and/or background and slowly moving objects or background are defocused whereas objects having substantial motion remain in focus.
  • Figure 1 is a block diagram of the system of the present invention.
  • Figure 2 is a flowchart illustrating the process performed by the video processor of Figure 1 to adjust the focus in accordance with motion vector magnitudes.
  • the system of the invention comprises a video source 11 which provides video frames in sequence, wherein the pixels of each frame are represented in digital form.
  • the video frames from source 11 are stored in storage device 13 by a video processor 15.
  • the video processor 15 is programmed to process the video data in the storage device 13 and is also capable of transmitting the video stored in the storage device 13 to the video display 17 to display the stored video.
  • the flowchart of Figure 2 illustrates a process carried out by the video processor 15 on the video stored in the storage device 13 in accordance with the invention.
  • the processor 15 generates dense motion vector fields representing the motion or change in position of image elements from frame to frame in the video.
  • a system for generating the dense vector motion vector fields is disclosed in the copending application serial number 9/593,521 filed June 14, 2000 entitled "System for the Estimation of Optical Flow” invented by Seigfried Wonneberger, Max Griessl and Marcus Wittcop.
  • the result of the process of generating the dense motion vector fields will be sets of vectors, one set for each pair of adjacent frames in the video and each set comprising a dense motion vector field.
  • each vector represents the change in position of a pixel sized image element from frame to frame.
  • the image elements represent small pixel sized pieces of depicted objects or background. If an object is moving as represented in adjacent video frames, then the image elements of such object will move with the object. Thus, each vector of a dense motion field will represent the motion of a corresponding image element.
  • the midpoints of the motion vectors are aligned with the pixels, this means that the origins of the vectors and termination points of the vectors will not be normally aligned with the pixels.
  • the video processor subjects the video data to a gaussian filter in dependence upon the vectors of the dense motion field to defocus or blur the stationary background and stationary or slowly moving objects while leaving the objects moving relatively rapidly in sharp focus.
  • the vectors of the dense motion field are each compared with a standard vector corresponding to a selected vector magnitude. If a motion vector magnitude exceeds the standard, then the pixels at or adjacent to the termination point of this motion vector are left unmodified.
  • each pixel at or adjacent to the termination point of the vector is replaced with a weighted average of the pixel and surrounding pixels in accordance with a selected gaussian distribution, wherein the pixels closest to the termination point receive the greatest weights and the pixels at the greatest distance from the termination point receive the lowest weights.
  • image components means objects, including background or scenery, depicted in the motion picture. It should be understood that motion vectors associated with image elements of image components which are truly stationary will have zero magnitude, which means that there are no motion vectors associated with these image elements.
  • the pixels corresponding to these image elements are also subjected to the gaussian filter to defocus them since the zero magnitude motion vectors are less than the selected predetermined vector magnitude.
  • the vector magnitude below which the corresponding pixels are defocused may be selected by the user.
  • the magnitude is selected so that only stationary or substantially stationary objects are defocused.
  • at least the pixels corresponding to stationary objects are defocused and preferably at least the pixels corresponding to pixels which are substantially stationary are defocused. If the video camera making the motion picture is panned, there may be no stationary image elements. If the panning is slower than the selected vector magnitude, the resulting moving background will be defocused.
  • stationary objects and background and slowly moving objects and background are all defocused to the same degree.
  • the amount of defocusing is adjusted in accordance with the magnitude of the motion vector.
  • the dimensions of the gaussian distribution are adjusted inversely in accordance with the vector magnitude wherein motion vectors with zero magnitudes will invoke gaussian distribution with the widest dimension so that stationary objects and background are shown with the greatest amount of blurring or defocusing.
  • the dimensions of the gaussian distribution are scaled inversely from this widest dimension with vector magnitude so that faster moving objects are shown in better focus and slower moving objects are shown with a greater degree of defocusing or blur.
  • the variation of the dimension of the gaussian distribution can be linear with respect to motion vector or can correspond with any nonlinear or arbitrary function.
  • the defocusing takes place at the termination points of the motion vectors.
  • the defocusing may take place at the origins of the vectors or anywhere along the length of the vectors. Since the motion vectors at which defocusing occurs are short, representing slowly moving image elements, pixels adjacent to the vectors anywhere along the length of the vectors may be defocused with substantially the same effect as defocusing the pixels at the termination points of the vectors. Also the focusing-defocusing effects may be made persistent so that the formerly moving objects remain focused after the objects stop moving. As a , result, attention will be drawn to a stationary or slowly moving object which was previously moving by the object being focused in contrast to other stationary or slowly moving background and objects, which are defocused.
  • the persistence may be made infinite or until reset by the viewer so that objects which had been moving and which are currently substantially stationary remain focused.
  • This feature has application in surveillance video cameras to draw attention to objects which have moved and which are now stationary in the surveillance field.
  • the cessation of defocusing may be made anticipatory of when an object starts moving so that an object which is about to begin moving rapidly becomes focused before it begins moving rapidly.
  • the movement of the image elements may be tracked through the number of frames over which the persistence or anticipation occurs.
  • the persistence or anticipation of the defocusing effect may be applied merely to the same pixels in the preceding or succeeding frames corresponding to the image elements representing the moving or non-moving objects.
  • the video data is defocused with a low pass gaussian filter.
  • a low pass gaussian filter may be used.
  • a high pass filter could be used to sharpen the focus of more rapidly moving objects relative to the slower moving objects and background.
  • the focus or defocus is controlled in accordance with the motion vector magnitude.
  • the focus or defocus may be controlled in accordance with vector direction.
  • pixels corresponding image elements corresponding to vertical vectors could be processed with a high-pass filter while not sharpening the focus horizontally moving objects to draw attention to the vertically moving objects.
  • the stationary and horizontally moving objects could be defocused leaving the vertical moving objects in focus. This feature would have application, for example, in a material or product handling system to draw attention to objects falling off of the moving belt.
  • the background is assumed to be stationary or at least slowly moving so that moving objects are shown more in focus than the background.
  • the background will move when the video camera moves as when the video camera is panned.
  • the motion of image elements due to camera motion can be detected and zeroed out so that the vector field is a net motion relative to the image component motion due to camera motion.
  • the net motion vectors are then used to control the focus as described above. Accordingly, when the camera is moved so that it follows a moving object, the object still would be shown more in focus than the background, even though the image elements of the moving object relative to the video frame are moving more slowly than the image elements of the background.
  • moving objects are shown with a sharp focus and stationary and slowly moving objects are shown blurred or defocused state in an effect achieved automatically. Accordingly, an automatic system is provided which enables the moving objects, which typically will be the subject of the motion picture, to be shown in focus relative to the unfocussed background and slowly moving objects and thus draws the attention of the viewer to the moving objects.
  • the moving objects could be defocused relative to the background or more slowly moving objects, which would be shown in focus. Since the averaging to achieve defocusing is omni-directional in the plane of the viewing screen, the defocusing effect as applied to moving objects is different then the motion blur which occurs naturally as result of motion and which is caused by the delay in the response of the photosensitive elements of the video camera. This motion blur is unidirectional parallel to the direction of motion. The omni-directional defocusing achieved the invention is visually a different effect than the unidirectional motion blur.
  • This embodiment of the invention has application in cases in which it is desired to de- emphasize moving objects and to emphasize stationary objects such as a building in a real estate marketing video.
  • the invention also has application in advertising and other videos to change the focus of rapidly moving object relative to the focus of slowly moving objects as a special effect.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)
  • Studio Circuits (AREA)

Abstract

L'invention concerne un système de commande de focalisation d'un film dans lequel des données vidéo sont automatiquement traitées pour défocaliser des composants des images présentées d'un film cinématographique, conformément au mouvement représenté des composants des images. Des composants d'images fixes et/ou défilant au ralenti sont défocalisés dans les données vidéo traitées, et des composants d'images défilant plus rapidement sont focalisés, ou vice versa. Des données vidéo représentant un film cinématographique sont traitées afin de produire des champs de vecteurs de mouvement dense représentant le mouvement d'éléments de la taille du pixel. Après production du champ de vecteurs de mouvement dense, les pixels associés aux vecteurs de mouvement ayant une magnitude inférieure à une magnitude de vecteur préétablie choisie ou nulle représentant les objets fixes ou défilant au ralenti, sont traités au moyen d'un filtre gaussien afin de défocaliser l'élément d'image. La défocalisation des pixels correspondants aux éléments d'image fixes ou défilant au ralenti est effectuée par remplacement des pixels en cours de défocalisation, les poids des moyennes pondérales étant conformes à une distribution gaussienne.
PCT/US2001/042808 2000-10-25 2001-10-25 Commande de focalisation de mouvement video WO2002035843A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US69504800A 2000-10-25 2000-10-25
US09/695,048 2000-10-25

Publications (1)

Publication Number Publication Date
WO2002035843A1 true WO2002035843A1 (fr) 2002-05-02

Family

ID=24791340

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2001/042808 WO2002035843A1 (fr) 2000-10-25 2001-10-25 Commande de focalisation de mouvement video

Country Status (1)

Country Link
WO (1) WO2002035843A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5790206A (en) * 1994-09-02 1998-08-04 David Sarnoff Research Center, Inc. Method and apparatus for global-to-local block motion estimation
US6072525A (en) * 1994-07-18 2000-06-06 Canon Kabushiki Kaisha Image pickup apparatus effecting object image tracking responsively to object image frame movement and object image movement
US6275617B1 (en) * 1993-07-12 2001-08-14 Sony Corporation Method and apparatus for separating/generating background and motion object planes

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6275617B1 (en) * 1993-07-12 2001-08-14 Sony Corporation Method and apparatus for separating/generating background and motion object planes
US6072525A (en) * 1994-07-18 2000-06-06 Canon Kabushiki Kaisha Image pickup apparatus effecting object image tracking responsively to object image frame movement and object image movement
US5790206A (en) * 1994-09-02 1998-08-04 David Sarnoff Research Center, Inc. Method and apparatus for global-to-local block motion estimation

Similar Documents

Publication Publication Date Title
US7911513B2 (en) Simulating short depth of field to maximize privacy in videotelephony
Nagahara et al. Flexible depth of field photography
US8023000B2 (en) Image pickup apparatus, image processing apparatus, image pickup method, and image processing method
US7092016B2 (en) Method and system for motion image digital processing
CN106911889B (zh) 图像模糊校正设备和倾斜校正设备及其控制方法
US7812880B2 (en) Optical apparatus
US8125542B2 (en) Imaging device using a first motion vector in an image that is obtained via an imaging system that can perform zooming and a second motion vector caused by performing zooming of the image
CN105635559B (zh) 用于终端的拍照控制方法及装置
JP2008271240A5 (fr)
JP3251127B2 (ja) 映像データ処理方式
JP2011166588A (ja) 撮像装置および撮像方法、ならびに前記撮像装置のための画像処理方法
JP2006343496A (ja) 自動合焦装置
TW202305739A (zh) 用於改良的場景完整性的時間濾波重新啟動
JP2957800B2 (ja) 自動焦点調節装置
Ueno et al. Motion-blur-free microscopic video shooting based on frame-by-frame intermittent tracking
TWI524757B (zh) 影像擷取裝置及數位變焦顯示方法
EP0956695A1 (fr) Systeme et procede pour regler dynamiquement la vitesse d'obturation sur la base du mouvement de la scene
JP2015154334A (ja) 撮像装置、その制御方法、および制御プログラム
WO2002035843A1 (fr) Commande de focalisation de mouvement video
Sonoda et al. Motion-invariant coding using a programmable aperture camera
US8994846B2 (en) Image processing apparatus and image processing method for detecting displacement between images having different in-focus positions
WO2022055424A3 (fr) Procédé et appareil de photographie vidéo, dispositif, et support de stockage
Khoenkaw et al. Automatic pan-and-scan algorithm for heterogeneous displays
JPH07107368A (ja) 画像処理装置
JP6016546B2 (ja) 撮像装置、その制御方法、および制御プログラム

Legal Events

Date Code Title Description
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
121 Ep: the epo has been informed by wipo that ep was designated in this application
122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP