WO2011065960A1 - Stabilizing a subject of interest in captured video - Google Patents
Stabilizing a subject of interest in captured video Download PDFInfo
- Publication number
- WO2011065960A1 WO2011065960A1 PCT/US2009/066139 US2009066139W WO2011065960A1 WO 2011065960 A1 WO2011065960 A1 WO 2011065960A1 US 2009066139 W US2009066139 W US 2009066139W WO 2011065960 A1 WO2011065960 A1 WO 2011065960A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- subject
- interest
- window
- bounding
- crop
- Prior art date
Links
- 230000000087 stabilizing effect Effects 0.000 title claims abstract description 11
- 238000000034 method Methods 0.000 claims abstract description 27
- 239000002131 composite material Substances 0.000 claims description 6
- 238000013459 approach Methods 0.000 claims description 2
- 238000001514 detection method Methods 0.000 description 23
- 238000003384 imaging method Methods 0.000 description 6
- 238000003708 edge detection Methods 0.000 description 4
- 230000008921 facial expression Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000006641 stabilisation Effects 0.000 description 2
- 238000011105 stabilization Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000010348 incorporation Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/61—Control of cameras or camera modules based on recognised objects
- H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
- H04N23/6811—Motion detection based on the image signal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
- H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
Definitions
- the camera may place the subject of interest at the center of the video frame. As the subject moves within the frame, the camera may reorient the video frame to maintain the subject of interest at or near the center of the frame. Face detection may also be used to identify the subject of interest.
- Figure 1 represents a subject of interest moving in a subject bounding window within a crop window according to an embodiment of the invention.
- Figure 2 shows a subject bounding window moving aiong with the crop window of Figure 1 in response to the subject of interest moving outside of subject bounding window of Figure 1 according to an embodiment of the invention.
- Figure 3 shows the movement of the background of the captured image to place the subject of Interest nearer to the center of the crop window according to an embodiment of the invention.
- Figure 4 shows a subject of interest that has moved outside of a subject bounding window according to an embodiment of the invention.
- Figure 5 shows a composite of two subjects within a subject bounding window according to an embodiment of the invention.
- Figure 6 is a method of stabilizing a subject of interest in captured video according to an embodiment of the invention.
- Figure 7 is a method of stabilizing a subject of interest in captured video according to another embodiment of the invention.
- Figure 8 is a block diagram for a module used to stabilize a subject of interest in captured video according to embodiments of the invention.
- Stabilizing a subject of interest in captured video allows a video camera user to record natural-looking video of the subject of interest as the subject moves within a crop window
- the subject bounding window remains unchanged and the camera records the user moving within the subject bounding window against a stable background.
- the window is moved and the current location of the subject of interest is placed near the center of the new subject bounding window. The new window is then held steady as the subject of interest moves within the new window.
- Embodiments of the invention differ from conventional solutions at least in that the subject bounding window is moved only when the subject of interest exits the bounding window.
- the video camera constantly re-centers the subject of interest.
- the recorded video appears shaky and unnatural as the subject is forced towards the center of the frame, often several times within a few seconds.
- Figure 1 represents a subject of interest moving in a subject bounding window within a crop window according to an embodiment of the invention.
- full-resolution window 20 surrounds crop window 15. Within crop window 15, subject of interest 30 moves inside of subject bounding window 10.
- full-resolution window 20 represents the largest area that can be captured by the imaging array of the camera recording the scene.
- Vector 50 indicates the two-dimensional motion of subject of interest 30 from right to left. In this embodiment of the invention, so long as subject of interest 30 stays within subject bounding window 10, the bounding window does not move within crop window 15, Consequently, background 40 is held steady.
- subject bounding window 10 is not visible to the user of the camera capturing the scene depicted in Figure 1.
- Subject bounding window 10 represents a bounding area used by embodiments of the invention to enable an algorithm to function without requiring the user to exercise direct control over the window, in contrast to window 10, crop window 15 is displayed to the user in at ieast some embodiments of the invention.
- subject bounding window 10 and crop window 15 move together as will be seen in reference to Figure 2.
- An exemplary detection algorithm is provided in US patent number 7,099,510 entitled “Method and System for Object Detection in Digital images” or perhaps as provided in numerous other patent applications and issued US and foreign patents.
- subject bounding window 10 is positioned around the subject of interest.
- Figure 2 shows a subject bounding window moving along with the crop window of Figure 1 in response to the subject of interest moving outside of subject bounding window of Figure 1 according to an embodiment of the invention, in Figure 2, subject of interest 30 has moved far enough to the left so as to cause subject bounding window 10 and crop window 15 to be repositioned to the left.
- subject bounding window 10 is moved if (and only if) the subject of interest has moved outside of the original subject bounding window, shown in Figure 1.
- full-resolution window 20 is shown as encompassing perhaps twice the area of crop window 15.
- crop window 15 is shown as encompassing perhaps twice the area of subject bounding window 10.
- full- resolution window 20 may encompass many times the area as crop window 15, such as 15 or 20 times the area of the crop window, or maybe even larger.
- crop window 15 may be many, many times larger than subject bounding window 10.
- a full-resolution window that is much larger than the crop window may allow the captured images to remain steady while the crop window moves relatively freely to accommodate the motion of the subject of interest and the subject bounding window.
- subject bounding window 10 may be shaped as an oval, whiie crop window 15 is shaped as a square. These shapes may be used within a rectangular-shaped full resolution window 20.
- Figure 3 shows the movement of the background of the captured image to place the subject of interest nearer to the center of the crop window according to an embodiment of the invention.
- subject of interest 30 has moved toward the tree shown at the right hand side of subject bounding window 10. in this embodiment of the invention, it is desirable to place the subject of interest 30 at the center of subject bounding window 10. Accordingly, subject bounding window 10 and crop window 15 have been repositioned to the right.
- Figure 4 shows subject of interest 30 having moved outside of subject bounding window 10 according to an embodiment of the invention. As the subject of interest is no longer within subject bounding window 10, a new subject of interest (35) has been identified. In this embodiment of the invention, when subject of interest 30 moves to a position that cannot be compensated for, subject bounding window 10 is slowly and gracefully re-centered on new subject of interest 35.
- Figure 5 shows a composite of two subjects within subject bounding window 10 according to an embodiment of the invention
- centroid 37 represents a spatially averaged location between the two subjects of interest.
- the location of centroid 37 is tracked and the subject bounding window relocated based on the location and the two-dimensional motion vector of the centroid.
- subject bounding window 10 has been shown as being fixed at or near the center of crop window 15. However, in other embodiments of the invention, subject bounding window 10 may be allowed to move more freely within crop window 15 and may not be required to occupy a center portion of the crop window.
- Figure 6 is a method of stabilizing a subject of interest in captured video according to an embodiment of the Invention.
- the method of Figure 8 begins at step 100 in which a subject of interest is identified within the crop window.
- the identification of the subject of interest includes using a saliency algorithm such as face detection, edge detection, facial expression detection, skin tone detection, and so forth to detect a single human face (such as the face of subject of interest 30) within a full-resolution window
- the subject of interest is a composite of two or more subjects (represented by centroid 37 ⁇ that indicates the iocation of two or more human faces whose location has been averaged in the subject bounding window.
- step 1 10 includes establishing a first subject bounding window that includes the subject of interest.
- the subject bounding window is formed inside the crop window.
- step 120 inciudes computing a two-dimensional motion vector that characterizes the movement of the subject of interest.
- the two- dimensional motion vector may be used to determine the direction that subject bounding window 10 should move in order to best accommodate subject of interest 30 when the subject of interest exits window 10.
- Step 130 when the subject of interest exits the subject bounding window, the camera establishes a second subject bounding window that inciudes the new location of the subject of interest.
- the camera maintains the subject of interest within the second subject bounding window so long as the subject does not move outside of the second subject bounding window.
- Step 130 may also include centering the subject of interest within the within the second subject bounding window.
- Other embodiments of the invention may also include the step of identifying a second subject of interest when a first subject of interest approaches and edge of a full resolution window that encompasses the crop window,
- Figure 7 is a method of stabilizing a subject of interest in captured video according to another embodiment of the invention.
- the method of Figure 7 begins at step 200, which includes identifying the subject of interest in the
- the identification of the subject of interest in the successive frames of the video may occur without user input, perhaps as a result of the video camera identifying one or more human faces or other salient features of a subject of interest within the successive frames of the video using a suitable face or other type of detection algorithm.
- the identification in step 200 may also occur by way of receiving a user input that designates the subject of interest in the successive frames of the video.
- a user input designates the subject of interest in the successive frames of the video.
- touchscreen that displays the video frame to the user may receive an input from the user that designates the subject of interest
- the user may select the pet on a display of the video camera.
- the camera may establish a subject bounding window and holds the subject bounding window steady while the pet moves within the window.
- step 210 a first subject bounding window that encompasses the identified subject of interest is established.
- the camera holds steady the background within the first crop window as the identified subject of interest moves within the first subject bounding window.
- step 230 a second crop window is established when the identified subject of interest exits the first subject bounding window.
- the method of Figure 7 can aiso be performed on a video stream stored on a storage media located outside of a video camera.
- a processor stores successive frames of the video in a memory.
- a saiiency algorithm such as face detection, edge detection, facial expression detection, skin tone detection, and so forth, and/or perhaps with the assistance of a user, the subject of interest is identified within the successive frames of the video (as in step 200 of Figure 7).
- the processor establishes a first subject bounding that encompasses the identified subject of interest.
- FIG. 8 is a block diagram for a module used to stabilize a subject of interest in captured video according to embodiments of the invention. Although module 301 of f igure 8 can be used to perform the method of Figures 6 and 7, nothing prevents the use of alternately-configured hardware, software, or firmware modules to perform the methods.
- lens 310 focuses incoming light from scene 300 onto imaging array 320.
- Imaging array 320 includes a CCD or CMOS imaging array that converts the incident optical signals that represent scene 300 into discrete electrical charges.
- the electrical charges are processed by image processing module 330, which applies correction factors to compensate for vignetting, color dependent shading across imaging array 320, channel balancing, and conversion from the raw outputs of the imaging array to a standard color space, such as RGB, sRGB, and so forth.
- the output of each frame of image processing module 330 is then mapped to memory array 340.
- processor 360 operates in conjunction with subject of interest detection module 365 to automatically (in which "automatically” implies “without user input") detect the subject present in scene 300.
- Subject of interest detection module 365 may include a saSiency algorithm such as face detection, edge detection, facial expression detection, skin tone detection and so forth to determine the presence of a subject of interest.
- processor 360 identifies the subject of interest within scene 300 and establishes a subject bounding window that encompasses the subject of interest along with at least a portion of the background of the scene. When the subject of interest exits the subject bounding window, processor 300 determines the new location of the subject of interest and establishes a subject bounding window within memory array 340 around the new location of the subject of interest.
- processor 360 operating in conjunction with subject of interest detection module 385, averages the location of more than one face detected in scene 300 and identifies a composite as representing the two or more subjects.
- processor 360 and subject of interest detection module 365 establish a new subject bounding window within memory array 340, Within the second subject bounding window, the subject of interest may move about while processor 360 holds the second subject bounding window steady. The successive frames of the subject of interest and the background within the subject bounding windows are then stored within storage media 370,
- the user may interact with touchscreen 350 and/or user interface 355 to select the subject of interest, in this embodiment of the invention, the user may surround a shape of the subject of interest using a stylus or his/her finger to directly interact with the touchscreen.
- a subject bounding window may be established around the subject of interest
- processor 360 establishes a second subject bounding window around the new location of the subject of interest
- memory array 340 stores a previousiy captured succession of video frames to which the methods of the claimed invention can be applied.
- processor 360 operating in conjunction with subject of interest detection module 365, may make use of an algorithm to detect the presence of one or more subjects of interest within the video stream.
- Processor 380 may establish and hold steady the background of a subject bounding window within the video stream in a manner that aiiows the subject of interest to move within the subject bounding window.
- processor 360 and subject of interest detection module 365 operate to reorient the subject bounding window to include a new portion of memory array 340 in which the subject of interest is re-centered.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
Abstract
A method of stabilizing a subject of interest in captured video includes identifying the subject of interest within a crop window and establishing a first subject bounding window that includes the subject of interest. The method continues with refraining from substantially changing the first subject bounding window while the subject of interest moves within the first subject bounding window and establishing a second subject bounding window within the crop window that includes the subject of interest when the subject of interest exits the first subject bounding window.
Description
Stabilizing a Subject of interest in Captured Video Background
[001] When using a digital video camera that includes an image stabilization feature, the camera may place the subject of interest at the center of the video frame. As the subject moves within the frame, the camera may reorient the video frame to maintain the subject of interest at or near the center of the frame. Face detection may also be used to identify the subject of interest.
[002| However, when the subject of interest moves within the camera's field of view, an image stabilization algorithm operating within the camera may constantly attempt to re-center the subject. When this happens, the recorded video takes on a very unnatural quality. In many instances, the recorded video appears jerky as the subject is continually (perhaps numerous times within just a few seconds) forced fay the camera to the center of the video frame as the subject moves within the frame.
Brief Description of the Drawings
[003] Figure 1 represents a subject of interest moving in a subject bounding window within a crop window according to an embodiment of the invention.
[004] Figure 2 shows a subject bounding window moving aiong with the crop window of Figure 1 in response to the subject of interest moving outside of subject bounding window of Figure 1 according to an embodiment of the invention.
[005] Figure 3 shows the movement of the background of the captured image to place the subject of Interest nearer to the center of the crop window according to an embodiment of the invention.
[006] Figure 4 shows a subject of interest that has moved outside of a subject bounding window according to an embodiment of the invention.
[007] Figure 5 shows a composite of two subjects within a subject bounding window according to an embodiment of the invention.
[008] Figure 6 is a method of stabilizing a subject of interest in captured video according to an embodiment of the invention.
[009] Figure 7 is a method of stabilizing a subject of interest in captured video according to another embodiment of the invention.
[0010] Figure 8 is a block diagram for a module used to stabilize a subject of interest in captured video according to embodiments of the invention.
Description of the Embodiments
[0011] Stabilizing a subject of interest in captured video allows a video camera user to record natural-looking video of the subject of interest as the subject moves within a crop window, in one embodiment of the invention, when the subject moves within a subject bounding window located within a crop window, the subject bounding window remains unchanged and the camera records the user moving within the subject bounding window against a stable background. When the subject of interest moves outside of the subject bounding window, the window is moved and the current location of the subject of interest is placed near the center of the new subject bounding window. The new window is then held steady as the subject of interest moves within the new window.
[0012] Embodiments of the invention differ from conventional solutions at least in that the subject bounding window is moved only when the subject of interest exits the bounding window, In conventional solutions, as the subject moves within the field of view, the video camera constantly re-centers the subject of interest. As a result, the recorded video (especially the background) appears shaky and unnatural as the subject is forced towards the center of the frame, often several times within a few seconds.
[0013] Figure 1 represents a subject of interest moving in a subject bounding window within a crop window according to an embodiment of the invention. In Figure 1 , full-resolution window 20 surrounds crop window 15. Within crop window 15, subject of interest 30 moves inside of subject bounding window 10. In the embodiment of Figure 1 , full-resolution window 20 represents the largest area that can be captured by the imaging array of the camera recording the scene. Vector 50, as shown by the arrow in Figure 1, indicates the two-dimensional motion of subject of interest 30 from right to left. In this embodiment of the invention, so long as subject of interest 30 stays within subject bounding window 10, the bounding window does not move within crop window 15, Consequently, background 40 is held steady.
[0014] It should be noted that in at ieast some embodiments of the invention, subject bounding window 10 is not visible to the user of the camera capturing the scene depicted in Figure 1. Subject bounding window 10 represents a bounding area used by embodiments of the invention to enable an algorithm to function without requiring the user to exercise direct control over the window, in contrast to window 10, crop window 15 is displayed to the user in at ieast some embodiments of the invention. Further, the inventors contemplate that in at ieast some embodiments of the invention, subject bounding window 10 and crop window 15 move together as will be seen in reference to Figure 2.
[0015] The inventors contemplate that through the use of a saliency algorithm such as face detection, edge detection, facial expression detection, skin tone detection, and so forth, the location of the subject within crop window 15 has already been found. An exemplary detection algorithm is provided in US patent number 7,099,510 entitled "Method and System for Object Detection in Digital images" or perhaps as provided in numerous other patent applications and issued US and foreign patents. When the location of the subject's face or other salient feature has been determined, subject bounding window 10 is positioned around the subject of interest.
[0016] Figure 2 shows a subject bounding window moving along with the crop window of Figure 1 in response to the subject of interest moving outside of subject bounding window of Figure 1 according to an embodiment of the invention, in Figure 2, subject of interest 30 has moved far enough to the left so as to cause subject bounding window 10 and crop window 15 to be repositioned to the left.
Accordingly, a different portion of background 40 is shown within subject bounding window 10. In the embodiment of Figure 2, subject bounding window 10 is moved if (and only if) the subject of interest has moved outside of the original subject bounding window, shown in Figure 1.
[0017] it should be pointed out that the relationship between full-resolution window 20, crop window 15, and subject bounding window 10 may be much different than that shown in Figures 1 and 2. in Figures 1 and 2, full-resolution window 20 is shown as encompassing perhaps twice the area of crop window 15. Additionally, crop window 15 is shown as encompassing perhaps twice the area of subject bounding window 10. However, in some embodiments of the invention, full- resolution window 20 may encompass many times the area as crop window 15, such as 15 or 20 times the area of the crop window, or maybe even larger. In a similar manner, crop window 15 may be many, many times larger than subject bounding window 10. A full-resolution window that is much larger than the crop window may allow the captured images to remain steady while the crop window moves relatively freely to accommodate the motion of the subject of interest and the subject bounding window.
[0018] In addition to the variability in the relative sizes of subject bounding window 10, crop window 15, and full-resolution window 20, these windows may also be of varying shapes. For example, subject bounding window 10 may be shaped as an oval, whiie crop window 15 is shaped as a square. These shapes may be used within a rectangular-shaped full resolution window 20.
[0019] Figure 3 shows the movement of the background of the captured image to place the subject of interest nearer to the center of the crop window
according to an embodiment of the invention. In Figure 3, subject of interest 30 has moved toward the tree shown at the right hand side of subject bounding window 10. in this embodiment of the invention, it is desirable to place the subject of interest 30 at the center of subject bounding window 10. Accordingly, subject bounding window 10 and crop window 15 have been repositioned to the right.
[0020] Figure 4 shows subject of interest 30 having moved outside of subject bounding window 10 according to an embodiment of the invention. As the subject of interest is no longer within subject bounding window 10, a new subject of interest (35) has been identified. In this embodiment of the invention, when subject of interest 30 moves to a position that cannot be compensated for, subject bounding window 10 is slowly and gracefully re-centered on new subject of interest 35.
[0021] Figure 5 shows a composite of two subjects within subject bounding window 10 according to an embodiment of the invention, in Figure 5, centroid 37 represents a spatially averaged location between the two subjects of interest. In the embodiment of Figure 5, the location of centroid 37 is tracked and the subject bounding window relocated based on the location and the two-dimensional motion vector of the centroid.
[0022] in Figures 1 -5, subject bounding window 10 has been shown as being fixed at or near the center of crop window 15. However, in other embodiments of the invention, subject bounding window 10 may be allowed to move more freely within crop window 15 and may not be required to occupy a center portion of the crop window.
[0023] Figure 6 is a method of stabilizing a subject of interest in captured video according to an embodiment of the Invention. The method of Figure 8 begins at step 100 in which a subject of interest is identified within the crop window. In one embodiment of the invention, the identification of the subject of interest includes using a saliency algorithm such as face detection, edge detection, facial expression detection, skin tone detection, and so forth to detect a single human face (such as the face of subject of interest 30) within a full-resolution window, in another
embodiment of the invention, the subject of interest is a composite of two or more subjects (represented by centroid 37} that indicates the iocation of two or more human faces whose location has been averaged in the subject bounding window.
[0024] The method continues at step 1 10, which includes establishing a first subject bounding window that includes the subject of interest. In this step, the subject bounding window is formed inside the crop window. At step 120, so long as the subject of interest moves within the subject bounding window, the camera refrains from substantially changing the location of the subject bounding window, in some embodiments of the invention, step 120 inciudes computing a two-dimensional motion vector that characterizes the movement of the subject of interest. The two- dimensional motion vector may be used to determine the direction that subject bounding window 10 should move in order to best accommodate subject of interest 30 when the subject of interest exits window 10.
[0025] At step 130, when the subject of interest exits the subject bounding window, the camera establishes a second subject bounding window that inciudes the new location of the subject of interest. The camera maintains the subject of interest within the second subject bounding window so long as the subject does not move outside of the second subject bounding window. Step 130 may also include centering the subject of interest within the within the second subject bounding window. Other embodiments of the invention may also include the step of identifying a second subject of interest when a first subject of interest approaches and edge of a full resolution window that encompasses the crop window,
[0026] Figure 7 is a method of stabilizing a subject of interest in captured video according to another embodiment of the invention. The method of Figure 7 begins at step 200, which includes identifying the subject of interest in the
successive frames of the video, in step 200, the identification of the subject of interest in the successive frames of the video may occur without user input, perhaps as a result of the video camera identifying one or more human faces or other salient features of a subject of interest within the successive frames of the video using a
suitable face or other type of detection algorithm. The identification in step 200 may also occur by way of receiving a user input that designates the subject of interest in the successive frames of the video. In an example of this embodiment, a
touchscreen that displays the video frame to the user may receive an input from the user that designates the subject of interest Thus, for example, in the event that the user is filming his or her new pet, the user may select the pet on a display of the video camera. After the user has designated the pet, the camera may establish a subject bounding window and holds the subject bounding window steady while the pet moves within the window.
[0027] The method continues at step 210, in which a first subject bounding window that encompasses the identified subject of interest is established. At step 220, the camera holds steady the background within the first crop window as the identified subject of interest moves within the first subject bounding window. At step 230, a second crop window is established when the identified subject of interest exits the first subject bounding window.
[0028] It should be noted that the method of Figure 7 can aiso be performed on a video stream stored on a storage media located outside of a video camera. In this embodiment, a processor stores successive frames of the video in a memory. Using a saiiency algorithm such as face detection, edge detection, facial expression detection, skin tone detection, and so forth, and/or perhaps with the assistance of a user, the subject of interest is identified within the successive frames of the video (as in step 200 of Figure 7). At step 210, the processor establishes a first subject bounding that encompasses the identified subject of interest. As the subject of interest (either as aided by a user, or perhaps as identified automatically by way of a detection algorithm) moves within the first subject bounding window, the background is held steady within the first crop window, as in step 220. When the subject of interest moves outside of the first subject bounding window, a second subject bounding window is established, as in step 230 of Figure 7.
[0029] Figure 8 is a block diagram for a module used to stabilize a subject of interest in captured video according to embodiments of the invention. Although module 301 of f igure 8 can be used to perform the method of Figures 6 and 7, nothing prevents the use of alternately-configured hardware, software, or firmware modules to perform the methods.
[0030] In Figure 8, lens 310 focuses incoming light from scene 300 onto imaging array 320. Imaging array 320 includes a CCD or CMOS imaging array that converts the incident optical signals that represent scene 300 into discrete electrical charges. The electrical charges are processed by image processing module 330, which applies correction factors to compensate for vignetting, color dependent shading across imaging array 320, channel balancing, and conversion from the raw outputs of the imaging array to a standard color space, such as RGB, sRGB, and so forth. The output of each frame of image processing module 330 is then mapped to memory array 340.
[0031] In one embodiment of the invention, processor 360 operates in conjunction with subject of interest detection module 365 to automatically (in which "automatically" implies "without user input") detect the subject present in scene 300. Subject of interest detection module 365 may include a saSiency algorithm such as face detection, edge detection, facial expression detection, skin tone detection and so forth to determine the presence of a subject of interest. In this embodiment of the invention, processor 360 identifies the subject of interest within scene 300 and establishes a subject bounding window that encompasses the subject of interest along with at least a portion of the background of the scene. When the subject of interest exits the subject bounding window, processor 300 determines the new location of the subject of interest and establishes a subject bounding window within memory array 340 around the new location of the subject of interest.
[0032] In another embodiment of the invention, processor 360, operating in conjunction with subject of interest detection module 385, averages the location of more than one face detected in scene 300 and identifies a composite as
representing the two or more subjects. In this embodiment, when the composite that represents the two or more subjects exits the subject bounding window, processor 360 and subject of interest detection module 365 establish a new subject bounding window within memory array 340, Within the second subject bounding window, the subject of interest may move about while processor 360 holds the second subject bounding window steady. The successive frames of the subject of interest and the background within the subject bounding windows are then stored within storage media 370,
[0033] in another embodiment of the invention, the user may interact with touchscreen 350 and/or user interface 355 to select the subject of interest, in this embodiment of the invention, the user may surround a shape of the subject of interest using a stylus or his/her finger to directly interact with the touchscreen.
Once the subject of interest has been selected, a subject bounding window may be established around the subject of interest When the subject of interest exits the subject bounding window, processor 360 establishes a second subject bounding window around the new location of the subject of interest,
[0034] in other embodiments of the invention, memory array 340 stores a previousiy captured succession of video frames to which the methods of the claimed invention can be applied. As the successive frames of the video stream are written to memory array 340, processor 360, operating in conjunction with subject of interest detection module 365, may make use of an algorithm to detect the presence of one or more subjects of interest within the video stream. Processor 380 may establish and hold steady the background of a subject bounding window within the video stream in a manner that aiiows the subject of interest to move within the subject bounding window. When the subject of interest moves outside of the subject bounding window, processor 360 and subject of interest detection module 365 operate to reorient the subject bounding window to include a new portion of memory array 340 in which the subject of interest is re-centered.
[0035] in conclusion, while the present invention has been particularly shown and described with reference to various embodiments, those skilled in the art will understand that many variations may be made therein without departing from the spirit and scope of the invention as defined in the following claims. This description of the invention should be understood to include the novel and non-obvious combinations of elements described herein, and claims may be presented in this or a later application to any novel and non-obvious combination of these elements. The foregoing embodiments are illustrative, and no single feature or element is essential to all possible combinations that may be claimed in this or a later application. Where the claims recite "a" or "a first" element or the equivalent thereof, such claims should be understood to include incorporation of one or more such elements, neither requiring nor excluding two or more such elements.
Claims
1. A method of stabilizing a subject of interest in captured video, comprising: identifying the subject of interest within a crop window;
establishing a first subject bounding window that includes the subject of interest: refraining from substantiaJly changing the first subject bounding window while the subject of interest moves within the first subject bounding window; and
establishing a second subject bounding window within the crop window that includes the subject of interest when the subject of interest exits the first subject bounding window.
2. The method of claim 1 , wherein the identifying step further comprises employing an algorithm to identify the subject of interest.
3. The method of claim 2, wherein the subject of interest is a composite that represents two or more subjects,
4. The method of claim 1, additionally comprising computing a two-dimensional motion vector that characterizes the movement of the subject of interest.
5. The method of claim 1 wherein the step of establishing a second subject bounding window includes centering the subject of interest within the second subject bounding window,
6. The method of claim 1 , additionally comprising identifying a second subject of interest when the subject of interest approaches an edge of a fuli resolution window that encompasses the crop window.
7. A method of stabilizing a subject of interest within successive frames of a video, comprising;
identifying the subject of interest in the successive frames of the video;
establishing a first subject bounding window that encompasses the identified subject of interest;
holding steady the background within the first crop window as the identified subject of interest moves within the first subject bounding window; and
establishing a second subject bounding window when the identified subject of interest exits the first subject bounding window,
8. The method of claim 7, wherein the identifying step further comprises employing an algorithm to determine that a face is present in the successive frames of the video.
9. The method of claim 7, wherein the identifying step includes a user selecting the subject of interest by way of a user interface.
10. The method of claim 7, wherein the step of establishing the second subject bounding window further comprises centering the identified subject of interest within a second crop window, the second crop window encompassing the second subject bounding window.
11. The method of claim 7, wherein the establishing step further comprises establishing a two-dimensional motion vector for the subject of interest using the successive frames of the video.
12. The method of claim 7, wherein the step of holding steady the background further comprises detecting an object in the background and maintaining the relative position of the object in the successive frames of the video.
13. A module for stabilizing a subject of interest in captured video, comprising: a memory array for storing captured images;
a processor for determining the presence of a subject of interest in the captured images, wherein
the processor establishes a first crop window within the memory array that encompasses the subject bounding window, and wherein
the processor establishes a fixed relationship between the area encompassed by the subject bounding window and the crop window.
14. The module of claim 13, wherein the processor additionally performs an algorithm to determine the presence of the subject of interest in the video frame;
15. The module of claim 13, wherein the processor relocates the subject of interest window and the crop window when the subject of interest moves outside of the subject bounding window,
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/387,059 US8922661B2 (en) | 2009-11-30 | 2009-11-30 | Stabilizing a subject of interest in captured video |
EP09851762.6A EP2507764A4 (en) | 2009-11-30 | 2009-11-30 | Stabilizing a subject of interest in captured video |
PCT/US2009/066139 WO2011065960A1 (en) | 2009-11-30 | 2009-11-30 | Stabilizing a subject of interest in captured video |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/US2009/066139 WO2011065960A1 (en) | 2009-11-30 | 2009-11-30 | Stabilizing a subject of interest in captured video |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2011065960A1 true WO2011065960A1 (en) | 2011-06-03 |
Family
ID=44066827
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2009/066139 WO2011065960A1 (en) | 2009-11-30 | 2009-11-30 | Stabilizing a subject of interest in captured video |
Country Status (3)
Country | Link |
---|---|
US (1) | US8922661B2 (en) |
EP (1) | EP2507764A4 (en) |
WO (1) | WO2011065960A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2013121082A1 (en) * | 2012-02-14 | 2013-08-22 | Nokia Corporation | Video image stabilization |
GB2529435A (en) * | 2014-08-19 | 2016-02-24 | Apical Ltd | A Method of Generating A Framed Video Stream |
US11263769B2 (en) | 2015-04-14 | 2022-03-01 | Sony Corporation | Image processing device, image processing method, and image processing system |
Families Citing this family (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5159189B2 (en) * | 2007-06-29 | 2013-03-06 | キヤノン株式会社 | Image processing apparatus, imaging apparatus, image processing method, and program |
JP5431083B2 (en) * | 2009-09-16 | 2014-03-05 | オリンパスイメージング株式会社 | Image capturing apparatus and method for controlling image capturing apparatus |
JP2011223565A (en) * | 2010-03-26 | 2011-11-04 | Panasonic Corp | Imaging device |
US8810666B2 (en) * | 2012-01-16 | 2014-08-19 | Google Inc. | Methods and systems for processing a video for stabilization using dynamic crop |
WO2014021005A1 (en) * | 2012-07-31 | 2014-02-06 | 日本電気株式会社 | Image processing system, image processing method, and program |
US9279983B1 (en) * | 2012-10-30 | 2016-03-08 | Google Inc. | Image cropping |
EP2797308A3 (en) * | 2013-04-22 | 2015-01-07 | Technologies Humanware Inc | Live panning system and method |
US11743402B2 (en) * | 2015-02-13 | 2023-08-29 | Awes.Me, Inc. | System and method for photo subject display optimization |
US10397527B2 (en) * | 2015-07-08 | 2019-08-27 | Omar Barlas | Remotely controlled robotic sensor ball |
JP6641763B2 (en) * | 2015-08-03 | 2020-02-05 | セイコーエプソン株式会社 | Display system |
US10868955B2 (en) * | 2017-09-05 | 2020-12-15 | Facebook, Inc. | Modifying capture of video data by an image capture device based on video data previously captured by the image capture device |
US20210136135A1 (en) * | 2019-10-31 | 2021-05-06 | Sony Interactive Entertainment Inc. | Image stabilization cues for accessible game stream viewing |
CN112135188A (en) * | 2020-09-16 | 2020-12-25 | 咪咕文化科技有限公司 | Video clipping method, electronic device and computer-readable storage medium |
CN114222181B (en) * | 2021-11-11 | 2024-03-12 | 北京达佳互联信息技术有限公司 | Image processing method, device, equipment and medium |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2003030686A (en) * | 2001-07-19 | 2003-01-31 | Konami Co Ltd | Video game device, pseudo camera view point movement control method in video game and program |
JP2005115544A (en) * | 2003-10-06 | 2005-04-28 | Fuji Xerox Co Ltd | Operation identification apparatus and object posture identification apparatus |
JP2009157821A (en) * | 2007-12-27 | 2009-07-16 | Toyota Central R&D Labs Inc | Range image generating device, environment recognition device, and program |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR100252080B1 (en) * | 1997-10-10 | 2000-04-15 | 윤종용 | Apparatus for stabilizing video signals through revising the motion of the video signals using bit plane matching and a stabilizing method therefor |
JP2004191906A (en) | 2002-10-18 | 2004-07-08 | Konica Minolta Holdings Inc | Optical compensating film, integrated type angle-of-field compensation polarization plate, and liquid crystal display |
US20040100560A1 (en) * | 2002-11-22 | 2004-05-27 | Stavely Donald J. | Tracking digital zoom in a digital video camera |
GB0229096D0 (en) | 2002-12-13 | 2003-01-15 | Qinetiq Ltd | Image stabilisation system and method |
US7643066B2 (en) * | 2004-02-19 | 2010-01-05 | Robert Bosch Gmbh | Method and apparatus for producing frame accurate position data in a PTZ dome camera with open loop control |
GB0502369D0 (en) * | 2005-02-04 | 2005-03-16 | British Telecomm | Classifying an object in a video frame |
JP4845414B2 (en) | 2005-04-13 | 2011-12-28 | 中井銘鈑株式会社 | Pearl pattern decorative body |
KR101398475B1 (en) * | 2007-11-21 | 2014-05-26 | 삼성전자주식회사 | Apparatus for processing digital image and method for controlling thereof |
DE102008033144A1 (en) | 2008-07-15 | 2010-01-21 | Dürkopp Adler AG | Actuating device for operating a sewing machine |
-
2009
- 2009-11-30 WO PCT/US2009/066139 patent/WO2011065960A1/en active Application Filing
- 2009-11-30 US US13/387,059 patent/US8922661B2/en not_active Expired - Fee Related
- 2009-11-30 EP EP09851762.6A patent/EP2507764A4/en not_active Withdrawn
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2003030686A (en) * | 2001-07-19 | 2003-01-31 | Konami Co Ltd | Video game device, pseudo camera view point movement control method in video game and program |
JP2005115544A (en) * | 2003-10-06 | 2005-04-28 | Fuji Xerox Co Ltd | Operation identification apparatus and object posture identification apparatus |
JP2009157821A (en) * | 2007-12-27 | 2009-07-16 | Toyota Central R&D Labs Inc | Range image generating device, environment recognition device, and program |
Non-Patent Citations (1)
Title |
---|
See also references of EP2507764A4 * |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2013121082A1 (en) * | 2012-02-14 | 2013-08-22 | Nokia Corporation | Video image stabilization |
US8743222B2 (en) | 2012-02-14 | 2014-06-03 | Nokia Corporation | Method and apparatus for cropping and stabilization of video images |
GB2529435A (en) * | 2014-08-19 | 2016-02-24 | Apical Ltd | A Method of Generating A Framed Video Stream |
US9904979B2 (en) | 2014-08-19 | 2018-02-27 | Apical Ltd. | Method of generating a framed video system |
GB2529435B (en) * | 2014-08-19 | 2020-09-02 | Apical Ltd | A Method of Generating A Framed Video Stream |
US11263769B2 (en) | 2015-04-14 | 2022-03-01 | Sony Corporation | Image processing device, image processing method, and image processing system |
EP3285477B1 (en) * | 2015-04-14 | 2023-06-28 | Sony Group Corporation | Image processing device, image processing method, and image processing system |
Also Published As
Publication number | Publication date |
---|---|
US20120127329A1 (en) | 2012-05-24 |
US8922661B2 (en) | 2014-12-30 |
EP2507764A1 (en) | 2012-10-10 |
EP2507764A4 (en) | 2013-06-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8922661B2 (en) | Stabilizing a subject of interest in captured video | |
JP5136669B2 (en) | Image processing apparatus, image processing method, and program | |
JP4748244B2 (en) | Image selection apparatus, image selection method, and program | |
US8786760B2 (en) | Digital photographing apparatus and method using face recognition function | |
EP3562143B1 (en) | Image processing device, image processing method, and program | |
US20110096143A1 (en) | Apparatus for generating a panoramic image, method for generating a panoramic image, and computer-readable medium | |
US9113075B2 (en) | Image processing method and apparatus and digital photographing apparatus using the same | |
US20120057786A1 (en) | Image processing apparatus, image processing method, image pickup apparatus, and storage medium storing image processing program | |
US10455154B2 (en) | Image processing device, image processing method, and program including stable image estimation and main subject determination | |
US9743000B2 (en) | Moving image processing apparatus, imaging apparatus, and moving image processing method | |
US20130286217A1 (en) | Subject area detection apparatus that extracts subject area from image, control method therefor, and storage medium, as well as image pickup apparatus and display apparatus | |
US8970711B2 (en) | Imaging apparatus for correcting distortion in image captured using rolling shutter method and distortion correction method | |
JP6011569B2 (en) | Imaging apparatus, subject tracking method, and program | |
JP5888348B2 (en) | Imaging apparatus, imaging control method, and program | |
KR101423432B1 (en) | Imaging apparatus, imaging method and storage medium | |
JP4894708B2 (en) | Imaging device | |
KR101665175B1 (en) | Image acquisition apparatus,image acquisition method and recording medium | |
JP5370555B2 (en) | Imaging apparatus, imaging method, and program | |
KR20130035207A (en) | Image processing device, image processing metho and recording medium | |
KR20180017591A (en) | Camera apparatus, display apparatus and method of correcting a movement therein | |
KR20060056235A (en) | Auto image pickup apparatus and control program thereof | |
WO2017104102A1 (en) | Imaging device | |
JP2007251532A (en) | Imaging device and face area extraction method | |
US9781337B2 (en) | Image processing device, image processing method, and recording medium for trimming an image based on motion information | |
US10178298B2 (en) | Image processing device, image processing method, and recording medium for optimal trimming of a captured image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 09851762 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13387059 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2009851762 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |