US20070046782A1 - Method and system for determining the motion of an imaging apparatus - Google Patents
Method and system for determining the motion of an imaging apparatus Download PDFInfo
- Publication number
- US20070046782A1 US20070046782A1 US11/212,460 US21246005A US2007046782A1 US 20070046782 A1 US20070046782 A1 US 20070046782A1 US 21246005 A US21246005 A US 21246005A US 2007046782 A1 US2007046782 A1 US 2007046782A1
- Authority
- US
- United States
- Prior art keywords
- motion
- sensor
- motion vectors
- windows
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 50
- 238000000034 method Methods 0.000 title claims description 48
- 239000013598 vector Substances 0.000 claims abstract description 83
- 230000006641 stabilisation Effects 0.000 description 11
- 238000011105 stabilization Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 9
- 230000002596 correlated effect Effects 0.000 description 4
- 230000000875 corresponding effect Effects 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B3/00—Simple or compound lenses
- G02B3/0006—Arrays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
- H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
- H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
- H01L27/144—Devices controlled by radiation
- H01L27/146—Imager structures
- H01L27/14601—Structural or functional details thereof
- H01L27/14625—Optical elements or arrangements associated with the device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
- H04N1/024—Details of scanning heads ; Means for illuminating the original
- H04N1/028—Details of scanning heads ; Means for illuminating the original for picture information pick-up
- H04N1/03—Details of scanning heads ; Means for illuminating the original for picture information pick-up with photodetectors arranged in a substantially linear array
- H04N1/031—Details of scanning heads ; Means for illuminating the original for picture information pick-up with photodetectors arranged in a substantially linear array the photodetectors having a one-to-one and optically positive correspondence with the scanned picture elements, e.g. linear contact sensors
- H04N1/0318—Integral pick-up heads, i.e. self-contained heads whose basic elements are a light-source, a lens array and a photodetector array which are supported by a single-piece frame
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
- H04N23/6811—Motion detection based on the image signal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/144—Movement detection
- H04N5/145—Movement estimation
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/64—Imaging systems using optical elements for stabilisation of the lateral and angular position of the image
- G02B27/646—Imaging systems using optical elements for stabilisation of the lateral and angular position of the image compensating for small deviations, e.g. due to vibration or shake
Definitions
- An image apparatus user such as a still or video camera user, often captures images that are blurred or shaky.
- the blurred or shaky images may be caused by a subject moving during image capture or by vibrations caused by the movement of the imaging apparatus itself during image capture. For example, camera shake, a slow shutter speed, or the use of a telephoto lens without a tripod can result in blurry or shaky images because the imaging apparatus moves during image capture.
- An image stabilization technique is used to compensate for the motion of the imager during image capture.
- Image stabilization creates sharper images by dampening or counteracting the vibrations generated by the movement of the imaging apparatus or imager.
- One image stabilization technique employs a built-in gyro sensor to measure and correct for camera shake as it occurs.
- Another image stabilization technique uses an imaging sensor to capture images that are used to compensate for the motion of the imager.
- FIG. 1 is a block diagram of a front view of an imaging apparatus according to the prior art.
- Imaging apparatus 100 includes imager 102 and motion sensor 104 . As illustrated in FIG. 2 , both imager 102 and motion sensor 104 capture images of scene 106 .
- Motion sensor 104 captures images at a high frame rate and the motion of image apparatus 100 is calculated by correlating successive images of sensor 104 .
- Motion sensor 104 has a smaller number of pixels compared to imager 102 , resulting in sensor 104 having a smaller field of view 110 compared to imager 102 .
- the smaller field of view 110 can cause motion sensor 104 to detect motion that occurs in scene 106 and incorrectly compensate for that motion rather than for any movement in the imaging apparatus.
- motion sensor 104 may capture images of a person's head when imager 102 is capturing an image of a person. Motion sensor 104 will detect motion f the person moves their head during image capture. But imaging apparatus 100 may not have moved or may have moved less than the motion of the head. Thus, an image stabilization technique may incorrectly compensate for motion that did not occur or overcompensate for the motion of imaging apparatus 100 .
- a method and system for determining the motion of an imaging apparatus are provided.
- Multiple motion sensor devices are included in an imaging apparatus.
- One or more of the motion sensor devices captures images and at least a portion of the sensor images are then processed to determine one or more motion vectors for the processed sensor images.
- At least one motion vector representing an amount of motion of the imager during image capture is generated when a predetermined portion of the motion vectors match.
- FIG. 1 is a block diagram of a front view of an imaging apparatus according to the prior art
- FIG. 2 is a block diagram of a left side view of the imaging apparatus shown in FIG. 1 ;
- FIG. 3 is a block diagram of a front view of an imaging apparatus in an embodiment in accordance with the invention.
- FIG. 4 is a block diagram of a left side view of imaging apparatus 300 ;
- FIGS. 5A-5B illustrate a flowchart of a method for determining the motion of an imaging apparatus in an embodiment in accordance with the invention
- FIG. 6 is a block diagram of a scene and sensor images in an embodiment in accordance with the invention.
- FIG. 7 depicts sensor image 602 with windows in an embodiment in accordance with the invention.
- FIG. 8 illustrates window 700 with sub-windows in an embodiment in accordance with the invention
- FIG. 9 depicts a first sensor image with windows processed in accordance with the method shown in FIGS. 5A-5B ;
- FIG. 10 illustrates a second sensor image with windows processed in accordance with the method shown in FIGS. 5A-5B .
- FIG. 3 is a block diagram of a front view of an imaging apparatus in an embodiment in accordance with the invention.
- Imaging apparatus 300 includes imager 102 and optical motion sensor 302 .
- Motion sensor 302 includes sensor devices 304 , 306 , 308 implemented as complementary metal oxide semiconductor (CMOS) devices in an embodiment in accordance with the invention.
- CMOS complementary metal oxide semiconductor
- sensor devices 304 , 306 , 308 may be implemented with different imaging devices, such as, for example, charged couple devices (CCD).
- CCD charged couple devices
- Imager 102 is implemented as any known imaging device capable of capturing still or video images. Imager 102 captures images of scene 106 located in its field of view. Sensor devices 304 , 306 , 308 capture one or more images of sections of the scene in an embodiment in accordance with the invention. In another embodiment in accordance with the invention, some or all of sensor devices 304 , 306 , 308 capture one or more images of a scene located outside of the field of view of imager 102 .
- the sensor images are used to determine one or more motion vectors (i.e., direction and distance values) associated with the movement of imaging apparatus 300 during image capture.
- the one or more motion vectors are used with an image stabilization technique to stabilize the images captured by imager 102 in an embodiment in accordance with the invention.
- image stabilization technique to stabilize the images captured by imager 102 in an embodiment in accordance with the invention.
- FIG. 4 is a block diagram of a left side view of imaging apparatus 300 .
- Imager 102 is not shown for clarity.
- Lenslet array 400 includes three lenses 402 , 404 , 406 positioned in an imaging path of sensor devices 304 , 306 , 308 , respectively. Each lens may be tilted or rotated to allow its corresponding sensor device to capture different sections of scene 106 .
- lens 402 is positioned (i.e., tilted) to allow sensor device 304 to capture images of sub-field 408 of scene 106
- lens 404 positioned to allow sensor device 306 to capture images of sub-field 410
- lens 406 is positioned such that sensor device 308 captures images of sub-field 412 in an embodiment in accordance with the invention.
- the lenses in lenslet array 400 may be positioned in any given direction in other embodiments in accordance with the invention.
- the sensor images captured by sensor devices 304 , 306 , 308 are processed by processing device 414 to determine one or more motion vectors for each sensor image.
- the sensor images are stored in memory 416 prior to processing in an embodiment in accordance with the invention.
- One technique that may be used to determine motion vectors is image correlation. Two images captured by the same sensor are correlated in order to determine whether motion occurred between the times the two images were captured. The motion may be from the movement of the imaging apparatus, movement of an object in the scene, or both the movement of the imaging apparatus and the movement of an object in the scene.
- FIGS. 5A-5B illustrate a flowchart of a method for determining the motion of an imaging apparatus in an embodiment in accordance with the invention.
- the difference image is then correlated with one of the sensor images used to create the difference image.
- the difference image is correlated with image I(t).
- the difference image is correlated with image I(t+ ⁇ ).
- the results of the correlation are then used to determine whether motion occurred between the time images I(t) and I(t+ ⁇ ) were captured.
- the motion may be from the movement of the imaging apparatus, movement of an object in the scene, or both the movement of the imaging apparatus and the movement of an object in the scene.
- FIGS. 5A-5B illustrate a flowchart of a method for determining the motion of an imaging apparatus in an embodiment in accordance with the invention. Initially some or all of the sensor devices in an imaging apparatus capture sensor images, as shown in block 500 . The sensor images may be captured simultaneously or successively.
- FIG. 6 is a block diagram of a scene and sensor images in an embodiment in accordance with the invention.
- Scene 600 includes sensor images 602 , 604 , 606 .
- the locations of sensor images 602 , 604 , 606 correspond to sensor devices 304 , 306 , 308 in FIG. 3 .
- Other embodiments in accordance with the invention may include any given number of sensor images.
- the sensor images may be arranged in any given configuration.
- FIG. 7 depicts sensor image 602 with windows in an embodiment in accordance with the invention.
- Sensor image 602 includes nine windows 700 with each window 700 including a portion of the image in sensor image 602 .
- Windows 700 may be equal in size or may have different sizes.
- the shape of windows 700 is not limited to square or rectangular shapes. Windows 700 may be configured in any given shape.
- the windows in the sensor image are processed individually to determine a motion vector for each window.
- not all windows are processed and the windows to be processed are selected.
- the windows may be grouped together in various configurations and the groups processed.
- the motion vectors are then stored at block 510 and a determination made as to whether there are any remaining sensor images to process (block 512 ). If there is a remaining sensor image to process, the method returns to block 506 and repeats until all of the sensor images have been processed.
- the process passes to block 514 where two or more windows are generated in a sensor image. Two or more sub-windows are then generated in a window at block 516 . All of the windows are sub-divided in the embodiment shown in FIGS. 5A-5B . In another embodiment in accordance with the invention, not all windows are sub-divided. And in yet another embodiment in accordance with the invention, only a portion of some or all of the windows are sub-divided.
- FIG. 8 illustrates window 700 with sub-windows in an embodiment in accordance with the invention.
- Window 700 includes four sub-windows 800 with each sub-window including a portion of the image in window 700 .
- Sub-windows 800 may be equal in size or may have different sizes.
- the shape of sub-windows 800 is not limited to square or rectangular shapes.
- Sub-windows 800 may be configured in any given shape.
- the sub-windows in the window are then processed individually at block 518 to determine a motion vector for each sub-window.
- not all sub-windows are processed and the sub-windows to be processed are selected.
- the sub-windows may be grouped together in various configurations and the groups processed.
- the motion vectors are stored and a determination made as to whether there are any remaining windows to process. If there are remaining windows to process, the method returns to block 514 and repeats until all of the windows have been processed. When there are no remaining windows to be processed the process continues at block 524 where a determination is made as to whether there are any remaining sensor images to process. If so, the method returns to block 514 and repeats until all of the sensor images have been processed.
- FIG. 9 depicts a first sensor image with windows processed in accordance with the method shown in FIGS. 5A-5B .
- the motion vector for each window is indicated by arrow 900 .
- the motion vectors for all of the windows match.
- motion vectors 900 represent the amount of motion of the imager during image capture in an embodiment in accordance with the invention.
- the values associated with motion vectors 900 are used with an image stabilization technique to stabilize the images captured by an imager in an imaging apparatus (e.g. imager 102 in imaging apparatus 300 in FIG. 3 ).
- FIG. 10 illustrates a second sensor image with windows processed in accordance with the method shown in FIGS. 5A-5B .
- the motion vectors for four windows are indicated by arrow 1000 while the motion vectors for five windows are indicated by arrow 1002 . Since five out of nine windows is a majority, the values associated with motion vectors 1002 represent the amount of motion of the imager and are used with an image stabilization technique to stabilize the images captured by an imager in an embodiment in accordance with the invention.
- Motion vectors 1000 represent motion occurring in the scene and as such, are not utilized by an image stabilization technique in an embodiment in accordance with the invention.
- a motion vector equaling zero is output at block 532 since the motion is not due to the movement of the imaging apparatus. If all of the motion vectors match at block 528 , or a majority of the motion vectors match at block 530 , a motion vector equaling the motion vectors in all or in the majority is output at block 534 .
- the motion vector output at block 534 is then used in an image stabilization technique to stabilize the images captured by an imager in an imaging apparatus.
- the method passes to block 536 where the sensor images are processed individually to determine a motion vector for each sensor image in an embodiment in accordance with the invention.
- the motion vectors are then compared at block 526 and a determination made as to whether all of the motion vectors match (block 528 ). If the motion vectors for the sensor images do not match, the method continues at block 530 where a determination is made as to whether the motion vectors for a majority of the sensor images match.
- motion vectors equaling zero are output at block 532 . If the motion vectors for all or a majority of the sensor images match, a motion vector equaling the motion vectors for all or for the majority of sensors is output at block 534 . The motion vector output at block 534 is then used in an image stabilization technique to stabilize the images captured by an imager in the imaging apparatus.
- FIGS. 5A-5B compares the motion vectors associated with the sensor images to determine if all or a majority match
- the motion vectors are compared to determine whether a predetermined portion of the motion vectors match. Moreover, the comparison does not have to include all of the computed motion vectors.
- Other embodiments in accordance with the invention can compare only a portion of the motion vectors or select which motion vectors to compare.
- FIGS. 5A-5B depict one method for determining the motion of an imaging apparatus.
- Embodiments in accordance with the invention are not limited to the order of the blocks or the blocks as shown in FIGS. 5A-5B .
- Other embodiments in accordance with the invention may include additional blocks not shown in FIGS. 5A-5B .
- one or more of the blocks shown in FIGS. 5A-5B may be deleted or re-arranged.
- the windows may not be sub-divided and as such blocks 514 - 524 would not be included in this embodiment in accordance with the invention.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Power Engineering (AREA)
- Computer Hardware Design (AREA)
- Condensed Matter Physics & Semiconductors (AREA)
- Electromagnetism (AREA)
- Microelectronics & Electronic Packaging (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Optics & Photonics (AREA)
- Studio Devices (AREA)
- Image Analysis (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
Description
- An image apparatus user, such as a still or video camera user, often captures images that are blurred or shaky. The blurred or shaky images may be caused by a subject moving during image capture or by vibrations caused by the movement of the imaging apparatus itself during image capture. For example, camera shake, a slow shutter speed, or the use of a telephoto lens without a tripod can result in blurry or shaky images because the imaging apparatus moves during image capture.
- An image stabilization technique is used to compensate for the motion of the imager during image capture. Image stabilization creates sharper images by dampening or counteracting the vibrations generated by the movement of the imaging apparatus or imager. One image stabilization technique employs a built-in gyro sensor to measure and correct for camera shake as it occurs. Another image stabilization technique uses an imaging sensor to capture images that are used to compensate for the motion of the imager.
-
FIG. 1 is a block diagram of a front view of an imaging apparatus according to the prior art.Imaging apparatus 100 includesimager 102 andmotion sensor 104. As illustrated inFIG. 2 , bothimager 102 andmotion sensor 104 capture images ofscene 106.Motion sensor 104 captures images at a high frame rate and the motion ofimage apparatus 100 is calculated by correlating successive images ofsensor 104.Motion sensor 104 has a smaller number of pixels compared toimager 102, resulting insensor 104 having a smaller field ofview 110 compared toimager 102. The smaller field ofview 110 can causemotion sensor 104 to detect motion that occurs inscene 106 and incorrectly compensate for that motion rather than for any movement in the imaging apparatus. For example,motion sensor 104 may capture images of a person's head whenimager 102 is capturing an image of a person.Motion sensor 104 will detect motion f the person moves their head during image capture. Butimaging apparatus 100 may not have moved or may have moved less than the motion of the head. Thus, an image stabilization technique may incorrectly compensate for motion that did not occur or overcompensate for the motion ofimaging apparatus 100. - In accordance with the invention, a method and system for determining the motion of an imaging apparatus are provided. Multiple motion sensor devices are included in an imaging apparatus. One or more of the motion sensor devices captures images and at least a portion of the sensor images are then processed to determine one or more motion vectors for the processed sensor images. At least one motion vector representing an amount of motion of the imager during image capture is generated when a predetermined portion of the motion vectors match.
-
FIG. 1 is a block diagram of a front view of an imaging apparatus according to the prior art; -
FIG. 2 is a block diagram of a left side view of the imaging apparatus shown inFIG. 1 ; -
FIG. 3 is a block diagram of a front view of an imaging apparatus in an embodiment in accordance with the invention; -
FIG. 4 is a block diagram of a left side view ofimaging apparatus 300; -
FIGS. 5A-5B illustrate a flowchart of a method for determining the motion of an imaging apparatus in an embodiment in accordance with the invention; -
FIG. 6 is a block diagram of a scene and sensor images in an embodiment in accordance with the invention; -
FIG. 7 depictssensor image 602 with windows in an embodiment in accordance with the invention; -
FIG. 8 illustrateswindow 700 with sub-windows in an embodiment in accordance with the invention; -
FIG. 9 depicts a first sensor image with windows processed in accordance with the method shown inFIGS. 5A-5B ; and -
FIG. 10 illustrates a second sensor image with windows processed in accordance with the method shown inFIGS. 5A-5B . - The following description is presented to enable one skilled in the art to make and use embodiments of the invention, and is provided in the context of a patent application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the generic principles herein may be applied to other embodiments. Thus, the invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the appended claims and with the principles and features described herein. It should be understood that the drawings referred to in this description are not drawn to scale.
- Like reference numerals designate corresponding parts throughout the figures.
FIG. 3 is a block diagram of a front view of an imaging apparatus in an embodiment in accordance with the invention.Imaging apparatus 300 includesimager 102 andoptical motion sensor 302.Motion sensor 302 includessensor devices sensor devices -
Imager 102 is implemented as any known imaging device capable of capturing still or video images.Imager 102 captures images ofscene 106 located in its field of view.Sensor devices sensor devices imager 102. - The sensor images are used to determine one or more motion vectors (i.e., direction and distance values) associated with the movement of
imaging apparatus 300 during image capture. The one or more motion vectors are used with an image stabilization technique to stabilize the images captured byimager 102 in an embodiment in accordance with the invention. Although only three sensor devices are shown inFIG. 3 , embodiments in accordance with the invention can include any number of multiple sensors. -
FIG. 4 is a block diagram of a left side view ofimaging apparatus 300.Imager 102 is not shown for clarity.Lenslet array 400 includes threelenses sensor devices scene 106. For example,lens 402 is positioned (i.e., tilted) to allowsensor device 304 to capture images ofsub-field 408 ofscene 106,lens 404 positioned to allowsensor device 306 to capture images ofsub-field 410, whilelens 406 is positioned such thatsensor device 308 captures images ofsub-field 412 in an embodiment in accordance with the invention. The lenses inlenslet array 400 may be positioned in any given direction in other embodiments in accordance with the invention. - The sensor images captured by
sensor devices processing device 414 to determine one or more motion vectors for each sensor image. The sensor images are stored inmemory 416 prior to processing in an embodiment in accordance with the invention. One technique that may be used to determine motion vectors is image correlation. Two images captured by the same sensor are correlated in order to determine whether motion occurred between the times the two images were captured. The motion may be from the movement of the imaging apparatus, movement of an object in the scene, or both the movement of the imaging apparatus and the movement of an object in the scene.FIGS. 5A-5B illustrate a flowchart of a method for determining the motion of an imaging apparatus in an embodiment in accordance with the invention. - Another technique that may be used to determine motion vectors is disclosed in commonly assigned U.S. patent application Ser. No. 11/014,482, which is incorporated herein by reference. For each
sensor device imaging apparatus 300, two sensor images are captured and a difference image generated by subtracting one sensor image I(t) from the other sensor image I(t+Δ). For example, the grayscale values from image I(t) are subtracted from the grayscale values in image I(t+Δ). Thus, the difference image represents the difference in grayscale values between the two sensor images. - The difference image is then correlated with one of the sensor images used to create the difference image. For example, in one embodiment in accordance with the invention, the difference image is correlated with image I(t). In another embodiment in accordance with the invention, the difference image is correlated with image I(t+Δ). The results of the correlation are then used to determine whether motion occurred between the time images I(t) and I(t+Δ) were captured. Again, the motion may be from the movement of the imaging apparatus, movement of an object in the scene, or both the movement of the imaging apparatus and the movement of an object in the scene.
-
FIGS. 5A-5B illustrate a flowchart of a method for determining the motion of an imaging apparatus in an embodiment in accordance with the invention. Initially some or all of the sensor devices in an imaging apparatus capture sensor images, as shown inblock 500. The sensor images may be captured simultaneously or successively. - As discussed earlier, each sensor image depicts a section of a scene.
FIG. 6 is a block diagram of a scene and sensor images in an embodiment in accordance with the invention.Scene 600 includessensor images FIG. 6 , the locations ofsensor images sensor devices FIG. 3 . Other embodiments in accordance with the invention may include any given number of sensor images. Moreover, the sensor images may be arranged in any given configuration. - Referring again to
FIG. 5A , a determination is made atblock 502 as to whether the sensor images are to be sub-divided. All of the sensor images are sub-divided in the embodiment shown inFIGS. 5A-5B . In another embodiment in accordance with the invention, not all sensor images are sub-divided. And in yet another embodiment in accordance with the invention, only a portion of some or all of the sensor images are sub-divided. - If the sensor images are to be sub-divided, the process passes to block 504 where a determination is made as to whether the windows in a sensor image are to be sub-divided. If the windows are not sub-divided, two or more windows are generated in a sensor image at
block 506.FIG. 7 depictssensor image 602 with windows in an embodiment in accordance with the invention.Sensor image 602 includes ninewindows 700 with eachwindow 700 including a portion of the image insensor image 602.Windows 700 may be equal in size or may have different sizes. Moreover, the shape ofwindows 700 is not limited to square or rectangular shapes.Windows 700 may be configured in any given shape. - Next, at
block 508, the windows in the sensor image are processed individually to determine a motion vector for each window. In another embodiment in accordance with the invention, not all windows are processed and the windows to be processed are selected. And in yet another embodiment in accordance with the invention, the windows may be grouped together in various configurations and the groups processed. - The motion vectors are then stored at
block 510 and a determination made as to whether there are any remaining sensor images to process (block 512). If there is a remaining sensor image to process, the method returns to block 506 and repeats until all of the sensor images have been processed. - Returning to block 504, if the windows are to be sub-divided the process passes to block 514 where two or more windows are generated in a sensor image. Two or more sub-windows are then generated in a window at
block 516. All of the windows are sub-divided in the embodiment shown inFIGS. 5A-5B . In another embodiment in accordance with the invention, not all windows are sub-divided. And in yet another embodiment in accordance with the invention, only a portion of some or all of the windows are sub-divided. -
FIG. 8 illustrateswindow 700 with sub-windows in an embodiment in accordance with the invention.Window 700 includes foursub-windows 800 with each sub-window including a portion of the image inwindow 700.Sub-windows 800 may be equal in size or may have different sizes. Moreover, the shape ofsub-windows 800 is not limited to square or rectangular shapes.Sub-windows 800 may be configured in any given shape. - The sub-windows in the window are then processed individually at
block 518 to determine a motion vector for each sub-window. In another embodiment in accordance with the invention, not all sub-windows are processed and the sub-windows to be processed are selected. And in yet another embodiment in accordance with the invention, the sub-windows may be grouped together in various configurations and the groups processed. - Next, at
blocks block 524 where a determination is made as to whether there are any remaining sensor images to process. If so, the method returns to block 514 and repeats until all of the sensor images have been processed. - When the motion vectors are determined for all sensor images, the motion vectors are compared at
block 526. A determination is then made atblock 528 as to whether all of the motion vectors match.FIG. 9 depicts a first sensor image with windows processed in accordance with the method shown inFIGS. 5A-5B . The motion vector for each window is indicated byarrow 900. As illustrated inFIG. 9 , the motion vectors for all of the windows match. Thus,motion vectors 900 represent the amount of motion of the imager during image capture in an embodiment in accordance with the invention. The values associated withmotion vectors 900 are used with an image stabilization technique to stabilize the images captured by an imager in an imaging apparatus (e.g. imager 102 inimaging apparatus 300 inFIG. 3 ). - If the motion vectors for all of the windows or sub-windows do not match, the method passes to block 530 where a determination is made as to whether a majority of motion vectors for the windows or sub-windows match.
FIG. 10 illustrates a second sensor image with windows processed in accordance with the method shown inFIGS. 5A-5B . The motion vectors for four windows are indicated byarrow 1000 while the motion vectors for five windows are indicated byarrow 1002. Since five out of nine windows is a majority, the values associated withmotion vectors 1002 represent the amount of motion of the imager and are used with an image stabilization technique to stabilize the images captured by an imager in an embodiment in accordance with the invention.Motion vectors 1000 represent motion occurring in the scene and as such, are not utilized by an image stabilization technique in an embodiment in accordance with the invention. - If a majority of the motion vectors do not match, the motion vectors represent motion that occurred in the scene during image capture. Consequently, a motion vector equaling zero is output at
block 532 since the motion is not due to the movement of the imaging apparatus. If all of the motion vectors match atblock 528, or a majority of the motion vectors match atblock 530, a motion vector equaling the motion vectors in all or in the majority is output atblock 534. The motion vector output atblock 534 is then used in an image stabilization technique to stabilize the images captured by an imager in an imaging apparatus. - Referring again to block 502 in
FIG. 5A , if the sensor images are not sub-divided, the method passes to block 536 where the sensor images are processed individually to determine a motion vector for each sensor image in an embodiment in accordance with the invention. The motion vectors are then compared atblock 526 and a determination made as to whether all of the motion vectors match (block 528). If the motion vectors for the sensor images do not match, the method continues atblock 530 where a determination is made as to whether the motion vectors for a majority of the sensor images match. - If a majority of motion vectors do not match, motion vectors equaling zero are output at
block 532. If the motion vectors for all or a majority of the sensor images match, a motion vector equaling the motion vectors for all or for the majority of sensors is output atblock 534. The motion vector output atblock 534 is then used in an image stabilization technique to stabilize the images captured by an imager in the imaging apparatus. - Although the method shown in
FIGS. 5A-5B compares the motion vectors associated with the sensor images to determine if all or a majority match, other embodiments in accordance with the invention are not limited to this implementation. In other embodiments in accordance with the invention, the motion vectors are compared to determine whether a predetermined portion of the motion vectors match. Moreover, the comparison does not have to include all of the computed motion vectors. Other embodiments in accordance with the invention can compare only a portion of the motion vectors or select which motion vectors to compare. -
FIGS. 5A-5B depict one method for determining the motion of an imaging apparatus. Embodiments in accordance with the invention are not limited to the order of the blocks or the blocks as shown inFIGS. 5A-5B . Other embodiments in accordance with the invention may include additional blocks not shown inFIGS. 5A-5B . Moreover, one or more of the blocks shown inFIGS. 5A-5B may be deleted or re-arranged. For example, the windows may not be sub-divided and as such blocks 514-524 would not be included in this embodiment in accordance with the invention.
Claims (20)
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/212,460 US7777781B2 (en) | 2005-08-26 | 2005-08-26 | Method and system for determining the motion of an imaging apparatus |
TW095130406A TWI394435B (en) | 2005-08-26 | 2006-08-18 | Method and system for determining the motion of an imaging apparatus |
CN2006101115718A CN1921565B (en) | 2005-08-26 | 2006-08-23 | Method and system for determining the motion of an imaging apparatus |
JP2006227558A JP4963384B2 (en) | 2005-08-26 | 2006-08-24 | Method and system for determining motion of an imaging device |
GB0616797A GB2429596B (en) | 2005-08-26 | 2006-08-24 | Method and system for determining the motion of an imaging apparatus |
DE102006039946A DE102006039946A1 (en) | 2005-08-26 | 2006-08-25 | Method and system for determining the motion of an image capture device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/212,460 US7777781B2 (en) | 2005-08-26 | 2005-08-26 | Method and system for determining the motion of an imaging apparatus |
Publications (2)
Publication Number | Publication Date |
---|---|
US20070046782A1 true US20070046782A1 (en) | 2007-03-01 |
US7777781B2 US7777781B2 (en) | 2010-08-17 |
Family
ID=37102778
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/212,460 Expired - Fee Related US7777781B2 (en) | 2005-08-26 | 2005-08-26 | Method and system for determining the motion of an imaging apparatus |
Country Status (6)
Country | Link |
---|---|
US (1) | US7777781B2 (en) |
JP (1) | JP4963384B2 (en) |
CN (1) | CN1921565B (en) |
DE (1) | DE102006039946A1 (en) |
GB (1) | GB2429596B (en) |
TW (1) | TWI394435B (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070248167A1 (en) * | 2006-02-27 | 2007-10-25 | Jun-Hyun Park | Image stabilizer, system having the same and method of stabilizing an image |
US20080030586A1 (en) * | 2006-08-07 | 2008-02-07 | Rene Helbing | Optical motion sensing |
US20080136932A1 (en) * | 2003-02-25 | 2008-06-12 | Matsushita Electric Industrial Co., Ltd. | Image Pickup Processing Method and Image Pickup Apparatus |
US20080260366A1 (en) * | 2007-04-19 | 2008-10-23 | Michael John Brosnan | Image stabilization with adaptive shutter control |
DE102008037013A1 (en) | 2007-08-09 | 2009-02-12 | Avago Technologies Ecbu Ip (Singapore) Pte. Ltd. | Image stabilization with user feedback |
US20100302383A1 (en) * | 2005-09-21 | 2010-12-02 | Chiou-Shann Fuh | Image Processing Method and Apparatus Thereof |
US20120275521A1 (en) * | 2010-08-02 | 2012-11-01 | Bin Cui | Representative Motion Flow Extraction for Effective Video Classification and Retrieval |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8199212B2 (en) * | 2008-05-03 | 2012-06-12 | Olympus Imaging Corp. | Image recording and playback device, and image recording and playback method |
JP5725725B2 (en) | 2010-04-07 | 2015-05-27 | 三菱日立パワーシステムズ株式会社 | Wet flue gas desulfurization equipment |
US9160917B2 (en) | 2013-07-09 | 2015-10-13 | Htc Corporation | Handheld electronic apparatus and image capturing apparatus and focusing method thereof |
US20130208107A1 (en) * | 2012-02-14 | 2013-08-15 | Nokia Corporation | Apparatus and a Method for Producing a Depth-Map |
US8988341B2 (en) | 2012-06-08 | 2015-03-24 | Apple Inc. | Camera-assisted motion estimation for application control |
KR101357659B1 (en) | 2013-05-29 | 2014-02-03 | 주식회사 비전테크 | Apparatus for monitoring edge |
JP6604998B2 (en) * | 2017-07-20 | 2019-11-13 | キヤノン株式会社 | Image processing apparatus, imaging apparatus, image processing apparatus control method, and program |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4340731A (en) * | 1979-06-20 | 1982-07-20 | Montedison S.P.A. | Polycarboxyalkyl-cellulose having high fluid absorbing and retaining properties, and process for preparing same |
US4525039A (en) * | 1982-02-19 | 1985-06-25 | Defuans Jean Louis | Objective lens |
US5786847A (en) * | 1995-06-09 | 1998-07-28 | Canon Kabushiki Kaisha | Apparatus for adjusting the convergence in a stereoscopic image system |
US5977535A (en) * | 1992-09-30 | 1999-11-02 | Lsi Logic Corporation | Light sensing device having an array of photosensitive elements coincident with an array of lens formed on an optically transmissive material |
US6744931B2 (en) * | 1992-04-09 | 2004-06-01 | Olympus Optical Co., Ltd. | Image processing apparatus |
US20040212677A1 (en) * | 2003-04-25 | 2004-10-28 | Uebbing John J. | Motion detecting camera system |
US20060043515A1 (en) * | 2004-08-25 | 2006-03-02 | Loriston Ford | Light block for pixel arrays |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2902966B2 (en) * | 1994-12-16 | 1999-06-07 | 三洋電機株式会社 | Camera shake correction device and video camera using the same |
JP3424440B2 (en) | 1996-06-11 | 2003-07-07 | ミノルタ株式会社 | Camera with image stabilization function |
KR100683849B1 (en) * | 2000-06-28 | 2007-02-15 | 삼성전자주식회사 | Decoder having digital image stabilization function and digital image stabilization method |
JP3601506B2 (en) | 2001-11-08 | 2004-12-15 | ソニー株式会社 | Apparatus and method for detecting motion vector, and apparatus and method for correcting image vibration |
CN1332556C (en) * | 2002-12-26 | 2007-08-15 | 三菱电机株式会社 | Image processor |
JP4258383B2 (en) * | 2004-01-14 | 2009-04-30 | パナソニック株式会社 | Imaging device |
-
2005
- 2005-08-26 US US11/212,460 patent/US7777781B2/en not_active Expired - Fee Related
-
2006
- 2006-08-18 TW TW095130406A patent/TWI394435B/en active
- 2006-08-23 CN CN2006101115718A patent/CN1921565B/en not_active Expired - Fee Related
- 2006-08-24 JP JP2006227558A patent/JP4963384B2/en active Active
- 2006-08-24 GB GB0616797A patent/GB2429596B/en not_active Expired - Fee Related
- 2006-08-25 DE DE102006039946A patent/DE102006039946A1/en not_active Withdrawn
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4340731A (en) * | 1979-06-20 | 1982-07-20 | Montedison S.P.A. | Polycarboxyalkyl-cellulose having high fluid absorbing and retaining properties, and process for preparing same |
US4525039A (en) * | 1982-02-19 | 1985-06-25 | Defuans Jean Louis | Objective lens |
US6744931B2 (en) * | 1992-04-09 | 2004-06-01 | Olympus Optical Co., Ltd. | Image processing apparatus |
US5977535A (en) * | 1992-09-30 | 1999-11-02 | Lsi Logic Corporation | Light sensing device having an array of photosensitive elements coincident with an array of lens formed on an optically transmissive material |
US5786847A (en) * | 1995-06-09 | 1998-07-28 | Canon Kabushiki Kaisha | Apparatus for adjusting the convergence in a stereoscopic image system |
US20040212677A1 (en) * | 2003-04-25 | 2004-10-28 | Uebbing John J. | Motion detecting camera system |
US20060043515A1 (en) * | 2004-08-25 | 2006-03-02 | Loriston Ford | Light block for pixel arrays |
Cited By (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080136932A1 (en) * | 2003-02-25 | 2008-06-12 | Matsushita Electric Industrial Co., Ltd. | Image Pickup Processing Method and Image Pickup Apparatus |
US7859568B2 (en) * | 2003-02-25 | 2010-12-28 | Panasonic Corporation | Image capturing processing method and system for performing roll correction |
US20100302383A1 (en) * | 2005-09-21 | 2010-12-02 | Chiou-Shann Fuh | Image Processing Method and Apparatus Thereof |
US7944475B2 (en) * | 2005-09-21 | 2011-05-17 | Inventec Appliances Corp. | Image processing system using motion vectors and predetermined ratio |
US20070248167A1 (en) * | 2006-02-27 | 2007-10-25 | Jun-Hyun Park | Image stabilizer, system having the same and method of stabilizing an image |
US20080030586A1 (en) * | 2006-08-07 | 2008-02-07 | Rene Helbing | Optical motion sensing |
US8013895B2 (en) * | 2006-08-07 | 2011-09-06 | Avago Technologies General Ip (Singapore) Pte. Ltd. | Optical motion sensing |
US20080260366A1 (en) * | 2007-04-19 | 2008-10-23 | Michael John Brosnan | Image stabilization with adaptive shutter control |
US7734161B2 (en) | 2007-04-19 | 2010-06-08 | Avago Technologies Ecbu Ip (Singapore) Pte. Ltd. | Image stabilization with adaptive shutter control |
DE102008019410A1 (en) | 2007-04-19 | 2008-10-23 | Avago Technologies Ecbu Ip (Singapore) Pte. Ltd. | Image stabilization with adaptive shutter control |
US20090040318A1 (en) * | 2007-08-09 | 2009-02-12 | Michael John Brosnan | Image stabilization with user feedback |
DE102008037013A1 (en) | 2007-08-09 | 2009-02-12 | Avago Technologies Ecbu Ip (Singapore) Pte. Ltd. | Image stabilization with user feedback |
US20120275521A1 (en) * | 2010-08-02 | 2012-11-01 | Bin Cui | Representative Motion Flow Extraction for Effective Video Classification and Retrieval |
US8995531B2 (en) * | 2010-08-02 | 2015-03-31 | Peking University | Representative motion flow extraction for effective video classification and retrieval |
US9268794B2 (en) | 2010-08-02 | 2016-02-23 | Peking University | Representative motion flow extraction for effective video classification and retrieval |
Also Published As
Publication number | Publication date |
---|---|
TW200715832A (en) | 2007-04-16 |
CN1921565B (en) | 2010-12-08 |
JP2007068168A (en) | 2007-03-15 |
GB0616797D0 (en) | 2006-10-04 |
GB2429596A (en) | 2007-02-28 |
US7777781B2 (en) | 2010-08-17 |
GB2429596B (en) | 2010-11-24 |
CN1921565A (en) | 2007-02-28 |
TWI394435B (en) | 2013-04-21 |
JP4963384B2 (en) | 2012-06-27 |
DE102006039946A1 (en) | 2007-03-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7777781B2 (en) | Method and system for determining the motion of an imaging apparatus | |
JP3770271B2 (en) | Image processing device | |
CN100478728C (en) | System and method for image capture device | |
JP4956401B2 (en) | Imaging apparatus, control method thereof, and program | |
US8929452B2 (en) | Image processing apparatus, image capturing apparatus, and computer program | |
US9413923B2 (en) | Imaging apparatus | |
JP4434939B2 (en) | Imaging apparatus and control method thereof | |
CN109417593B (en) | Imaging device, operation method, image processing device, and image processing method | |
JP2006295626A (en) | Fish-eye image processing apparatus, method thereof and fish-eye imaging apparatus | |
US8310553B2 (en) | Image capturing device, image capturing method, and storage medium having stored therein image capturing program | |
US20090135261A1 (en) | Imaging apparatus and recording medium | |
JP2013165485A (en) | Image processing apparatus, image capturing apparatus, and computer program | |
JP2006186592A5 (en) | ||
KR102592745B1 (en) | Posture estimating apparatus, posture estimating method and computer program stored in recording medium | |
JP2012133194A (en) | Imaging apparatus | |
JP6362473B2 (en) | Image processing apparatus, imaging apparatus, image processing method, program, and storage medium | |
JP2004289709A (en) | Image pickup device, and image pickup method | |
WO2016009199A2 (en) | Minimisation of blur in still image capture | |
JP2008053787A (en) | Multiple-lens electronic camera and parallax correcting method of multi-lens electronic camera | |
JP3257035B2 (en) | Apparatus and method for correcting image vibration | |
US8619149B2 (en) | Camera with an image sensor that doubles as a motion sensor | |
JP6728005B2 (en) | Imaging device, imaging method, and program | |
CN111698427B (en) | Image processing method and device and electronic equipment | |
JP2007214666A (en) | Digital camera with moving picture photographing function | |
US20130136340A1 (en) | Arithmetic processing device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: AGILENT TECHNOLOGIES, INC., COLORADO Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HELBING, RENE P.;TRUTNA, JR., WILLIAM R.;REEL/FRAME:017133/0670 Effective date: 20050826 |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES GENERAL IP PTE. LTD.,SINGAPORE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AGILENT TECHNOLOGIES, INC.;REEL/FRAME:017206/0666 Effective date: 20051201 Owner name: AVAGO TECHNOLOGIES GENERAL IP PTE. LTD., SINGAPORE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AGILENT TECHNOLOGIES, INC.;REEL/FRAME:017206/0666 Effective date: 20051201 |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES ECBU IP (SINGAPORE) PTE. LTD.,S Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:017675/0518 Effective date: 20060127 Owner name: AVAGO TECHNOLOGIES ECBU IP (SINGAPORE) PTE. LTD., Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:017675/0518 Effective date: 20060127 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD Free format text: MERGER;ASSIGNOR:AVAGO TECHNOLOGIES ECBU IP (SINGAPORE) PTE. LTD.;REEL/FRAME:030369/0528 Effective date: 20121030 |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
AS | Assignment |
Owner name: DEUTSCHE BANK AG NEW YORK BRANCH, AS COLLATERAL AG Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:032851/0001 Effective date: 20140506 Owner name: DEUTSCHE BANK AG NEW YORK BRANCH, AS COLLATERAL AGENT, NEW YORK Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:032851/0001 Effective date: 20140506 |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD., SINGAPORE Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENT RIGHTS (RELEASES RF 032851-0001);ASSIGNOR:DEUTSCHE BANK AG NEW YORK BRANCH, AS COLLATERAL AGENT;REEL/FRAME:037689/0001 Effective date: 20160201 Owner name: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENT RIGHTS (RELEASES RF 032851-0001);ASSIGNOR:DEUTSCHE BANK AG NEW YORK BRANCH, AS COLLATERAL AGENT;REEL/FRAME:037689/0001 Effective date: 20160201 |
|
AS | Assignment |
Owner name: BANK OF AMERICA, N.A., AS COLLATERAL AGENT, NORTH CAROLINA Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:037808/0001 Effective date: 20160201 Owner name: BANK OF AMERICA, N.A., AS COLLATERAL AGENT, NORTH Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:037808/0001 Effective date: 20160201 |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE NAME PREVIOUSLY RECORDED AT REEL: 017206 FRAME: 0666. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:AGILENT TECHNOLOGIES, INC.;REEL/FRAME:038632/0662 Effective date: 20051201 |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD., SINGAPORE Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS COLLATERAL AGENT;REEL/FRAME:041710/0001 Effective date: 20170119 Owner name: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS COLLATERAL AGENT;REEL/FRAME:041710/0001 Effective date: 20170119 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552) Year of fee payment: 8 |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITE Free format text: MERGER;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:047196/0687 Effective date: 20180509 |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITE Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE EFFECTIVE DATE OF MERGER TO 9/5/2018 PREVIOUSLY RECORDED AT REEL: 047196 FRAME: 0687. ASSIGNOR(S) HEREBY CONFIRMS THE MERGER;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:047630/0344 Effective date: 20180905 |
|
AS | Assignment |
Owner name: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITE Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE PROPERTY NUMBERS PREVIOUSLY RECORDED AT REEL: 47630 FRAME: 344. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNOR:AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.;REEL/FRAME:048883/0267 Effective date: 20180905 |
|
AS | Assignment |
Owner name: PIXART IMAGING INC., TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BROADCOM INC.;REEL/FRAME:048824/0863 Effective date: 20190322 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20220817 |