CN101911701A - Shared block comparison architecture for image registration and video coding - Google Patents
Shared block comparison architecture for image registration and video coding Download PDFInfo
- Publication number
- CN101911701A CN101911701A CN2009801014988A CN200980101498A CN101911701A CN 101911701 A CN101911701 A CN 101911701A CN 2009801014988 A CN2009801014988 A CN 2009801014988A CN 200980101498 A CN200980101498 A CN 200980101498A CN 101911701 A CN101911701 A CN 101911701A
- Authority
- CN
- China
- Prior art keywords
- image
- registration
- piece
- block
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/537—Motion estimation other than block-based
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
- H04N23/684—Vibration or motion blur correction performed by controlling the image sensor readout, e.g. by controlling the integration time
- H04N23/6845—Vibration or motion blur correction performed by controlling the image sensor readout, e.g. by controlling the integration time by combination of a plurality of images sequentially taken
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/745—Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
- H04N19/43—Hardware specially adapted for motion estimation or compensation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
- Studio Devices (AREA)
Abstract
This disclosure describes an efficient architecture for an imaging device that supports image registration for still images and video coding of a video sequence. For image registration, the described architecture uses block-based comparisons of image blocks of a captured image relative to blocks of another reference image to support image registration on a block by block basis. For video coding, the described architecture uses block-based comparisons, e.g., to support for motion estimation and motion compensation. According to this disclosure, a common block comparison engine is used on a shared basis for both block based image registration and block-based video coding. In this way, a hardware unit designed for block based comparisons may be implemented so as to work in both the image registration process for still images and the video coding process for coding a video sequence.
Description
Technical field
The present invention relates to image and Video processing, and more particularly, relate to the technology of the video coding that is used for the image registration of using and stable technology and is used for using in video capture at image capture.
Background technology
In imaging device, image that a little thus devices are captured or video sequence often are subject to the degrading quality owing to the involuntary vibration of installing during image capture.In video capture, the vibration of imaging device can cause unacceptable motion or the shake in institute's capture video sequence.In still image was captured, vibration can cause the fuzzy of institute's capture images.Fuzzy or shake can make people's vision system be difficult to focus on and concentrate on the image interior interesting areas or target.Under any circumstance, the quality of the viewing experience of video or image reduces.
Image registration techniques can be in order to compensate unintentional vibration to produce the better pictures quality.Figure registration system is divided into three primary categories substantially: optical image stabilizer (OIS), mechanical image stabilizer (MIS) and electronic image stabilizer (EIS).The OIS system uses adjustable lens, and it made the effect of anamorphose to reduce to vibrate before image arrives transducer.Mis system (for example) uses the health of center of gravity, the system that contends with and/or the camera operator of camera to make whole camera stable with minimized vibrations.The EIS system uses signal processing algorithm to change image of being captured and the effect that solves vibration whereby.In some image processing were used, image registration also can be fused into a picture in order to the short sequence with image.EIS system implementation get up comparable OIS system and mis system cheapness, and may be better than for example being equipped with the OIS system and the mis system of device of the wireless radio telephone hand-held set (for example, so-called camera phone) of imaging capability.
For video capture, imaging device can be implemented block-based video compression technology, for example by advanced video coding (AVC) standard, and the 10th part, H.261, H.263 or H.264/MPEG-4 defined those technology of MPEG-2, MPEG-4, ITU-T.Video compression technology carry out spatial prediction and time prediction with reduce or remove in video sequence intrinsic redundancy.Estimation and motion compensation (for example) are in order to remove the time redundancy in the encoded video sequence and to realize the common and effective block-based video coding technique of data compression whereby.
Summary of the invention
The present invention describes effective framework of imaging device of the video coding of the image registration of supporting still image and video sequence.For image registration, the image block that described framework can use the image through capturing with respect to the block-based comparison of the piece of another reference picture to be supported in the image registration on the block-by-block basis.Randomly, also can be used for based on the registration of pixel in any image registration, for described any, the piece of unidentified abundant coupling in reference picture.For video coding, described framework can use block-based comparison (for example) to support estimation and motion compensation.
According to certain aspects of the invention, can carry out block-based comparison via piece comparison engine, described comparison engine forms the part of image registration framework and video coding framework.In other words, the present invention describes the relatively use of engine of shared piece that is used for block-based image registration and block-based video coding.In this way, can implement the hardware cell that designs at block-based comparison, so that work at process of image registration that is used for still image and the video coding process that is used for encoded video sequence.
In an example, the invention provides a kind of method, it comprises: the picture frame to video sequence is carried out video coding process, wherein carries out video coding process and comprises that use piece comparison engine comes the video blocks of more encoded picture frame and the video blocks of search volume; And the carries out image registration process, wherein the carries out image registration process comprises that described comparison of use engine comes the piece of comparison first image and the piece of second image.
In another example, the invention provides a kind of equipment, it comprises: piece is engine relatively; Video encoder, its picture frame to video sequence is carried out video coding process, and wherein video coding process comprises that use piece comparison engine comes the video blocks of more encoded picture frame and the video blocks of search volume; And the image registration unit, its carries out image registration process, wherein said process of image registration comprise uses piece comparison engine to come the piece of comparison first image and the piece of second image.
In another example, the invention provides a kind of device, it comprises: be used for the picture frame of video sequence is carried out the device of video coding process, the device that wherein is used to carry out video coding process comprises the device of the video blocks of the video blocks that is used to use piece comparison engine to come more encoded picture frame and search volume; And the device that is used for the carries out image registration process, the device that wherein is used for the carries out image registration process comprises the device of the piece of the piece that is used to use piece comparison engine to come comparison first image and second image.
Can hardware, software, firmware or its combination implement the various aspects of technology described in the present invention.If with software implementation, then software can be executed in the processor, described processor can refer to one or more processors, for example, microprocessor, application-specific integrated circuit (ASIC) (ASIC), field programmable gate array (FPGA) or digital signal processor (DSP), or the integrated or discrete logic of other equivalence.Software can be executed in and comprise that piece compares in the imaging device of engine, as described herein.The software instruction of carrying out described technology can be initially stored in the computer-readable media and be loaded and carried out by the processor of imaging device.
Therefore, the computer-readable media of the computer-readable media that comprises instruction is also contained in the present invention, described instruction just causes described processor after carrying out in the processor of the imaging device that is comprising piece comparison engine: the picture frame to video sequence is carried out video coding process, and wherein said video coding process comprises that the use piece compares the video blocks of the more encoded picture frame of engine and the video blocks of search volume; And the carries out image registration process, wherein block-based process of image registration comprises uses piece comparison engine to come the piece of comparison first image and the piece of second image.
Set forth the details of one or more examples in the accompanying drawings and the description below.Will be from describing and graphic and accessory rights claim is understood other features, objects and advantages.
Description of drawings
The piece that Fig. 1 is used by image registration unit and video encoder for explanation comprises is the block diagram of the example imaging device of engine relatively.
Fig. 2 is the block diagram of the exemplary image registration unit that be coupled to memory consistent with technology of the present invention.
Fig. 3 is the block diagram of the exemplary video encoder that be coupled to memory consistent with technology of the present invention.
Fig. 4 is the flow chart of explanation according to the example operation of the imaging device of carries out image registration of the present invention and video coding.
Fig. 5 is the flow chart of the explanation exemplary image registration process consistent with the present invention.
Embodiment
Imaging device (those especially little and light imaging devices, for example so-called camera phone) is subject to during image capture the unacceptable moving influence that the user by device produces.These a little unacceptable moving (being sometimes referred to as the hand shake) cause the picture quality that reduces substantially owing to fuzzy or other visual artifacts.For compensating described moving, imaging device can use the image registration techniques of describing among the present invention that image stabilization is provided.In general, image registration refers to a technology, in described technology two or more still images through capture, through aiming at and combination compound to form (" registration ") still image.With respect to described two or more still images, image registration can be in order to reducing the shake in the still image of combination, or be fused into a picture in order to make up two or more images with the sequence with image.
For image registration, the image block that framework as herein described uses the image through capturing with respect to the block-based comparison of the piece of another reference picture to be supported in the image registration on the block-by-block basis.Two or more images can be divided into some or window, it can be similar to so-called " macro block " that defines in the video coding.Can be with the piece of these pieces or window and another image or window comparison to be supported in the image registration on the block-by-block basis.Described can be disjoint, (non-overlapped) that engage or overlapping.Randomly, also can be used for any based on the registration of pixel, for described any, the piece of unidentified abundant coupling in reference picture.
The present invention describe a kind of share use on the basis common piece relatively engine support effective framework of image capture device of the video coding of the image registration of still image and video sequence.Compare engine by image registration and the shared piece of video coding, the size, processing and the memory spending that reduce in the imaging device can become possibility.
For video coding, described framework also uses block-based comparison (for example) to support estimation and motion compensation.According to the present invention, as described above, can carry out block-based comparison via piece comparison engine, described comparison engine forms the shared portion of image registration framework and video coding framework.In other words, the present invention describes and to be used for the relatively use of engine of shared piece used at block-based image registration and block-based video coding.In this way, can implement the hardware cell that designs at block-based comparison, so that work with the video coding process that is used for encoded video sequence in the process of image registration of still image.Process of image registration also can be used together in conjunction with video capture, wherein the frame through registration of each picture frame of video sequence for being associated with a plurality of short exposure images.In the case, the video coding process of the image registration that is associated with the generation of picture frame and these a little picture frames of being used for coded sequence will use same relatively engine.
Fig. 1 is the block diagram of illustrated example imaging device 2, and described imaging device 2 comprises that the piece by image registration unit 16 and video encoder 18 both uses compares engine 20.Although piece comparison engine 20 is illustrated as independent assembly, in fact it can form both parts of image registration unit 16 and video encoder 18.Piece comparison engine 20 can be included in the hardware cell that is called and called by video encoder 18 by image registration unit 16 during the image registration during video coding.Perhaps, piece comparison engine 20 can comprise hardware cell, and both are integrated with image registration unit 16 and video encoder 18 in fact for described hardware cell, makes it form both parts of image registration unit 16 and video encoder 18.Can hardware, software, firmware or its any combination implement image registration unit 16 and video encoder 18.
Imaging device 2 can comprise digital camera, and it supports one or more digital still image capture patterns and video capture pattern.Imaging device 2 can comprise independently imaging device (for example, independently camera), maybe can form for example part of another device of wireless handset (being commonly referred to the celluler radio phone).As an example, imaging device 2 can comprise so-called camera phone or visual telephone.In the case, device 2 can comprise that many other assemblies (for example, receiver, reflector, modulator-demodulator and antenna) are to promote radio communication.For simple and be easy to not show these wireless communication components for the purpose of the explanation.
As shown in fig. 1, imaging device 2 can comprise the camera 10 that is coupled to graphics processing unit 12 via dedicated bus 11.Image device 2 can define is captured by camera 10 along with still image or frame of video and handles " embedded (in-line) " framework of described image and/or frame automatically.Yet technology of the present invention does not need camera 10 and processing unit 12.On the contrary, the data that send with respect to the data of having been captured by another device, from another device, be stored in data the memory 14 etc., also can use video coding described herein and process of image registration, it uses same relatively engine.
For capturing an image, camera 10 is exposed to image scene with its sensor element.Image sensor element in the camera 10 can (for example) be captured the intensity level of the light intensity of the scene that is illustrated in the specific pixel location place.In some cases, each in the image sensor element of camera 10 be attributable to covering sensor colored filter and only to a kind of color or colour band sensitivity.For instance, camera 10 can comprise the array of the transducer with suitable filter so that define redness, green and blue sensor.Yet camera 10 can utilize the colored filter of other type, for example the colored filter of CMYK colored filter or other type.In the imageing sensor of camera 10 each can be captured for a kind of intensity level of color only.Image information can comprise the pixel intensity and/or the color-values of being captured by the sensor element of camera 10.
Can during video capture, use video encoder 18.Video encoder 18 can be according to for example advanced video coding (AVC), the 10th part, and H.261, H.263 or H.264/MPEG-4 MPEG-2, MPEG-4, ITU-T wait standard to carry out video compression.In this way, the sequence of video encoder 18 compressed video frames is to form encoded video sequence.And image registration need not to be limited to the image capture pattern, but also can be used in the video capture pattern, makes the frame through registration of each picture frame for producing based on twice or twice above image exposure of video sequence.
According to technology of the present invention, both use relatively engine 20 of piece by image registration unit 16 and video encoder 18.And although piece comparison engine 20 is illustrated as independent assembly, in fact it can form both parts of image registration unit 16 and video encoder 18.Perhaps, piece comparison engine 20 can be regarded as the independent assembly that called and called by video encoder 18 by image registration unit 16 during the image registration during video coding.In a word, image registration unit 16 and video encoder 18 both all utilize relatively engine 20 of piece.In this way, can in imaging device 2, avoid the relatively repetition of engine 20 of piece, so that it is more effective to install 2 framework.For instance, the piece comparison engine 20 of Gong Xianging can produce and have than using independent piece to compare the integrated circuit of chip area (less silicon) under the situation of engine that the chip area of needs is little.With respect to software solution only, also can by implement piece relatively engine 20 reduce storage requirement.In addition, with respect to other framework or software solution only, can improve processing speed.
In the process of the piece of piece and second image of combination first image, but in front the pixel value of pixel of image on pixel value of the pixel of the image that image registration unit 16 equalizations are captured recently (for example, intensity and/or color-values) and time.Image registration unit 16 can use directly average, first image of first image and second image and second image weighted average, first image and second image and, or be used to make up the image information of two or more images any other technology for image registration.Image registration unit 16 can being stored in the memory 14 through the equalization pixel value the pixel position of the image of capturing recently.In this way, image registration unit 16 can during the image registration to substitute the pixel value of one pixel in capture images through the equalization pixel value.This through equalization image information (that is, pixel value) represent usually than the image information of registration not lack fuzzy, provide the picture quality of enhancing thus, especially in vibration mentioned above or mobile example occurring.Also can cause through the picture showing of registration than lack fuzzy of first image and second image and the noise that may lack through the equalization view data.
According to the present invention, image registration can take place on block-by-block basis.Yet in some cases, if in described some, the piece of unidentified abundant coupling can be carried out comparison by pixel to these a little pieces by image registration unit 16.Image registration can combine to reduce noise in order to the repeatedly short exposure of combine scenes and with these multiexposure, multiple exposures, increases signal and improves image sharpness.For example illumination difference and what comes into a driver's displacement problems such as (rotation, translations) is the challenge of handling at this type.The additional detail of process of image registration hereinafter is provided.
For time prediction, video encoder 18 is carried out estimation to follow the trail of moving of match video piece between two or more contiguous frames.Video encoder 18 can define motion vector with the displacement of instruction video piece with respect to the corresponding predicted video block in one or more reference frames of search volume.Motion compensation uses motion vector with from reference frame identification prediction video blocks.Form remaining video blocks via subtraction (for example, by deducting predicted video block) from original video block to be encoded.Remaining video blocks can be sent to Video Decoder together with motion vector.
In order to carry out the necessary block-based comparison of spatial prediction and time prediction, video encoder 18 uses by image registration unit 16 and uses in image registration to be used for same comparison engine 20 of block-based comparison.Yet video encoder 18 deducts piece with the formation residual block from the piece of data, but not the piece of data is made up (as being undertaken by image registration unit 16).Because the block-based compare operation of carrying out at image registration is similar to the block-based compare operation of carrying out at video coding, so can both use same relatively engine 20 by image registration unit 16 and video encoder 18.And the piece comparison engine of sharing 20 can produce to have than using independent piece to compare the integrated circuit (IC) apparatus of chip area (less silicon) under the situation of engine that the chip area of needs is little.With respect to software solution only, also can by implement piece relatively engine 20 reduce storage requirement.In addition, with respect to other framework or software solution only, can improve processing speed.
In discussing hereinafter, video encoder 18 mainly is described as time of implementation prediction, for example estimation and motion compensation.Yet video encoder 18 is gone back the executable space prediction, and can implement piece comparison engine 20 to be used for this spatial prediction.Yet, in the case, the piece of given frame of video and other piece of same frame of video are compared, can be to define in order to the similarity of packed data spatially.
Fig. 2 is the block diagram of the exemplary image registration unit 16 that be coupled to memory 14 consistent with technology of the present invention.Illustrated assembly among Fig. 2 is an example of image registration unit 16 shown in Fig. 1 and memory 14.As shown in Figure 2, memory 14 receptions are from the view data of graphics processing unit (for example, the unit 12 of Fig. 1).In Fig. 2 and Fig. 3, same relatively engine 20 is described.Dotted line compares engine 20 around piece and compares the part that engine 20 forms image registration unit 16 (Fig. 2) with illustrated block, and forms the idea of the part of video encoder 18 (Fig. 3).As mentioned above, piece comparison engine 20 can be considered as maybe it can being considered as the hardware cell of part through being integrated into image registration unit 16 and video encoder 18 by image registration unit 16 and the engine that calls by video encoder 18.
Mode that can block-by-block receives input at (ping pong) input buffer 22 places alternately in memory.Alternately input buffer 22 can comprise so-called " replacing " buffer arrangement.In the case, data only are stored in the buffer 22 at that time at needs, and in case handle these data just with the data overwrite in image registration unit 16.For instance, first image block can be stored in alternately in the input buffer 22 and and be forwarded to image registration unit 16 for processing with it.At this moment, can be in input buffer 22 alternately receive second of data from graphics processing unit 12.In case first of data handle, just second of data can be forwarded to image registration unit 16 by image registration unit 16.At this moment, can in input buffer 22 alternately, receive the 3rd of data.In addition, but this 3rd overwrite of data replaces first of data in the input buffer 22.This realizes effective use of memory 14, makes that data just need not to be stored in the memory 14 in case by image registration unit 16 deal with data.Can between memory 14 and image registration unit 16, transmit data via data communication bus 15.
Alternately input buffer 22 can comprise the memory space of the piece that is used for any number, in case but generally support reduced data piece toilet cover to write this a little pieces.As further explanation, alternately input buffer 22 can comprise two or more parallel buffers.When being written to a buffer, can be from another buffer reading of data, and vice versa.Therefore, another buffer of overwrite simultaneously substantially when alternate buffering device 22 can be supported in and read from a buffer, make read operation and write operation between two or more parallel buffers on basis alternately alternately.
The SAD technology relates to the task of carrying out absolute difference computation between the pixel value of the piece of the value of the piece of first image and second (reference) image.With result's summation (that is, adding up) of these absolute difference computation, so that define the value of the difference between indicating described.As an example, take advantage of 8 block of pixels for 8, can calculate 64 differences and, and take advantage of 16 block of pixels for 16, can calculate 256 differences and its summation to its summation.Low difference is indicated substantially with relatively one of another piece and is better coupling.
The SSD technology also relates to carries out the task that difference is calculated between the pixel value of the piece of the pixel value of the piece of first image and second (reference) image.In the SSD technology, with the result of absolute difference computation square, and then with square value summation (that is, adding up), so that define the difference of the difference between described of the indication.Perhaps, piece comparison engine 20 can be implemented other comparison techniques, for example mean square error (MSE), Normalized Cross Correlation Function (NCCF) or another suitable comparison techniques.
If by the abundant coupling of piece matching unit 27 identification, then the match block of the piece of piece combiner 28 combinations first image and second image with (for example) by equalization or otherwise the value of the respective pixel in the combination block produce piece through registration.Then will send to memory 14 from the piece combiner 28 of image registration unit through the piece of registration, wherein piece combiner 28 is in order to the part of assembling through the image 26 of registration.For combination block, as mentioned above, piece combiner 28 can be carried out add operation, equalization computing, weighted average computing, or is generally used for any computing in order to the respective pixel value in the combinations matches piece in the process of image registration.
On the other hand, if block matching unit 27 unidentified abundant couplings, one in then several different possibilities may take place.In an example, if block matching unit 27 unidentified abundant couplings, then piece combiner 28 can be avoided any combination, and the piece of first image directly can be forwarded to memory 14 for using in the process of the image 26 of registration in assembling.In the case, can only comprise the data that are associated with the piece of first image, and not have any combination with the piece of second image through that the given of image of registration.In some cases, if block combiner 28 is because of any former thereby avoid combination, and then selected be selected and be transmitted to piece combiner 28 can between the piece of the piece of first image and second image.In the case, can block-based known or measured character select, for example, when avoiding making up, can transmit the piece that represents minimal noise.
Perhaps, if block matching unit 27 unidentified abundant couplings, then piece combiner 28 can be forwarded to combination of pixels device 29 with the piece and the optimum Match (even good inadequately for the piece combination) of first image.In the case, combination of pixels device 29 can be carried out the comparison based on pixel of the piece of first image with respect to the pixel of the best matching blocks that is compared engine 20 and 27 identifications of piece matching unit by piece.The pixel of the pixel of the abundant coupling of the piece of combination of pixels device 29 first images capable of being combined and the best matching blocks of second image.Yet any pixel that does not have abundant coupling may not make up yet.Generation have one or more through packed-pixel after registration block, combination of pixels device 29 can be forwarded to memory with the piece through registration, wherein through the piece of registration through storage in the process of the image 26 of registration, using in assembling.Combination of pixels device 29 is optional, and can remove in certain embodiments.
In some cases, in case through the image 26 of registration through assembling fully, through the image 26 of registration then just as reference picture 24.If combination three or three are with epigraph in process of image registration, then this is particularly useful.In the case, make up preceding two images to produce the image 26 through registration in the mode of block-by-block, it then becomes reference picture 24.Then, with the 3rd image and reference picture 24 compares and with its combination to produce new image 26 through registration, this stylish image 26 through registration is the combination of three images.This process can continue on for waiting being combined into the image through any number of the image of registration.If (for example) with respect to the available and image through capturing in advance of not capturing in real time or data and the off-line execution registration process then can be considered as first reference picture with any image.
As mentioned above, piece comparison engine 20 not only is used for the block-based comparison at image registration, and is used for the block-based comparison at video coding.Fig. 3 is the block diagram of the exemplary video encoder 18 that be coupled to memory 14 consistent with technology of the present invention.Illustrated assembly among Fig. 3 is an example of video encoder 18 shown in Fig. 1 and memory 14.As shown in Figure 3, memory 14 receives data from graphics processing unit (for example, the unit 12 of Fig. 1).These data are the sequence of the picture frame of formation video sequence.
Can be in the reception input of input buffer 32 places in memory.Input buffer 32 can be similar to alternately input buffer 22, and can comprise so-called " replacing " buffer arrangement, (for example) as described previously.Yet, can arrange input buffer 32 according to the buffer arrangement of many other types.
For video coding, memory 14 is stored in the piece of first picture frame in the input buffer 32, and memory search space 34.Search volume 34 can comprise the set corresponding to the pixel (or piece of pixel) of the picture frame of the previous coding of video sequence (or for bi-directional predicted possible subsequent frame).The search volume can comprise whole previous frames or subsequent frame, or the subclass of frame (if necessary).The search volume can be rectangular shape maybe can adopt in the extensive multiple shape and size any one.
Depend on time prediction to reduce or to remove time redundancy between the piece of successive image frame of video sequence via the interframe encode of ME unit 42 and mc unit 44.For interframe encode, moving of match block between two or more contiguous frames followed the trail of in ME unit 42.Estimation produces motion vector, and its indicator collet is with respect to the displacement of the prediction of the correspondence in one or more reference frames piece.Mc unit 44 uses motion vector to produce the prediction piece from reference frame.After motion compensation, difference unit 46 produces residual block by deduct the prediction piece from original block to be encoded.Can be sent to memory 14 from video encoder 16 with residual block with in order to the motion vector that produces residual block, wherein these data are through storing and be assembled into the residual data 36 of the encoded frame of video sequence.
And according to the present invention, the same comparison engine 20 that is used by image registration unit 16 also used to promote piece relatively by video encoder 18.Yet in video coding, piece relatively promotes the block-based coding of video sequence but not image registration.SAD technology, SSD technology or any suitable piece comparison techniques that piece comparison engine 20 can be implemented as mentioned to be summarized.
If desired, video encoder 18 can be used conversion, quantification and the entropy coding process bit rate further to reduce to be associated with residual block.Converter technique can comprise discrete cosine transform (DCT).Quantification can comprise to be compressed to the level of resolution that reduces coefficient quantization to improve.Entropy coding can comprise variable length code, binary arithmetic coding (CABAC), huffman coding, run-length encoding, coded block pattern (CBP) coding etc.For simple and be easy to for the purpose of the explanation undeclared these additional assemblies among Fig. 3.But the data compression of these or other coding method auxiliary residual data is especially will be under emission data conditions between the device.
Fig. 4 is the flow chart of explanation according to the example operation of the imaging device 2 of carries out image registration of the present invention and video coding.As mentioned above, technology of the present invention is without any need for image capture, but the data that can send with respect to the data of having been captured by another device, from another device, is stored in data the memory etc. and uses described technology.Yet Fig. 4 (it shows the image capture step) illustrates technology described herein under this situation.As shown in Figure 4, imaging device 2 operates in video mode or the still image capture mode (400).When imaging device 2 is video mode (400 video branches), the camera 10 capture video sequences (401) of imaging device 2.The video encoder 18 of imaging device 2 uses piece comparison engine 20 the video sequence of being captured is carried out block-based video coding (402).
Perhaps, when imaging device 2 was still image capture mode (400 still image branches), imaging device 2 was captured still image (403).In the case, the image registration unit 16 of imaging device 2 uses piece comparison engine 20 to come image registration (404).In this way, piece relatively engine 20 is carried out dual-functionality, thereby quickens process of image registration and video coding process.Although process of image registration is through being described as mainly being applicable to still image, process of image registration also can be in order to (for example) picture frame of registration video sequence before video coding.
In brief, the picture frame of 2 pairs of video sequences of device is carried out video coding process, wherein carries out video coding process and comprises that the video blocks of the picture frame that use piece comparison 20 engines will just be encoded and the video blocks of search volume compare.In addition, install 2 carries out image registration process, wherein carry out described process of image registration and comprise that use piece comparison engine 20 compares the piece of first image and the piece of second image.Described technology can be applicable in the device of capture images and video, or is applied in the device of only memory image and video.
As described herein, video coding process comprises and uses piece relatively estimation, the motion compensation of engine 20, and the difference between the match video piece of the video blocks of the picture frame that just is being encoded and search volume is calculated.Described difference is calculated and is comprised subtraction.Process of image registration comprises to be used piece relatively engine 20 is with the relatively piece and the piece of second image of first image, and the match block that makes up first image and second image is to produce the piece through the image of registration.The combination of match block can comprise at least one in add operation and the equalization computing.
The chip area that reduces that is associated with piece comparison engine 20 and the stand-by period of minimizing are that the shared piece of video coding and image registration compares engine 20 and attainable two advantages.Implement to share engine by image registration, can avoid in integrated circuit, adding the needs in the special chip space that is used for image registration.In addition, compare,, can quicken to handle by implementing to be used for hardware based comparison engine 20 of video coding and image registration with the only software solution that is used for video coding and image registration.
The exemplary image registration process that Fig. 5 explanation is consistent with the present invention.As shown in Figure 5, memory 14 stored reference images (it can be described as " second image ").Memory 14 is arranged the piece (402) that receives and handle first image subject to registration according to the alternate buffering device, and described layout can be implemented via replacing input buffer 22.In the case, alternately input buffer 22 with the block buffering of first image in buffer arrangement, make when given of first image with given combination of second image with generation during through the given of image of registration, in described buffer arrangement given of overwrite first image.As further explanation, the alternate buffering device arranges to have two or more parallel buffers usually.When being written to a buffer, can be from another person's reading of data, and vice versa.
The coupling level of unidentified one or more " second " pieces at first image of if block matching unit 27 is greater than threshold value (504 not branches), and then piece combiner 28 does not make up described one or more second piece with first image with the piece (506) of generation through registration.But the piece combiner is forwarded to combination of pixels device 29 with these second.Combination of pixels device 29 is carried out comparison and combination (507) based on pixel at second of first image individual pixel.Hereinafter discuss the more details of described combination based on pixel.Perhaps, " second " piece at its unidentified match block can be stored in the image of registration, and not carry out any combination with the piece of reference picture.
Assembling is through the image 26 (508) of registration in memory 14.Specifically, image registration unit 18 will be written to memory 14 through the piece of registration in the mode of block-by-block, so that assembling is through the image 26 of registration.This image 26 through registration can comprise from the piece of the complete registration of piece combiner 28 and have from one or more of combination of pixels device 29 through the pixel of registration and one or more piece of the pixel of registration not.Image 26 through registration can be stored as finally image through registration, if or will make up additional images to be used for many image registrations, then can with through the image 26 of registration as reference picture 24.
According to the present invention, the carries out image registration process can comprise: the piece that receives first image in the mode of block-by-block; The piece that makes up the piece of first image and second image with produce through the image of registration through registration block; With the block buffering of first image in buffer arrangement, make when given of first image with given combination of second image with generation during through the given of image of registration, in buffer arrangement given of overwrite first image; And assembling through the piece of registration to produce image through registration.
In some cases, if at the piece of given unidentified abundant coupling, then described given can be stored as through the piece of registration and not carry out any combination with the reference block of reference picture 24.In other cases, process of image registration can comprise: when recognition threshold coupling level, make up first of first image with the piece of second image with generation through the image of registration through registration block; When unidentified threshold value coupling level, do not make up second of first image piece with second image; And at second of first image, execution is with respect to the comparison based on pixel of the pixel of second image; And the pixel that makes up second one or more pixels of first image and second image is to produce the pixel through registration.
Process of image registration can relate to piece with two or more images be combined in respect to described two or more images in combination image.This can reduce in order to realize the hand shake,, reduces during image capture the kinetic shake by image capture device that is.Perhaps, image registration as described in the present invention can be handled in order to carry out so-called high dynamic range.In high dynamic range is handled, use different exposures to capture a plurality of pictures of scene.Excessively exposure is to capture shadow detail for some warps in the described image, and some pictures expose with the reservation highlight through deficiency, and some are through exposing to be used for the semi-tone of scene.
By in image registration unit 20, having flexibility, can use embedded framework that a plurality of exposures are combined into high dynamic range images in real time, as described herein.In a further example, process of image registration can comprise that the piece that makes up two or more images is fused into a picture with the sequence with image.In the case, but the carries out image registration merges a picture of capturing motion (for example, someone walking or running advances) with formation with the sequence with image.
As described herein, image registration is with two or more image combination (for example, addition or equalization) programs together.Described two or more images can have level and vertical pixel is shifted (if necessary) with corresponding objects in the alignment image or feature.Pixel shift can with location independent (constant in entire image) or relevant with the position (non-constant in entire image).Block-based registration is divided into first image some (for example, the pieces of 8 pixels, 16 pixels or 32 pixels) and forms the search from the piece of second image (reference picture) to the most approaching piece that just is being registered for each piece.(for example) process of image registration of being carried out by image registration unit 18 can be expressed from the next:
Wherein V and H represent to minimize the value of the v and the h of summation,
S1 represents the piece from first image,
S2 represents the piece from second image,
I and j be across block space,
H and v are two supposition level and vertical shifts between the piece,
M represents macro block support space (for example, 8 * 8,16 * 16 etc.),
Computing (II
DIST) expression measures the general fashion of the distance between two vectors,
Min represents minimum operation, and
∑ is represented summation operation.
As a replacement scheme, squared-distance calculates can be in order to define the piece similarity.If squared-distance is in order to define the piece similarity, then (for example) can be expressed from the next by the process of image registration of image registration unit 18 execution:
In case estimated the piece skew, image registration unit 18 just can use described skew to produce pixel-shift and to make up two images in the mode of block-by-block.For the block-based embodiment of this type, can suppose that pixel-shift in the piece is constant and same skew is used for all pixels of given.
Yet, owing to (for example may not existing from the splendid coupling between the piece of specific and second (reference) image of first image, because a certain mode that can eliminate piece coupling from the piece of second image is rotated or is moved) possibility, it may be impossible making up these pieces with simple shift.The mode of a kind of dealing with problems (it occurs in the time of may being to exist in the scene mobile object) under the situation of not finding enough couplings closely not combination block be used for image registration.In addition, a kind ofly verify that whether above minimization results is that the mode of closely coupling is to set up predetermined threshold, so as decision whether with combination block or not only first will be used for final image.This process can be expressed from the next:
Wherein T represents predetermined threshold.Therefore, if above summation less than threshold value T, then will be by two pieces of image registration unit 18 combination, and if above summation be equal to or greater than threshold value T, then image registration unit 18 will not make up two image blocks.
Whether based on the threshold value piece of (owing to rotation or extraneous moving) that comes the decision-making of combination block to comprise in addition to select not have abundant coupling and threat (threat) these a little pieces by different way.Yet, because make decisions based on total piece difference by image registration unit 18, so it is poor relatively little for some pixels of piece to accumulate (or on average), and bigger for other pixel of piece.In the case, can make the pixel level decision-making with packed-pixel (or not packed-pixel) based on measured pixel difference.This can be the function of optional combination of pixels device 28, as shown in Figure 2.
In the embodiment based on the comparison of pixel of the piece of supporting not have abundant piece coupling, can use poor between the pixel RGB vector according to following formula:
Wherein R1, G1 and B1 are the red, green and blue component from the pixel of the piece of first image, and image R2, G2 and B2 are the red, green and blue component from the pixel of the piece of second (reference) image.M can represent the threshold value based on pixel, makes up below described threshold value.Therefore, if above calculating less than M, then combination of pixels device 28 can be carried out combination based on pixel at that given pixel, and if above calculating more than or equal to M, then combination of pixels device 28 can be avoided the combination of that given pixel.
Can hardware, software, firmware or its any combination implement technology described herein.Any feature that is described as module or assembly can be implemented together in the integrated logic device or be embodied as individually and disperse but the logic device of interoperable.If with software implementation, then can realize described technology by the computer-readable media that comprises instruction to small part, described instruction is carried out one or more in the said method and is implemented piece comparison engine so to carry out when carrying out.Computer-readable media can form the part of the computer program that can comprise encapsulating material.Computer-readable media can comprise for example random-access memory (ram), read-only memory (ROM), nonvolatile RAM (NVRAM), Electrically Erasable Read Only Memory (EEPROM), flash memory, magnetic or the optical data storage media etc. of Synchronous Dynamic Random Access Memory (SDRAM).In addition or alternatively, can to small part by with the form carrying of instruction or data structure or transmit code and can realize described technology by computer access, the computer-readable communication medium that reads and/or carry out.
Code can be carried out by one or more processors, and described processor for example is one or more DSP, general purpose microprocessor, ASIC, field programmable logic array FPGA or the integrated or discrete logic of other equivalence.Therefore, term used herein " processor " can refer to said structure or be suitable for implementing in any other structure of technology described herein any one.In addition, in certain aspects, in the functional dedicated software modules or hardware module that is provided in to dispose described herein, or be incorporated in the video encoder-decoder (CODEC) of combination for Code And Decode.Therefore, the present invention is also contained and is comprised in order in the multiple integrated circuit (IC) apparatus of implementing the one or more circuit in the technology described in the present invention any one.Can in single IC for both chip or integrated circuit (IC) chip, provide this circuit at a plurality of interoperables.
Various technology have been described.These and other instance aspect is in the scope of appended claims.
Claims (35)
1. method, it comprises:
Picture frame to video sequence is carried out video coding process, wherein carries out described video coding process and comprises that the video blocks of the picture frame that use piece comparison engine will just be encoded and the video blocks of search volume compare; And
The carries out image registration process is wherein carried out described process of image registration and is comprised that described comparison of use engine compares the piece of first image and the piece of second image.
2. method according to claim 1,
Wherein said video coding process comprises and uses the described relatively difference calculating between the match video piece of described video blocks and the described search volume of estimation, motion compensation and the described picture frame that just is being encoded of engine, and
Wherein said process of image registration is included in after described of using described described that relatively engine will described first image and described second image compare, and the match block that makes up described first image and described second image is with the piece of generation through the image of registration.
3. method according to claim 2,
Wherein said difference is calculated and is comprised subtraction, and
Wherein the combinations matches piece comprise use add operation and use in the equalization computing one of at least.
4. method according to claim 1, wherein carry out described process of image registration and comprise:
Receive described of described first image in the mode of block-by-block;
Make up described of described first image with described of described second image with produce through the image of registration through registration block;
With the described block buffering of described first image in buffer arrangement, make when given of described first image with given combination of described second image when producing given of described image through registration, in described buffer arrangement described given of described first image of overwrite; And
Assemble described through registration block to produce described image through registration.
5. method according to claim 1, wherein carry out described process of image registration and comprise:
When identifying threshold value coupling level, make up first of described first image with the piece of described second image with produce through the image of registration through registration block;
When unidentified when going out described threshold value coupling level, do not make up second of described first image piece with described second image; And
At described second of described first image, execution is with respect to the comparison based on pixel of the pixel of described second image, and the pixel that makes up one or more described second pixels of described first image and described second image is to produce the pixel through registration.
6. method according to claim 1, wherein carry out described process of image registration and comprise: the piece that makes up two or more images is to reduce the shake in combination image with respect to described two or more images.
7. method according to claim 1, wherein carry out described process of image registration and comprise: the piece that makes up two or more images is fused into a picture with the sequence with image.
8. method according to claim 1 is wherein carried out the piece that described process of image registration comprises described first image of combination and described second image, and wherein uses different exposures to catch described first image and described second image.
9. equipment, it comprises:
Piece is engine relatively;
Video encoder, its picture frame to video sequence is carried out video coding process, and wherein said video coding process comprises that the video blocks of the picture frame that described comparison of use engine will just be encoded and the video blocks of search volume compare; And
Image registration unit, its carries out image registration process, wherein said process of image registration comprise that described comparison of use engine compares the piece of first image and the piece of second image.
10. equipment according to claim 9, wherein
Described video encoder comprises and uses the described relatively poor computing unit of the difference calculating between the match video piece of described video blocks and the described search volume of the described picture frame that just is being encoded of motion estimation unit, motion compensation units and execution of engine, and
Described image registration unit comprise in order to carry out by the piece of more described first image and described second image described of described process of image registration relatively engine, in order to the piece matching unit of identification and matching piece and the combination of described match block of carrying out described first image and described second image to produce assembled unit through the piece of the image of registration.
11. equipment according to claim 10,
Wherein said difference is calculated and is comprised subtraction, and
Wherein the described combination of match block comprise following one of at least: add operation and equalization computing.
12. equipment according to claim 9, wherein said equipment comprises the memory that comprises buffer arrangement, wherein for described process of image registration:
Described memory receives described of described first image in the mode of block-by-block;
Described of described first image of described image registration unit combination with described of described second image with produce through the image of registration through registration block;
Described memory with the described block buffering of described first image in buffer arrangement, make when given of described first image with given combination of described second image when producing given of described image through registration, in described buffer arrangement described given of described first image of overwrite; And
Described image registration unit sends to described memory to assemble described image through registration with described through registration block.
13. equipment according to claim 9, wherein said image registration unit comprises:
The piece matching unit, it determines whether there is threshold value coupling level between one or more pieces one or more pieces with respect to described second image of described first image;
The piece combiner, its when the described threshold value coupling level of identification, make up first of described first image with the piece of described second image with generation through the image of registration through registration block, and when unidentified described threshold value coupling level, do not make up second of described first image piece with described second image; And
The combination of pixels device, it is at described second comparison based on pixel of carrying out described second pixel of described first image with respect to the pixel of described second image of described first image, and the pixel of one or more described second pixels that make up described first image and described second image is with the pixel of generation through registration.
14. equipment according to claim 9 is wherein carried out described process of image registration and is comprised that the piece that makes up two or more images is to reduce the shake in combination image with respect to described two or more images.
15. equipment according to claim 9 is wherein carried out described process of image registration and is comprised that the piece that makes up two or more images is fused into a picture with the sequence with image.
16. equipment according to claim 9 wherein carry out the piece that described process of image registration comprises described first image of combination and described second image, and wherein said first image is to use different exposures to catch with described second image.
17. equipment according to claim 9, wherein said equipment comprises the wireless handset that comprises imaging device.
18. equipment according to claim 9, wherein said equipment comprises integrated circuit.
19. equipment according to claim 9, wherein said equipment comprises imaging device.
20. a computer-readable media that comprises instruction is comprising when described instruction causing described processor after carrying out in the processor of imaging device of piece comparison engine:
Picture frame to video sequence is carried out video coding process, and wherein said video coding process comprises that the video blocks of the picture frame that described comparison of use engine will just be encoded and the video blocks of search volume compare; And
Carries out image registration process, wherein said block-based process of image registration comprise uses described comparison engine that the piece of first image and the piece of second image are compared.
21. computer-readable media according to claim 20,
Wherein said video coding process comprises and uses the described relatively difference calculating between the match video piece of described video blocks and the described search volume of estimation, motion compensation and the described picture frame that just is being encoded of engine, and
Wherein said process of image registration be included in use described relatively behind more described first image of engine and described second image described the match block of described first image of combination and described second image to produce piece through the image of registration.
22. computer-readable media according to claim 21,
Wherein said difference is calculated and is comprised subtraction, and
Wherein make up described match block comprise use add operation and use in the equalization computing one of at least.
23. computer-readable media according to claim 20 is wherein carried out described process of image registration and is comprised:
Receive described of described first image in the mode of block-by-block;
Make up described of described first image with described of described second image with produce through the image of registration through registration block;
With the described block buffering of described first image in buffer arrangement, make when given of described first image with given combination of described second image when producing given of described image through registration, in described buffer arrangement described given of described first image of overwrite; And
Assemble described through registration block to produce described image through registration.
24. computer-readable media according to claim 20 is wherein carried out described process of image registration and is comprised:
When recognition threshold coupling level, make up first of described first image with the piece of described second image with generation through the image of registration through registration block;
When unidentified described threshold value coupling level, do not make up second of described first image piece with described second image; And
At described second of described first image, execution is with respect to the comparison based on pixel of the pixel of described second image, and the pixel that makes up one or more described second pixels of described first image and described second image is to produce the pixel through registration.
25. computer-readable media according to claim 20 is wherein carried out described process of image registration and comprised: the piece that makes up two or more images is to reduce the shake in combination image with respect to described two or more images.
26. computer-readable media according to claim 20 is wherein carried out described process of image registration and comprised: the piece that makes up two or more images is fused into a picture with the sequence with image.
27. computer-readable media according to claim 20 is wherein carried out described process of image registration and is comprised: make up the piece of described first image and described second image, and wherein said first image is to use different exposures to catch with described second image.
28. a device, it comprises:
Be used for the picture frame of video sequence is carried out the device of video coding process, the device that wherein is used to carry out described video coding process comprises the device that the video blocks of the video blocks that is used to use the picture frame that piece comparison engine will just be encoded and search volume compares; And
The device that is used for the carries out image registration process, the device that wherein is used to carry out described process of image registration comprise and are used to use described relatively engine device that the piece of the piece of first image and second image is compared.
29. device according to claim 28,
The device that wherein is used to carry out described video coding process comprise be used to use described relatively engine carry out the device of the difference calculating between the match video piece of the device of estimation, the described video blocks that is used to carry out the device of motion compensation and is used to carry out the described picture frame that just is being encoded and described search volume, and
The combination that the wherein said device that is used to carry out described process of image registration comprises the match block that is used to carry out described first image and described second image is to produce the device through the piece of the image of registration.
30. device according to claim 29,
Wherein saidly be used to carry out the device that difference calculates and comprise the device that is used to carry out subtraction, and
The wherein said device that is used to carry out the combination of match block comprise following one of at least: the device that is used to carry out the device of add operation and is used to carry out the equalization computing.
31. device according to claim 28, the device that wherein is used to carry out described process of image registration comprises:
Be used for receiving described device of described first image in the mode of block-by-block;
Be used to make up described of described first image with described second image described to produce device through registration block through the image of registration;
Be used for described block buffering with described first image in buffer arrangement so that when given of described first image with given combination of described second image when producing given of described image through registration in described buffer arrangement described given device of described first image of overwrite; And
Be used to assemble described through registration block to produce the device of described image through registration.
32. device according to claim 28, the device that wherein is used to carry out described process of image registration comprises:
Be used for when recognition threshold coupling level, making up first of described first image piece with described second image to produce device through registration block through the image of registration;
Be used for when unidentified described threshold value coupling level, not making up second of described first image device with the piece of described second image; And
At described second of described first image, be used to carry out with respect to the pixel of described second image based on the device of the comparison of pixel and the pixel that is used to make up one or more described second pixels of described first image and described second image to produce device through the pixel of registration.
33. device according to claim 28, the device that wherein is used for carrying out described process of image registration comprise that the piece that is used to make up two or more images is to reduce the device through the shake of combination image with respect to described two or more images.
34. device according to claim 28, the device that wherein is used to carry out described process of image registration comprise that the piece that is used to make up two or more images is fused into the device of a picture with the sequence with image.
35. device according to claim 28, the device that wherein is used to carry out described process of image registration comprises the device of the piece that is used to make up described first image and described second image, and wherein said first image is to use different exposures to catch with described second image.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US12/030,820 | 2008-02-13 | ||
| US12/030,820 US8098957B2 (en) | 2008-02-13 | 2008-02-13 | Shared block comparison architechture for image registration and video coding |
| PCT/US2009/033966 WO2009102897A1 (en) | 2008-02-13 | 2009-02-12 | Shared block comparison architecture for image registration and video coding |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101911701A true CN101911701A (en) | 2010-12-08 |
| CN101911701B CN101911701B (en) | 2013-03-27 |
Family
ID=40342340
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2009801014988A Expired - Fee Related CN101911701B (en) | 2008-02-13 | 2009-02-12 | Shared block comparison architecture for image registration and video coding |
Country Status (8)
| Country | Link |
|---|---|
| US (1) | US8098957B2 (en) |
| EP (1) | EP2091255B1 (en) |
| JP (2) | JP2011512111A (en) |
| KR (1) | KR101215045B1 (en) |
| CN (1) | CN101911701B (en) |
| AT (1) | ATE531201T1 (en) |
| TW (1) | TW200948086A (en) |
| WO (1) | WO2009102897A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107392846A (en) * | 2017-07-31 | 2017-11-24 | 四川长虹电器股份有限公司 | Drive recorder image split-joint method |
| WO2022241729A1 (en) * | 2021-05-20 | 2022-11-24 | 深圳市大疆创新科技有限公司 | Image processing method and apparatus, and movable platform and storage medium |
Families Citing this family (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7978925B1 (en) | 2005-04-16 | 2011-07-12 | Apple Inc. | Smoothing and/or locking operations in video editing |
| US7912337B2 (en) * | 2005-11-02 | 2011-03-22 | Apple Inc. | Spatial and temporal alignment of video sequences |
| KR101520068B1 (en) * | 2008-12-16 | 2015-05-13 | 삼성전자 주식회사 | Apparatus and method of blending multiple image |
| EP2302845B1 (en) | 2009-09-23 | 2012-06-20 | Google, Inc. | Method and device for determining a jitter buffer level |
| US9406155B2 (en) * | 2009-09-25 | 2016-08-02 | Arm Limited | Graphics processing systems |
| US9349156B2 (en) | 2009-09-25 | 2016-05-24 | Arm Limited | Adaptive frame buffer compression |
| GB0916924D0 (en) | 2009-09-25 | 2009-11-11 | Advanced Risc Mach Ltd | Graphics processing systems |
| US8885978B2 (en) | 2010-07-05 | 2014-11-11 | Apple Inc. | Operating a device to capture high dynamic range images |
| US8478076B2 (en) | 2010-07-05 | 2013-07-02 | Apple Inc. | Alignment of digital images and local motion detection for high dynamic range (HDR) imaging |
| TWI418220B (en) * | 2010-08-12 | 2013-12-01 | Via Telecom Co Ltd | Video processing methods and systems with software and hardware integration, and computer program products thereof |
| US9479712B2 (en) | 2010-09-28 | 2016-10-25 | Hewlett-Packard Development Company, L.P. | Image blending based on image reference information |
| PT3826306T (en) * | 2011-01-07 | 2023-01-16 | Ntt Docomo Inc | Predictive encoding method, predictive encoding device, and predictive encoding program of motion vector, and, predictive decoding method, predictive decoding device, and predictive decoding program of motion vector |
| US8520080B2 (en) * | 2011-01-31 | 2013-08-27 | Hand Held Products, Inc. | Apparatus, system, and method of use of imaging assembly on mobile terminal |
| US8856212B1 (en) | 2011-02-08 | 2014-10-07 | Google Inc. | Web-based configurable pipeline for media processing |
| GB201105716D0 (en) | 2011-04-04 | 2011-05-18 | Advanced Risc Mach Ltd | Method of and apparatus for displaying windows on a display |
| US9068831B2 (en) * | 2011-05-27 | 2015-06-30 | Panasonic Intellectual Property Management Co., Ltd. | Image processing apparatus and image processing method |
| US20130107064A1 (en) * | 2011-10-27 | 2013-05-02 | Qualcomm Incorporated | Sensor aided image stabilization |
| CN102609915B (en) * | 2012-01-19 | 2015-03-04 | 西南科技大学 | Grating automatic-splicing algorithm based on far-field spots |
| US8639894B2 (en) * | 2012-01-27 | 2014-01-28 | Comcast Cable Communications, Llc | Efficient read and write operations |
| DE102012101242A1 (en) * | 2012-02-16 | 2013-08-22 | Hseb Dresden Gmbh | inspection procedures |
| US8928680B1 (en) | 2012-07-10 | 2015-01-06 | Google Inc. | Method and system for sharing a buffer between a graphics processing unit and a media encoder |
| US9182934B2 (en) | 2013-09-20 | 2015-11-10 | Arm Limited | Method and apparatus for generating an output surface from one or more input surfaces in data processing systems |
| US9195426B2 (en) | 2013-09-20 | 2015-11-24 | Arm Limited | Method and apparatus for generating an output surface from one or more input surfaces in data processing systems |
| US9830567B2 (en) | 2013-10-25 | 2017-11-28 | Location Labs, Inc. | Task management system and method |
| GB2524467B (en) | 2014-02-07 | 2020-05-27 | Advanced Risc Mach Ltd | Method of and apparatus for generating an overdrive frame for a display |
| GB2528265B (en) | 2014-07-15 | 2021-03-10 | Advanced Risc Mach Ltd | Method of and apparatus for generating an output frame |
| GB2540562B (en) | 2015-07-21 | 2019-09-04 | Advanced Risc Mach Ltd | Method of and apparatus for generating a signature representative of the content of an array of data |
| JP6780271B2 (en) * | 2016-03-23 | 2020-11-04 | 富士ゼロックス株式会社 | Image processing equipment and image processing program |
| TWI553592B (en) * | 2016-03-29 | 2016-10-11 | Next Animation Studio Ltd | The method of tracking objects in the video |
| EP3902240B1 (en) | 2020-04-22 | 2022-03-30 | Axis AB | Method, device, camera and software for performing electronic image stabilization of a high dynamic range image |
| US20210382888A1 (en) * | 2020-06-08 | 2021-12-09 | Mongodb, Inc. | Hedged reads |
| KR102680344B1 (en) | 2022-11-25 | 2024-07-02 | 주식회사 와이즈오토모티브 | Image testing apparatus and method |
Family Cites Families (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR0154921B1 (en) * | 1994-12-30 | 1998-11-16 | 배순훈 | Motion estimation device of video encoding apparatus |
| JPH09261530A (en) * | 1996-03-19 | 1997-10-03 | Toshiba Corp | Video recorder |
| US5652625A (en) * | 1996-06-19 | 1997-07-29 | United Microelectronics Corp. | Multi-stage pipeline architecture for motion estimation block matching |
| US6075905A (en) * | 1996-07-17 | 2000-06-13 | Sarnoff Corporation | Method and apparatus for mosaic image construction |
| US6130911A (en) * | 1997-11-21 | 2000-10-10 | Sharp Laboratories Of America, Inc. | Method and apparatus for compressing reference frames in an interframe video codec |
| US6430317B1 (en) * | 1997-12-31 | 2002-08-06 | Sarnoff Corporation | Method and apparatus for estimating motion using block features obtained from an M-ary pyramid |
| US6285711B1 (en) * | 1998-05-20 | 2001-09-04 | Sharp Laboratories Of America, Inc. | Block matching-based method for estimating motion fields and global affine motion parameters in digital video sequences |
| US6501799B1 (en) * | 1998-08-04 | 2002-12-31 | Lsi Logic Corporation | Dual-prime motion estimation engine |
| US6411652B1 (en) * | 1998-10-23 | 2002-06-25 | International Business Machines Corp. | Motion estimation |
| US7837624B1 (en) * | 1998-11-20 | 2010-11-23 | Siemens Medical Solutions Usa, Inc. | Medical diagnostic ultrasound imaging methods for extended field of view |
| JP4284570B2 (en) * | 1999-05-31 | 2009-06-24 | ソニー株式会社 | Imaging apparatus and method thereof |
| US6496538B1 (en) * | 1999-12-31 | 2002-12-17 | Intel Corporation | Method and apparatus for motion estimation for digital video encoding |
| US7039246B2 (en) * | 2002-05-03 | 2006-05-02 | Qualcomm Incorporated | Video encoding techniques |
| US7362918B2 (en) * | 2003-06-24 | 2008-04-22 | Microsoft Corporation | System and method for de-noising multiple copies of a signal |
| US20060088104A1 (en) * | 2004-10-27 | 2006-04-27 | Stephen Molloy | Non-integer pixel sharing for video encoding |
| JP2006157568A (en) * | 2004-11-30 | 2006-06-15 | Konica Minolta Holdings Inc | Imaging apparatus and program |
| KR100639164B1 (en) | 2005-03-12 | 2006-10-26 | 전재춘 | Method for removing blurred image by motion of image equipment |
| JP5011504B2 (en) * | 2005-07-22 | 2012-08-29 | カシオ計算機株式会社 | Image composition apparatus, image composition method, and program |
| JP4529840B2 (en) * | 2005-08-15 | 2010-08-25 | カシオ計算機株式会社 | Imaging apparatus, image processing method, and program |
| US7639897B2 (en) * | 2006-01-24 | 2009-12-29 | Hewlett-Packard Development Company, L.P. | Method and apparatus for composing a panoramic photograph |
| US7746934B2 (en) * | 2006-04-19 | 2010-06-29 | Nethra Imaging Inc. | Image stabilization method |
-
2008
- 2008-02-13 US US12/030,820 patent/US8098957B2/en active Active
- 2008-03-31 AT AT08006422T patent/ATE531201T1/en not_active IP Right Cessation
- 2008-03-31 EP EP08006422A patent/EP2091255B1/en not_active Not-in-force
-
2009
- 2009-02-12 KR KR1020107020476A patent/KR101215045B1/en not_active IP Right Cessation
- 2009-02-12 JP JP2010546902A patent/JP2011512111A/en not_active Ceased
- 2009-02-12 CN CN2009801014988A patent/CN101911701B/en not_active Expired - Fee Related
- 2009-02-12 WO PCT/US2009/033966 patent/WO2009102897A1/en active Application Filing
- 2009-02-13 TW TW098104729A patent/TW200948086A/en unknown
-
2013
- 2013-07-19 JP JP2013150982A patent/JP2013258725A/en not_active Withdrawn
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107392846A (en) * | 2017-07-31 | 2017-11-24 | 四川长虹电器股份有限公司 | Drive recorder image split-joint method |
| WO2022241729A1 (en) * | 2021-05-20 | 2022-11-24 | 深圳市大疆创新科技有限公司 | Image processing method and apparatus, and movable platform and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| EP2091255A1 (en) | 2009-08-19 |
| US20090202176A1 (en) | 2009-08-13 |
| TW200948086A (en) | 2009-11-16 |
| KR101215045B1 (en) | 2012-12-24 |
| JP2011512111A (en) | 2011-04-14 |
| KR20100116211A (en) | 2010-10-29 |
| EP2091255B1 (en) | 2011-10-26 |
| WO2009102897A1 (en) | 2009-08-20 |
| US8098957B2 (en) | 2012-01-17 |
| ATE531201T1 (en) | 2011-11-15 |
| CN101911701B (en) | 2013-03-27 |
| JP2013258725A (en) | 2013-12-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101911701B (en) | Shared block comparison architecture for image registration and video coding | |
| KR102431537B1 (en) | Encoders, decoders and corresponding methods using IBC dedicated buffers and default value refreshing for luma and chroma components | |
| US10021392B2 (en) | Content adaptive bi-directional or functionally predictive multi-pass pictures for high efficiency next generation video coding | |
| JP7471328B2 (en) | Encoders, decoders, and corresponding methods | |
| CN101411181B (en) | Electronic video image stabilization device and method | |
| JP5224458B2 (en) | Method and apparatus for providing a low resolution update mode for multiview video encoding | |
| KR100918480B1 (en) | Stereo vision system and its processing method | |
| US20220408111A1 (en) | Image encoding method using a skip mode, and a device using the method | |
| US11895292B2 (en) | Encoder, decoder and corresponding methods of boundary strength derivation of deblocking filter | |
| US8989256B2 (en) | Method and apparatus for using segmentation-based coding of prediction information | |
| EP4020370A1 (en) | Image processing method and device | |
| CN110741641B (en) | Method and apparatus for video compression | |
| CN112995663B (en) | Video coding method, video decoding method and corresponding devices | |
| CN111277828B (en) | Video encoding and decoding method, video encoder and video decoder | |
| CN107646194B (en) | Apparatus and method for video motion compensation | |
| CN110692246B (en) | Method and apparatus for motion compensated prediction | |
| CN111526362B (en) | Inter-frame prediction method and device | |
| US11962783B2 (en) | Encoder, a decoder and corresponding methods of deblocking filter adaptation | |
| CN115834903A (en) | Inter-frame prediction method and device | |
| CN114270847A (en) | Construction method and device of fusion candidate motion information list and coder/decoder | |
| US20140056348A1 (en) | Methods and device for reconstructing and coding an image block | |
| CN111405277B (en) | Inter-frame prediction method and device and corresponding encoder and decoder | |
| CN111277840B (en) | Transform method, inverse transform method, video encoder and video decoder | |
| US20180084220A1 (en) | Device and method for p-phase data compression | |
| CN113766227A (en) | Quantization and inverse quantization method and apparatus for image encoding and decoding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20130327 Termination date: 20210212 |