CN104574331B - A kind of data processing method, device, computer storage medium and user terminal - Google Patents
A kind of data processing method, device, computer storage medium and user terminal Download PDFInfo
- Publication number
- CN104574331B CN104574331B CN201310500005.6A CN201310500005A CN104574331B CN 104574331 B CN104574331 B CN 104574331B CN 201310500005 A CN201310500005 A CN 201310500005A CN 104574331 B CN104574331 B CN 104574331B
- Authority
- CN
- China
- Prior art keywords
- view
- mapping model
- pixel coordinate
- coordinate mapping
- image data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
- G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20164—Salient point detection; Corner detection
Abstract
The invention discloses a kind of data processing method, device, computer storage medium and user terminals, wherein, this method comprises: when every frame stereoscopic image data to acquisition pre-processes, left view and right view to every frame stereoscopic image data extract characteristic value respectively and match, and the pixel coordinate mapping model between the view of left and right is obtained according to matching result；Pixel coordinate mapping model between corresponding one or so the view of every frame stereoscopic image data, obtains mean pixel coordinate mapping model according to the pixel coordinate mapping model between the corresponding left and right view of multiframe stereoscopic image data；Image procossing is carried out based on the mean pixel coordinate mapping model.Using the present invention, the problem of cannot obtaining stable stereoscopic image data is solved at least, reconstruction can be optimized to obtained stereoscopic image data.
Description
Technical field
The invention belongs to multimedia application technical field more particularly to a kind of data processing method, device, computer storages
Medium and user terminal.
Background technique
SIFT is of local feature description involved in SIFT local feature description algorithm, and SIFT feature uniqueness is good, letter
Breath amount is abundant, and has very strong invariance to the transformation of most of images.In Mikolajczyk to including SIFT operator
The invariance comparative experiments done of ten kinds of local descriptions in, SIFT and its expansion algorithm have been found in similar description
With strongest robustness, so-called robustness refers to a kind of description of stability.
SIFT algorithm contains two parts: the interest area detector of Scale invariant and one are based on interest region
Gray scale ladder distribution feature descriptor.Its main feature is as follows:
A) SIFT feature is the local feature of image, is maintained the invariance to rotation, scaling, brightness change, to view
Angle variation, affine transformation, noise also keep a degree of stability.
B) unique (Distinctiveness) is good, informative, fast suitable for carrying out magnanimity property data base
Speed, accurately matching.
C) volume can produce a large amount of SIFT feature vectors a small number of several objects.
D) high speed, optimized SIFT matching algorithm even can achieve real-time requirement.
E) scalability can very easily be combined with the feature vector of other forms.
SIFT feature matching algorithm mainly includes two stages, and the first stage is the generation of SIFT feature, i.e., from several figures
The feature vector unrelated to scaling, rotation, brightness change is extracted as in；Second stage is the matching of SIFT feature vector.
Nowadays, SIFT algorithm has been widely used for the fields such as target identification, image restoration, image mosaic.
Grab sample consistency algorithm (RANSAC, random sample consensus) is by Fishler and Bolles
A kind of Robust estimation method proposed.Nowadays, RANSAC technology has become linear, nonlinear model estimation important method.
Present inventor at least exists in the prior art during realizing the embodiment of the present application technical solution
Following technical problem:
In the multimedia application field of user terminal, such as mobile terminal, acquire the image data of equipment utilization acquisition into
Row three-dimensional imaging, the synthetic work of stereo-picture include that the preparation of stereo data, sub-pixel judgment criterion, each viewpoint pixel are adopted
Sample, the synthesis of each viewpoint arrangement of subpixels, the compression transmission of stereo-picture and the several parts of display.
In two viewpoint stereo-pictures acquisition synthesis process, since each frame stereo data only has left and right viewpoint two to open figure
Piece, so composition algorithm serious forgiveness is lower.When using mobile terminal acquisition double vision point stereoscopic image data, due to user's operation
Unstable problem, is easy to appear the problems such as finger blocks, when the problems such as noise is larger, and camera blocks occurs in acquisition equipment
When, it is be easy to cause the unavailable of stereoscopic image data, stability is poor, thus final resultant fault or second-rate solid
Picture.As it can be seen that the stability of the stereoscopic image data as data source is to final in entire stereo-picture acquisition synthesis process
Compound stereoscopic picture plays a crucial role, however, for how to obtain stable stereoscopic image data not yet there are
The solution of effect.
Summary of the invention
In view of this, the main purpose of the present invention is to provide a kind of data processing method, device and user terminals, at least
It solves the problems, such as that stable stereoscopic image data cannot be obtained, reconstruction can be optimized to obtained stereoscopic image data.
In order to achieve the above objectives, the technical scheme of the present invention is realized as follows:
The embodiment of the invention provides a kind of data processing methods, comprising:
Left view and right view when being pre-processed to every frame stereoscopic image data of acquisition, to every frame stereoscopic image data
Figure extracts characteristic value respectively and is matched, and the pixel coordinate mapping model between the view of left and right is obtained according to matching result；
Pixel coordinate mapping model between corresponding one or so the view of every frame stereoscopic image data, according to multiframe perspective view
As the pixel coordinate mapping model between the corresponding left and right view of data obtains mean pixel coordinate mapping model；
Image procossing is carried out based on the mean pixel coordinate mapping model.
In above scheme, the left view and right view to every frame stereoscopic image data extracts characteristic value respectively and carries out
Matching, specifically includes:
SIFT feature is extracted to left view and right view respectively, obtains left view and the corresponding SIFT feature of right view
Description；
Left view and corresponding SIFT feature description of right view are subjected to SIFT feature matching, obtain left and right view
The matching double points of the SIFT feature of figure.
It is described that pixel coordinate mapping model between the view of left and right is obtained according to matching result in above scheme, it is specific to wrap
It includes:
The pixel coordinate mapping model between the left and right view is obtained as input parameter using the matching double points.
It is described to obtain the pixel between the left and right view as input parameter using the matching double points in above scheme
Coordinate mapping model, specifically includes:
A data point set is randomly selected from the set S being made of the matching double points to be initialized；
The support point set Si met is filtered out according to preset threshold from the data point set, as consistent collection；
It is constantly chosen according to the comparison of the size of Si and preset threshold between new data sample and estimation left and right view
Pixel coordinate mapping model until obtain maximum consistent collection, according to it is described it is maximum it is consistent integrate obtained as final data sample
Pixel coordinate mapping model between required left and right view.
In above scheme, the pixel coordinate according between the corresponding left and right view of multiframe stereoscopic image data maps mould
Type obtains mean pixel coordinate mapping model, specifically includes:
Obtain the corresponding left and right view of multiframe stereoscopic image data of most freshly harvested specified number before stereo-picture synthesizes
Between pixel coordinate mapping model, between the corresponding left and right view of the multiframe stereoscopic image data pixel coordinate map
Model is averaged, and obtains mean pixel coordinate mapping model.
It is described that image procossing is carried out based on the mean pixel coordinate mapping model in above scheme, it specifically includes:
The repair process of damaged area is carried out based on the mean pixel coordinate mapping model；
Noise reduction process is carried out based on the mean pixel coordinate mapping model.
In above scheme, the repair process that damaged area is carried out based on the mean pixel coordinate mapping model, tool
Body includes:
It detects damaged area, determines damaged area in another normal view based on the mean pixel coordinate mapping model
In coordinate information, the picture material of current damaged area is replaced with the picture material of corresponding region in normal view, to detection
To the edge of damaged area, corrected with the average of two view corresponding pixel points gray values of left and right.
It is described that noise reduction process is carried out based on the mean pixel coordinate mapping model in above scheme, it specifically includes:
It detects suspicious noise spot, determines suspicious noise spot in another view based on the mean pixel coordinate mapping model
In the band of position, carry out gray scale relatively determine whether suspicious noise spot is noise spot；
By another view respective pixel of each grey scale pixel value in neighborhood of the determining noise spot according to predetermined position region
The gray value of point is modified.
The embodiment of the invention also provides a kind of data processing equipments, comprising:
Pretreatment unit, when for being pre-processed to every frame stereoscopic image data of acquisition, to every frame stereo-picture number
According to left view and right view extract characteristic value respectively and matched, the pixel between the view of left and right is obtained according to matching result
Coordinate mapping model；
Image processing unit, for carrying out image procossing based on the mean pixel coordinate mapping model；Every frame perspective view
As the pixel coordinate mapping model between corresponding one or so the view of data, regarded according to the corresponding left and right of multiframe stereoscopic image data
Pixel coordinate mapping model between figure obtains the mean pixel coordinate mapping model.
In above scheme, the pretreatment unit further comprises: characteristic matching subelement；
The characteristic matching subelement, for extracting SIFT feature respectively to left view and right view, obtain left view and
Corresponding SIFT feature description of right view；Left view and corresponding SIFT feature description of right view are carried out
SIFT feature matching, obtains the matching double points of the SIFT feature of left and right view.
In above scheme, the pretreatment unit further comprises: model estimates subelement；
The model estimates subelement, for using the matching double points obtained as parameter is inputted the left and right view it
Between pixel coordinate mapping model.
In above scheme, the model estimates subelement, for selecting at random from the set S being made of the matching double points
A data point set is taken to be initialized；The support point set Si met is filtered out according to preset threshold from the data point set,
As consistent collection；Constantly chosen according to the comparison of the size of Si and preset threshold new data sample and estimation left and right view it
Between pixel coordinate mapping model until obtain maximum consistent collection, according to it is described it is maximum it is consistent integrate obtained as final data sample
Pixel coordinate mapping model between required left and right view.
In above scheme, described image processing unit further comprises: model mean value obtains subelement；
The model mean value obtains subelement, for obtaining the multiframe of most freshly harvested specified number before stereo-picture synthesizes
Pixel coordinate mapping model between the corresponding left and right view of stereoscopic image data, it is corresponding to the multiframe stereoscopic image data
Pixel coordinate mapping model between the view of left and right is averaged, and obtains mean pixel coordinate mapping model.
In above scheme, described image processing unit further comprises: the first processing subelement and second processing are single
Member；
The first processing subelement, for carrying out the reparation of damaged area based on the mean pixel coordinate mapping model
Processing；
Second processing subelement, for carrying out noise reduction process based on the mean pixel coordinate mapping model.
In above scheme, the first processing subelement is further used for detecting damaged area, based on the average picture
Plain coordinate mapping model determines coordinate information of the damaged area in another normal view, with the figure of corresponding region in normal view
As content replaces the picture material of current damaged area, to the edge for detecting damaged area, with two view respective pixels of left and right
The average amendment of point gray value.
In above scheme, the second processing subelement is further used for detecting suspicious noise spot, based on described average
Pixel coordinate mapping model determines the band of position of the suspicious noise spot in another view, carries out gray scale and relatively determines suspicious noise
Whether point is noise spot；By another view pair of each grey scale pixel value in neighborhood of the determining noise spot according to predetermined position region
The gray value of pixel is answered to be modified.
The embodiment of the present invention provides a kind of computer storage medium again, and the computer storage medium includes one group and refers to
It enables, when executed, at least one processor is caused to execute the data as claimed in any one of claims 1 to 8
Processing method.
The embodiment of the invention also provides a kind of user terminal, the user terminal includes that above-mentioned data processing such as fills
It sets.
The method comprise the steps that when every frame stereoscopic image data to acquisition pre-processes, to every frame stereo-picture
The left view and right view of data extract characteristic value respectively and are matched, and obtain the picture between the view of left and right according to matching result
Plain coordinate mapping model；Pixel coordinate mapping model between corresponding one or so the view of every frame stereoscopic image data, according to more
Pixel coordinate mapping model between the corresponding left and right view of frame stereoscopic image data obtains mean pixel coordinate mapping model；Base
Image procossing is carried out in the mean pixel coordinate mapping model.
Using the present invention, due to being pre-processed every frame stereoscopic image data of acquisition to obtain between the view of left and right
Pixel coordinate mapping model optimizes the model to obtain mean pixel coordinate mapping model, is based ultimately upon the mean pixel
Coordinate mapping model carries out image procossing, can optimize reconstruction to obtained stereoscopic image data.
Detailed description of the invention
Fig. 1 is the implementation flow chart of present invention method principle；
Fig. 2 is the basic composed structure schematic diagram of the device of that embodiment of the invention；
The implementation flow chart that Fig. 3 SIFT feature of the embodiment of the present invention is extracted；
Fig. 4 is the implementation flow chart of estimated coordinates of embodiment of the present invention mapping model；
Fig. 5 is the implementation flow chart of breakage of embodiment of the present invention reparation.
Specific embodiment
The implementation of technical solution is described in further detail with reference to the accompanying drawing.
The application scenarios of the data processing method of the embodiment of the present invention are user terminals, especially the multimedia of mobile terminal
Applied technical field, for example, two viewpoint naked eye stereoscopic image data optimized reconstructions in two viewpoint stereo-pictures acquisition synthesis process
Data processing scheme, solve at least the problem of cannot obtaining stable stereoscopic image data in the prior art, it is ensured that user
Terminal, especially mobile terminal will not block when acquiring stereoscopic image data because of camera, noise is larger and lead to generation
It remains to finally synthesize quality relatively high stereotome when stereoscopic image data is unavailable.
The embodiment of the present invention is based primarily upon SIFT algorithm and RANSAC algorithm, is to utilize left and right view by RANSAC algorithm
SIFT match point establish left and right view pixels mapping model, according to the left and right view pixels mapping model, with normal view pair
Damaged view carries out repair and reconstruction, qualified stereoscopic image data is provided for composition algorithm, using the embodiment of the present invention, most
Amended stereoscopic image data can be obtained eventually, it is more smart for repairing breakage image using the left and right view pixels mapping model
Standard, calculation amount is smaller, and repairing effect is good.
Wherein, SIFT algorithm before, cross by by the agency of, and the basic thought of RANSAC algorithm is: estimating carrying out parameter
All possible input data is not treated in timing not instead of without distinction, designs a search first against particular problem
Engine iteratively rejects those input datas inconsistent with estimated parameter (Outliers) using this search engine, then
Estimation parameter is entered data to using correct.The embodiment of the present invention obtains left and right view using the specific implementation of RANSAC algorithm
Image element mapping model.
Three-dimensional imaging is that three-dimensional principle is created based on parallax, and so-called parallax creates three-dimensional principle, refers to two of people
Eyes watch the world from different angles, i.e., have subtle difference between the same object that the left eye object seen and right eye are seen
Not, the eyes average headway about 65mm of people, thus the mode for describing scene profile is also not quite similar.Brain has carefully according to the two
The other scene of elementary errors carries out integrated treatment (physiology fusion), generates accurate three-dimension object perception and the object in scene
In positioning, here it is the three-dimensional senses with depth.
The work of stereo imaging system is exactly at least to generate two images to each scene, and a width represents what left eye was seen
Image, another width represent the image that right eye is seen, this two related images are known as stereo pairs (stereo
pair).And three-dimensional display system must make left eye can only see left image, right eye can only see right image.The embodiment of the present invention
Targeted display methods is double vision point free stereo display method, i.e. left and right viewpoint two images, by sub- pixel
Combination is rearranged, stereo-picture is generated, send to display device.It is saturating by being added before CRT monitor or flat-panel monitor
Mirror cylinder or parallax barrier control the injection direction of each pixel light, and the image of left view point is made only to inject left eye, right viewpoint
Image only injects right eye, using binocular parallax, generates stereoscopic vision.
Stereo-picture synthetic work includes the preparation of stereo data, sub-pixel judgment criterion, each viewpoint pixel sub-sampling, each
The synthesis of viewpoint arrangement of subpixels, the compression transmission of stereo-picture and the several parts of display, the embodiment of the present invention is mainly for solid
The preparation link of image data is to optimize reconstruction to obtained stereoscopic image data, by the optimization of the embodiment of the present invention
After rebuilding stereoscopic image data, even if user terminal, because camera hides when especially mobile terminal acquires stereoscopic image data
Gear, noise are larger and when causing the stereoscopic image data generated unavailable, remain to finally synthesize the relatively high perspective view of quality
Piece.
The data processing method of the embodiment of the present invention, as shown in Figure 1, comprising:
Step 101, when being pre-processed to every frame stereoscopic image data of acquisition, to the left view of every frame stereoscopic image data
Figure and right view extract characteristic value respectively and are matched, and obtain the mapping of the pixel coordinate between the view of left and right according to matching result
Model；
Here, stereoscopic image data can also become stereo-picture material, not repeat them here.
Pixel coordinate mapping model between corresponding one or so the view of step 102, every frame stereoscopic image data, according to more
Pixel coordinate mapping model between the corresponding left and right view of frame stereoscopic image data obtains mean pixel coordinate mapping model；
Step 103 carries out image procossing based on the mean pixel coordinate mapping model.
The data processing equipment of the embodiment of the present invention, as shown in Figure 2, comprising:
Pretreatment unit, when for being pre-processed to every frame stereoscopic image data of acquisition, to every frame stereo-picture number
According to left view and right view extract characteristic value respectively and matched, the pixel between the view of left and right is obtained according to matching result
Coordinate mapping model.
Image processing unit, for carrying out image procossing based on the mean pixel coordinate mapping model；Every frame perspective view
As the pixel coordinate mapping model between corresponding one or so the view of data, regarded according to the corresponding left and right of multiframe stereoscopic image data
Pixel coordinate mapping model between figure obtains the mean pixel coordinate mapping model.
The computer storage medium of the embodiment of the present invention, the computer storage medium include one group of instruction, when execution institute
When stating instruction, at least one processor is caused to execute the data processing method.
The user terminal of the embodiment of the present invention, the basic structure of the data processing equipment including the embodiment of the present invention and its each
Kind deformation and equivalent replacement, do not repeat them here.
An application scenarios of the embodiment of the present invention (scene when user terminal is mobile phone photograph) are specifically described below:
Under scene in the mobile phone photograph, the data processing method of the embodiment of the present invention is specially that two viewpoint naked eyes are three-dimensional
Image data optimized reconstruction processing scheme is required to first carry out preview, user presses when user is using mobile phone shooting stereotome
In general shutter key preceding one end time is in relatively reliable and stable stereo data acquisition state, and press the process of shutter
It is easy to appear burst noise.In order to realize the purpose of the embodiment of the present invention, need to carry out before taking pictures using the embodiment of the present invention
Pretreatment extracts SIFT feature to preview picture or so view, and matched to obtain stable stereoscopic image data,
And then determine pixel coordinate mapping model between current scene or so view, according to this mapping model, we are to single view low-quality
Amount stereo data carries out blocking reparation and denoising, to generate reliable double vision point stereo data
The following are step-by-step procedures, and wherein the first step to the 6th step is that realization is (i.e. above-mentioned pre- during image preview
The process of processing), the 7th step to the 11st step is the step of realization after pressing shutter key and being taken pictures (i.e. based on pretreated
The stereoscopic image data for the optimization and reconstruction that process obtains carries out the synthesis of stereo-picture, after breakage image reparation and denoising
To final result).
Step 1: being pre-processed in preview to preview or so view image, including scaling and image smoothing.By
It is different from photographing mode setting in mobile phone preview mode, so first being zoomed in and out to picture collected under preview mode, such as
5Mp.Since SIFT feature describes algorithm with scale invariability, extracts and match so not interfering with SIFT feature.Image
Gauss low frequency filter is smoothly applied, because gauss low frequency filter can effectively overcome ringing effect, and is imitated to noise is eliminated
Fruit is obvious.In this step, having the beneficial effect that for playing carries out picture smooth treatment with gauss low frequency filter, is mainly
Reduce influence of the scaling for picture quality, to guarantee the reliability of coordinate mapping model.
Step 2: extracting SIFT feature to processed left and right view image.
Here, SIFT matched process as shown in figure 3, left view operating procedure (step 111-116) and right view
Operating procedure (step 121-126) is the same processing, by taking left view as an example, comprising:
Step 111 prepares to extract the SIFT feature of image；
Step 112, building scale space；
Step 113, detection spatial extrema point；
Step 114 is accurately positioned spatial extrema point；
Step 115, removal skirt response point；
Step 116 generates SIFT feature description.
SIFT feature description, and the behaviour for passing through right view are obtained by the operating procedure (step 111-116) of left view
Make step (step 121-126) obtain SIFT feature description son after, execute step 13.
Step 13 describes son eventually by the SIFT feature obtained by the operating procedure (step 111-116) of left view,
SIFT feature description obtained with the operating procedure (step 121-126) by right view carries out SIFT feature matching, with
Just the SIFT feature matching double points of left and right view, subsequent referred to as matching double points are obtained.
In conclusion the basic point of Fig. 3 process is main are as follows:
(a) scale spatial extrema point, the generation including scale space, the detection of spatial extrema point are detected.
Specifically, SIFT has used difference of Gaussian (DoG) scale space, its Gaussian difference pyrene and input by adjacent scale
Image convolution forms.DoG core is not only the linear approximation to LoG, and enormously simplifies the calculating of scale space.To each picture
Vegetarian refreshments searches for extreme point in the neighborhood of its image space and DoG scale space, tentatively obtains the position of characteristic point.It is intermediate to
Test point compares in corresponding 9 × 2 points neighborhood that 26 points are constituted with it with 8 consecutive points of scale and neighbouring scale totally
Compared with to ensure all to detect extreme point in scale space and two dimensional image space.If a point DOG this layer of scale space with
And when being maximum or minimum value in bilevel 26 fields, being considered as the point is a feature of the image under the scale
Point.
(b) it is accurately positioned spatial extrema point, the key point including removing low contrast removes skirt response point.
(c) 128 dimension SIFT feature description are generated.
Specifically, in practical calculating process, in order to enhance matched robustness, 4 × 4 totally 16 are used to each key point
Seed point describes, and each seed point has 8 direction vector information, can generate 128 numbers for a key point in this way
According to ultimately forming the SIFT feature vector of 128 dimensions.
Step 3: being matched to the SIFT feature of left and right view.As the SIFT of two images (image 1 and image 2)
After feature vector generates, similitude of the Euclidean distance of key point feature vector as key point in two images is used in next step
Decision metric.Euclidean distance definition is as shown in formula 1:
Formula 1
Some key point in left view is taken, and finds out its first two key point nearest with Euclidean distance in image 2, so
It is matched afterwards according to distance-ratio criterion, i.e., in the two key points, if nearest distance d1 is divided by secondary close
Distance d2 obtain ratio, if ratio is greater than some proportion threshold epsilon, receive this pair of of match point.The proportion threshold value according to
Experimental result can be 1.5, which, which takes, 1.5 can guarantee that matching double points quantity is used enough, and can effectively control meter
Calculation amount, matching double points quantity are improved with the proportion threshold value and are reduced, and reduce this proportion threshold value, SIFT match point number can subtract
It is few but more stable.Definition is as shown in formula 2.
ratio=d1/d2
Formula 2
With the raising of ratio value, SIFT match point quantity can decline, but precision can improve.Due to of the invention real
Apply higher to the required precision of SIFT matching double points in example, and left and right view difference is smaller, will cause match point excessive number,
So that calculation amount is too big.So the value of ratio should suitably increase in this example, in the present implementation, ratio value is 1.8.
The coordinate information of record matching point pair.
The point that Euclidean distance is nearest in right view can also be used as the match point of current left view SIFT key point, recorded
The coordinate information of matching double points.
Step 4: with the pixel coordinate mapping model between RANSAC algorithm estimation current scene or so view.
Here, estimate that the process of the pixel coordinate mapping model between the view of left and right is as shown in Figure 4, comprising:
Step 201 randomly selects 8 groups of matching double points, for initializing the mapping mould of the pixel coordinate between the left and right view
Type.
Step 202, the support point set for finding out "current" model.
Step 203 judges whether the size for supporting point set meets threshold value, if so, thening follow the steps 204, otherwise, executes
Step 201.
Here, the threshold value can be 67 the percent of entire data point set according to experimental result, and threshold value is rounded a
67 the percent of data point set can guarantee the validity of model.
Step 204 estimates pixel coordinate mapping model between the left and right view.
The process of Fig. 4 be using matching double points respectively the coordinate information in the view of left and right as RANSAC model estimating system
Input parameter, to estimate left and right view pixels mapping model.
In conclusion the basic point of Fig. 4 process is main are as follows:
(a) data point set is randomly selected from matching double points set S, and by this subset initialization model.
(b) find out according to threshold value Td become "current" model support point set Si, set Si be exactly sample consistent collection, determined
Justice is available point；
If (c) size of set Si has been more than some threshold value T, model is reevaluated with Si and is terminated；
If (d) size of set Si is less than threshold value Ts, a new sample is chosen, repeats above step；
It is attempted by n times, maximum consistent collection Si is selected, reevaluates model with it, obtains result to the end.This
Left and right view pixels coordinate information is respectively P1 and P2 in example, by Epipolar geometry principle it is found that left and right view passes through two images
Fundamental matrix F it is associated, meet P2TFP1=0.The process that model is estimated in the step is actually the process solved to F,
Since matching double points are enough, so reliable coordinate mapping model can be generated.
Step 5: only retaining the correct match point for meeting the pixel coordinate mapping model in step 4 between the view of left and right
Right, i.e. in Si point set saves as this frame image reference information together with obtained left and right view pixels coordinate mapping model.
Step 6: the Infinite Cyclic first step to the 5th step, only retains the image reference letter of newest collected four frames image
Breath, establishes queue.Team's head frame information is deleted, latest frame reference information is stored in tail of the queue by one frame stereoscopic image data of every acquisition.
Step 7: taking pictures, and left and right view is smoothed, is mapped with when the pixel coordinate between front left and right view
First three frame of modeling queue solves mean pixel coordinate mapping model.It after taking pictures, chooses whether to need to repair view by user, if needing
It wants, then continues.
Step 8: verifying whether to block.Cutting that left and right view is respectively averaged is 8 pieces, takes average gray value to each piece,
The average gray value of left and right view respective block is compared, if block average gray value relative different is incited somebody to action 10% or more
Low ash angle value block is considered as damaged block, if damaged block counts are 0, skips to the tenth step.
Step 9: repairing occlusion area.
Here, damaged reparation process is as shown in Figure 5, comprising:
Step 301 accurately detects damaged area (sobel operator).
Step 302 calculates coordinate information of the damaged area in normal view.
Step 303 repairs damaged area.
Step 304 repairs edge.
In conclusion the basic point of Fig. 5 process is main are as follows:
(a) damaged area is accurately determined.In damaged block, gray scale break edge is detected with sobel operator.The operator packet
It is respectively horizontal and vertical containing two group of 3 × 3 matrix, it is done into planar convolution with image, can obtain transverse direction and longitudinal direction respectively
Brightness difference approximation.
(b) damaged area coordinate information in another view is determined with current scene or so view coordinate mapping model.
(c) current damaged area picture material is replaced with corresponding region picture material in normal view.
(d) edge reparation.To the edge detected in (a), every pixel 3 × 3 faces each grey scale pixel value left and right in domain
The average of two view corresponding pixel points gray values is corrected.
Step 10: noise spot detects.During taking pictures, left and right view may be with certain salt-pepper noise, simultaneously
After the 9th step, due to the limitation of coordinate mapping model, it will be accompanied by a small amount of salt-pepper noise, this example is examined with median filter
The noise of left and right view is surveyed, and noise spot is marked.Face in domain (N is odd number) in current point N × N, takes gray scale most
Big value, minimum value and mean value if the gray value of current point is this maximum or minimum value for facing in domain, and are more than given threshold
(average gray value 60%-150% is basic threshold value in the neighborhood, is more than given threshold except this range), it is likely that make an uproar
Point is labeled as suspicious points.Suspicious points band of position in another view is determined by coordinate mapping model at this time, current point is set
In this position, gray scale comparison is carried out again, and then determines whether current point is noise spot.
Step 11: repairing noise spot.By grey scale pixel value each in 3 × 3 neighborhood of noise spot confirmed in the tenth step with separately
The gray value of one viewpoint corresponding pixel points is corrected.
Step 12: the stereoscopic image data after optimization is submitted to composition algorithm, carried out using existing composition algorithm
The synthesis of stereo-picture.
If the module integrated described in the embodiment of the present invention is realized in the form of software function module and as independent production
Product when selling or using, also can store in a computer readable storage medium.Based on this understanding, the present invention is real
Applying the technical solution of example, substantially the part that contributes to existing technology can embody in the form of software products in other words
Come, which is stored in a storage medium, including some instructions are used so that a computer equipment (can
To be personal computer, server or network equipment etc.) execute the whole or portion of each embodiment the method for the present invention
Point.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), deposits at random
The various media that can store program code such as access to memory (RAM, Random Access Memory), magnetic or disk.
It is combined in this way, the embodiment of the present invention is not limited to any specific hardware and software.
The foregoing is only a preferred embodiment of the present invention, is not intended to limit the scope of the present invention.
Claims (18)
1. a kind of data processing method, which is characterized in that the described method includes:
Left view and right view point when being pre-processed to every frame stereoscopic image data of acquisition, to every frame stereoscopic image data
It indescribably takes characteristic value and is matched, the pixel coordinate mapping model between the view of left and right is obtained according to matching result；
Pixel coordinate mapping model between corresponding one or so the view of every frame stereoscopic image data, according to multiframe stereo-picture number
Mean pixel coordinate mapping model is obtained according to the pixel coordinate mapping model between corresponding left and right view；
Image procossing is carried out based on the mean pixel coordinate mapping model.
2. the method according to claim 1, wherein the left view to every frame stereoscopic image data and right view
Figure extracts characteristic value respectively and is matched, and specifically includes:
SIFT feature is extracted to left view and right view respectively, obtains left view and the corresponding SIFT feature description of right view
Son；
Left view and corresponding SIFT feature description of right view are subjected to SIFT feature matching, obtain left and right view
The matching double points of SIFT feature.
3. according to the method described in claim 2, it is characterized in that, described obtain the picture between the view of left and right according to matching result
Plain coordinate mapping model, specifically includes:
The pixel coordinate mapping model between the left and right view is obtained as input parameter using the matching double points.
4. according to the method described in claim 3, it is characterized in that, described obtained using the matching double points as input parameter
Pixel coordinate mapping model between the left and right view, specifically includes:
A data point set is randomly selected from the set S being made of the matching double points to be initialized；
The support point set Si met is filtered out according to preset threshold from the data point set, as consistent collection；
The picture between new data sample and estimation left and right view is constantly chosen according to the comparison of the size of Si and preset threshold
Plain coordinate mapping model until obtain maximum consistent collection, according to it is described it is maximum it is consistent integrate obtained as final data sample it is required
Left and right view between pixel coordinate mapping model.
5. method according to any one of claims 1 to 4, which is characterized in that described according to multiframe stereoscopic image data pair
The pixel coordinate mapping model between the view of left and right answered obtains mean pixel coordinate mapping model, specifically includes:
It obtains before stereo-picture synthesizes between the corresponding left and right view of multiframe stereoscopic image data of most freshly harvested specified number
Pixel coordinate mapping model, to the pixel coordinate mapping model between the corresponding left and right view of the multiframe stereoscopic image data
It is averaged, obtains mean pixel coordinate mapping model.
6. according to the method described in claim 5, it is characterized in that, described carried out based on the mean pixel coordinate mapping model
Image procossing specifically includes:
The repair process of damaged area is carried out based on the mean pixel coordinate mapping model；
Noise reduction process is carried out based on the mean pixel coordinate mapping model.
7. according to the method described in claim 6, it is characterized in that, described carried out based on the mean pixel coordinate mapping model
The repair process of damaged area, specifically includes:
It detects damaged area, determines damaged area in another normal view based on the mean pixel coordinate mapping model
Coordinate information replaces the picture material of current damaged area with the picture material of corresponding region in normal view, to detecting brokenly
The edge for damaging region is corrected with the average of two view corresponding pixel points gray values of left and right.
8. according to the method described in claim 6, it is characterized in that, described carried out based on the mean pixel coordinate mapping model
Noise reduction process specifically includes:
It detects suspicious noise spot, determines suspicious noise spot in another view based on the mean pixel coordinate mapping model
The band of position carries out gray scale and relatively determines whether suspicious noise spot is noise spot；
By another view corresponding pixel points of each grey scale pixel value in neighborhood of the determining noise spot according to predetermined position region
Gray value is modified.
9. a kind of data processing equipment, which is characterized in that described device includes:
Pretreatment unit, when for being pre-processed to every frame stereoscopic image data of acquisition, to every frame stereoscopic image data
Left view and right view extract characteristic value respectively and are matched, and obtain the pixel coordinate between the view of left and right according to matching result
Mapping model；
Image processing unit, for carrying out image procossing based on mean pixel coordinate mapping model；Every frame stereoscopic image data pair
The pixel coordinate mapping model between one or so view is answered, according between the corresponding left and right view of multiframe stereoscopic image data
Pixel coordinate mapping model obtains the mean pixel coordinate mapping model.
10. device according to claim 9, which is characterized in that the pretreatment unit further comprises: characteristic matching
Subelement；
The characteristic matching subelement obtains left view and right view for extracting SIFT feature respectively to left view and right view
Scheme corresponding SIFT feature description；It is special that left view and corresponding SIFT feature description of right view are subjected to SIFT
Sign point matching, obtains the matching double points of the SIFT feature of left and right view.
11. device according to claim 10, which is characterized in that the pretreatment unit further comprises: model estimation
Subelement；
The model estimates subelement, for being obtained between the left and right view using the matching double points as input parameter
Pixel coordinate mapping model.
12. device according to claim 11, which is characterized in that the model estimates subelement, for from by described
It initializes with point to randomly selecting a data point set in the set S of composition；According to default threshold from the data point set
Value filters out the support point set Si met, as consistent collection；It is new constantly to choose according to the comparison of the size of Si and preset threshold
Data sample and estimation left and right view between pixel coordinate mapping model until obtain maximum consistent collection, according to it is described most
Big unanimously integrating obtains the pixel coordinate mapping model between required left and right view as final data sample.
13. according to the described in any item devices of claim 9 to 12, which is characterized in that described image processing unit further wraps
Include: model mean value obtains subelement；
The model mean value obtains subelement, and the multiframe for obtaining most freshly harvested specified number before stereo-picture synthesizes is three-dimensional
Pixel coordinate mapping model between the corresponding left and right view of image data, left and right corresponding to the multiframe stereoscopic image data
Pixel coordinate mapping model between view is averaged, and obtains mean pixel coordinate mapping model.
14. device according to claim 13, which is characterized in that described image processing unit further comprises: at first
Manage subelement and second processing subelement；
The first processing subelement, for carrying out the repair place of damaged area based on the mean pixel coordinate mapping model
Reason；
Second processing subelement, for carrying out noise reduction process based on the mean pixel coordinate mapping model.
15. device according to claim 14, which is characterized in that the first processing subelement is further used for detecting
To damaged area, determine that coordinate of the damaged area in another normal view is believed based on the mean pixel coordinate mapping model
Breath, the picture material of current damaged area is replaced with the picture material of corresponding region in normal view, to detecting damaged area
Edge, with left and right two view corresponding pixel points gray values average correct.
16. device according to claim 14, which is characterized in that the second processing subelement is further used for detecting
To suspicious noise spot, position area of the suspicious noise spot in another view is determined based on the mean pixel coordinate mapping model
Domain carries out gray scale and relatively determines whether suspicious noise spot is noise spot；By determining noise spot according to the neighbour in predetermined position region
Each grey scale pixel value is modified with the gray value of another view corresponding pixel points in domain.
17. a kind of computer storage medium, which is characterized in that the computer storage medium includes one group of instruction, when execution institute
When stating instruction, at least one processor is caused to execute the data processing method as claimed in any one of claims 1 to 8.
18. a kind of user terminal, which is characterized in that the user terminal includes such as the described in any item numbers of claim 9 to 16
According to processing unit.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310500005.6A CN104574331B (en) | 2013-10-22 | 2013-10-22 | A kind of data processing method, device, computer storage medium and user terminal |
| PCT/CN2014/075991 WO2014180255A1 (en) | 2013-10-22 | 2014-04-22 | Data processing method, apparatus, computer storage medium and user terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310500005.6A CN104574331B (en) | 2013-10-22 | 2013-10-22 | A kind of data processing method, device, computer storage medium and user terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104574331A CN104574331A (en) | 2015-04-29 |
| CN104574331B true CN104574331B (en) | 2019-03-08 |
Family
ID=51866712
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310500005.6A Active CN104574331B (en) | 2013-10-22 | 2013-10-22 | A kind of data processing method, device, computer storage medium and user terminal |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN104574331B (en) |
| WO (1) | WO2014180255A1 (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10122996B2 (en) * | 2016-03-09 | 2018-11-06 | Sony Corporation | Method for 3D multiview reconstruction by feature tracking and model registration |
| CN110800020B (en) * | 2017-07-28 | 2021-07-09 | 深圳配天智能技术研究院有限公司 | Image information acquisition method, image processing equipment and computer storage medium |
| CN107786866B (en) * | 2017-09-30 | 2020-05-19 | 深圳睛灵科技有限公司 | Binocular vision image synthesis system and method |
| CN110147598B (en) * | 2019-05-10 | 2023-08-22 | 上海理工大学 | Ultrahigh-speed impact fragment cloud modeling and damage evaluation method based on image processing |
| CN110321858B (en) * | 2019-07-08 | 2022-06-14 | 北京字节跳动网络技术有限公司 | Video similarity determination method and device, electronic equipment and storage medium |
| CN110599513B (en) * | 2019-09-04 | 2022-02-11 | 南京邮电大学 | Binocular vision image edge detection and target tracking method |
| CN113362233B (en) * | 2020-03-03 | 2023-08-29 | 浙江宇视科技有限公司 | Picture processing method, device, equipment, system and storage medium |
| CN112215871B (en) * | 2020-09-29 | 2023-04-21 | 武汉联影智融医疗科技有限公司 | Moving target tracking method and device based on robot vision |
| CN112562063A (en) * | 2020-12-08 | 2021-03-26 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for carrying out three-dimensional attempt on object |
| CN113421248B (en) * | 2021-06-30 | 2024-02-09 | 上海申瑞继保电气有限公司 | Substation equipment rotating image numerical value processing method |
| CN116630203B (en) * | 2023-07-19 | 2023-11-07 | 科大乾延科技有限公司 | Integrated imaging three-dimensional display quality improving method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1463154A (en) * | 2002-05-28 | 2003-12-24 | 华为技术有限公司 | Method for shielding errors happening in video signal |
| CN1774920A (en) * | 2003-02-19 | 2006-05-17 | 索尼电子有限公司 | Method and system for copy protection |
| CN102081687A (en) * | 2010-12-30 | 2011-06-01 | 上海电机学院 | Three-dimensional hairstyling method and device |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100556153C (en) * | 2007-12-05 | 2009-10-28 | 宁波大学 | A kind of preprocess method of multi-view image |
| JP4737573B2 (en) * | 2009-02-05 | 2011-08-03 | 富士フイルム株式会社 | 3D image output apparatus and method |
| CN102096915B (en) * | 2011-02-09 | 2013-08-07 | 北京航空航天大学 | Camera lens cleaning method based on precise image splicing |
| CN103345736B (en) * | 2013-05-28 | 2016-08-31 | 天津大学 | A kind of virtual viewpoint rendering method |
-
2013
- 2013-10-22 CN CN201310500005.6A patent/CN104574331B/en active Active
-
2014
- 2014-04-22 WO PCT/CN2014/075991 patent/WO2014180255A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1463154A (en) * | 2002-05-28 | 2003-12-24 | 华为技术有限公司 | Method for shielding errors happening in video signal |
| CN1774920A (en) * | 2003-02-19 | 2006-05-17 | 索尼电子有限公司 | Method and system for copy protection |
| CN102081687A (en) * | 2010-12-30 | 2011-06-01 | 上海电机学院 | Three-dimensional hairstyling method and device |
Non-Patent Citations (3)
| Title |
|---|
| 一种新的基于条件数的图像配准算法;马丽涛 等;《中国图象图形学报》;20080229;第13卷(第2期);第279页右栏第5段 |
| 基于Harris角点检测的立体视觉里程计;彭勃 等;《兵工学报》;20071231;第28卷(第12期);第1499页1.3节第1段 |
| 基于SIFT特征描述子的立体匹配算法;宰小涛，赵宇明;《微计算机信息》;20070725;第23卷(第8-3期);285-287 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104574331A (en) | 2015-04-29 |
| WO2014180255A1 (en) | 2014-11-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104574331B (en) | A kind of data processing method, device, computer storage medium and user terminal | |
| Lu et al. | Depth enhancement via low-rank matrix completion | |
| Zhu et al. | Depth image based view synthesis: New insights and perspectives on hole generation and filling | |
| CN104205826B (en) | For rebuilding equipment and the method for density three-dimensional image | |
| KR100793076B1 (en) | Edge-adaptive stereo/multi-view image matching apparatus and its method | |
| Cheng et al. | Spatio-temporally consistent novel view synthesis algorithm from video-plus-depth sequences for autostereoscopic displays | |
| CN101610425B (en) | Method for evaluating stereo image quality and device | |
| CN106981078B (en) | Sight line correction method and device, intelligent conference terminal and storage medium | |
| CN103345736A (en) | Virtual viewpoint rendering method | |
| Lin et al. | Real photographs denoising with noise domain adaptation and attentive generative adversarial network | |
| Zhang et al. | Mm-bsn: Self-supervised image denoising for real-world with multi-mask based on blind-spot network | |
| Luo et al. | Foreground removal approach for hole filling in 3D video and FVV synthesis | |
| KR20130105969A (en) | Method for depth video filtering and apparatus thereof | |
| Bleyer et al. | Temporally consistent disparity maps from uncalibrated stereo videos | |
| KR100943635B1 (en) | Method and apparatus for generating disparity map using digital camera image | |
| Ling et al. | Image quality assessment for DIBR synthesized views using elastic metric | |
| Hsu et al. | Spatio-temporally consistent view synthesis from video-plus-depth data with global optimization | |
| CN109302600B (en) | Three-dimensional scene shooting device | |
| Zhou et al. | Hierarchical visual comfort assessment for stereoscopic image retargeting | |
| Tian et al. | A full-reference image quality assessment metric for 3D synthesized views | |
| DE102004026782A1 (en) | Method and apparatus for computer-aided motion estimation in at least two temporally successive digital images, computer-readable storage medium and computer program element | |
| CN109712230B (en) | Three-dimensional model supplementing method and device, storage medium and processor | |
| Sun et al. | Seamless view synthesis through texture optimization | |
| CN104463958A (en) | Three-dimensional super-resolution method based on disparity map fusing | |
| Liu et al. | Texture-adaptive hole-filling algorithm in raster-order for three-dimensional video applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |