CN108961304A

CN108961304A - Identify the method for sport foreground and the method for determining target position in video in video

Info

Publication number: CN108961304A
Application number: CN201710392042.8A
Authority: CN
Inventors: 胡康康; 任沛然
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2017-05-23
Filing date: 2017-05-27
Publication date: 2018-12-07
Anticipated expiration: 2037-05-27
Also published as: CN108961304B

Abstract

This application discloses a kind of methods of sport foreground in identification video, comprising: acquisition includes the video frame to be processed of video implantation target area；It is implanted into target area according to the video, determines corresponding area-of-interest；For the area-of-interest, through prospect compared with background, coarse segmentation is carried out to the sport foreground of the area-of-interest；For the coarse segmentation as a result, by image edge processing, the sport foreground subdivision of the area-of-interest is cut, motion foreground segmentation result is obtained.It can play the role of quickly real-time and accurately identifying sport foreground in video frame.

Description

Identify the method for sport foreground and the method for determining target position in video in video

This application claims in submission on May 23rd, 2017 Patent Office of the People's Republic of China, application No. is 201710370518.8, invention names Referred to as the Chinese patent application of " method of sport foreground and the method for determining target position in video in identification video " is preferential Power, entire contents are hereby incorporated by reference in the application.

Technical field

This application involves video implanted prosthetics, and in particular to a method of sport foreground in identification video further relates to one Kind determine video in target position method, further relate to it is a kind of realizes video implantation target occlusion method, further relate to one kind with Mesh calibration method in track video.The application further relates to a kind of storage equipment of sport foreground in video for identification, and the application is also It is related to a kind of for determining that the storage equipment of target position in video, the application further relate to a kind of for realizing video implantation target The storage equipment blocked, the application further relate to a kind of for tracking the storage equipment of target in video.The application further relates to one kind The electronic equipment that can identify sport foreground in video further relates to a kind of electronic equipment that can determine target position in video, A kind of electronic equipment that can be realized video implantation target occlusion is further related to, a kind of electricity that can track target in video is further related to Sub- equipment.

Background technique

Video implanted prosthetics can seamlessly be embedded in poster, text or three-dimension object in existing video, therefore Video ads dispensing, virtual reality (Virtual Reality, VR) and augmented reality (Argumented Reality, AR) etc. Field has broad application prospects.

It, can be embedding by the image of object after we have selected Implantation Time and implantation position by video implanted prosthetics Enter into scene.This mode is static in video camera and effect is fine when not having extensive moving object.But when video camera moves When having a large amount of moving objects when dynamic or in video, the image of the object of insertion often can not be merged ideally with scene, performance The movement of camera lens can not be followed in the image of the object of insertion and is moved in picture, or the object being embedded in when there is sport foreground The effect etc. being blocked can not be presented in the image of body.

In order to achieve the effect that seamless insertion, traditional video method for implantation generally requires professional person in the later period to each Secondary to be embedded into row artificial treatment, which results in relatively high costs, cause the practicality little.

Cost undoubtedly significantly can be improved efficiency and reduce by automating implanted prosthetics, in order to realize that automation is planted Enter, needs to solve occlusion issue.Occlusion issue when how preferably to solve video implantation is that those skilled in the art have to In face of the problem of.

Summary of the invention

The application provides a kind of method for identifying sport foreground in video, comprising:

Acquisition includes the video frame to be processed of video implantation target area；

It is implanted into target area according to the video, determines corresponding area-of-interest；

For the area-of-interest, through prospect compared with background, the sport foreground of the area-of-interest is carried out Coarse segmentation；

For the coarse segmentation as a result, by image edge processing, the sport foreground subdivision of the area-of-interest is cut, Obtain motion foreground segmentation result.

Optionally, described that target area is implanted into according to the video, determine that corresponding area-of-interest includes:

It determines in video frame to be processed and is implanted into target area；

The minimum circumscribed rectangle of the implantation target area is calculated according to the implantation target area in the video frame to be processed Or triangle；

Resulting minimum circumscribed rectangle or triangle Shape definition area-of-interest are calculated according to described.

Optionally, described to include: according to the resulting minimum circumscribed rectangle of the calculating or triangle Shape definition area-of-interest

The minimum circumscribed rectangle or triangle are considered as area-of-interest；

Or:

In threshold value of the area size of the minimum circumscribed rectangle or triangle not less than setting, to the area size Scaled down, until meeting the requirement of the threshold value, using the region after reducing as area-of-interest.

Optionally, described to be directed to the area-of-interest, through prospect compared with background, in the area-of-interest Sport foreground carries out coarse segmentation

Prospect gray scale binary map is obtained using the background modeling algorithm based on k nearest neighbor for the area-of-interest,

Realize the coarse segmentation to the sport foreground of the video to be processed.

Optionally, further includes:

Prospect is smoothed using morphologic closed operation for the prospect gray scale binary map.

Optionally, described to be directed to the coarse segmentation as a result, by image edge processing, to the fortune in the area-of-interest Dynamic prospect subdivision, which is cut, includes:

Described image edge processing algorithm is the Grab cut algorithm based on graph theory；

The Grab cut algorithm specifically includes:

Using the coarse segmentation result as boundary condition,

Foreground and background region is marked to the video frame to be processed using non-fully labeling method；

Gauss hybrid models are established respectively to the foreground area and background area color space；

The foreground zone of the video frame to be processed is determined by the interactive iteration process that partitioning estimation and model parameter learn Domain and background area.

Optionally, further includes: Gaussian smoothing is carried out to the result after described image edge processing algorithm process and is filled out Hole operation.

Optionally, the acquisition includes that the video frame to be processed of video implantation target area includes:

Obtain the location information in tracked target region and video frame to be processed in reference video frame；

The reference video frame and video frame to be processed are converted into single channel black white image；

Reference video frame and video frame to be processed for the single channel black white image are obtained from the reference video frame To the affine transformation matrix of the video frame to be processed；

Position of the tracked target region in the video frame to be processed is determined using the affine transformation matrix；

Wherein, the tracked target region is that video is implanted into target area.

Optionally, the reference video frame for single channel black white image and video frame to be processed are obtained from the ginseng The affine transformation matrix for examining video frame to the video frame to be processed includes:

The characteristic point for meeting the requirements quantity is determined in preset characteristic point region of search in reference video frame；

It in the video frame to be processed, presets in the match point region of search of size, using single-point template matching Method determines the characteristic point corresponding match point in the video frame to be processed；

Using the characteristic point and the match point determine reference video frame to video frame to be processed affine transformation matrix.

Optionally, described in the video frame to be processed, determine that the characteristic point exists using single-point template matching method Corresponding match point in the video frame to be processed, including

To each characteristic point determined in reference video frame, setting includes the region conduct of the specified point being sized Pattern plate bolster；

With aforementioned pattern plate bolster, search and the most matched region of the pattern plate bolster in video frame to be processed, to determine with reference to view Each of frequency frame characteristic point corresponding match point in video frame to be processed.

Optionally, described with aforementioned pattern plate bolster, search and the most matched region of the pattern plate bolster in video frame to be processed, with Determine each of reference video frame characteristic point corresponding match point in video frame to be processed；Including

By row or column point by point search and the most matched region of the pattern plate bolster in video frame to be processed, to determine reference video Each of frame characteristic point corresponding match point in video frame to be processed；

Or in video frame to be processed, corresponding to the match point field of search of the setting of the characteristic point position of reference video frame In domain, search and the most matched region of the pattern plate bolster, to determine each of reference video frame characteristic point in video to be processed Corresponding match point in frame.

Optionally, described with aforementioned pattern plate bolster, search and the most matched region of the pattern plate bolster in video frame to be processed, with Determine each of reference video frame characteristic point corresponding match point in video frame to be processed；Specially

In video frame to be processed, corresponding to the match point region of search of the setting of the characteristic point position of reference video frame It is interior, search and the most matched region of the pattern plate bolster, to determine each of reference video frame characteristic point in video frame to be processed In corresponding match point；

If not searching match point in matching point search matching area, the range of matching area is spread.

Optionally, the preset characteristic point region of search is includes tracked target region, and area is not small In the region of the two times of sizes in tracked target region.

Optionally, determination meets quantitative requirement in the characteristic point region of search preset in reference video frame Characteristic point includes:

When determining characteristic point quantity is less than preset characteristic point amount threshold, expand characteristic point region of search；

Characteristic point is determined in widened characteristic point region of search.

Optionally, the preset characteristic point amount threshold is 20.

Optionally, it is 1/1st to seven/15th size of reference video frame area that the pattern plate bolster, which includes: area, Rectangle.

Optionally, the characteristic point include Harris angle point, ShiTomasi angle point, SURF angle point, FAST characteristic point or SIFT feature.

Optionally it is determined that the location information in tracked target region includes in reference video frame

Determine the corner location information of no less than three angle points of tracked target frame in reference video frame.

Optionally, described to determine the tracked target region in the video to be processed using the affine transformation matrix Behind position in frame further include:

When displacement of position of the tracked target region in the video frame to be processed relative to reference video frame Less than setting threshold value when, use the position in tracked target region in the previous video frame of video frame to be processed as tracked target Position of the region in video frame to be processed.

The application also provides a kind of method of target position in determining video, comprising:

Position of the tracked target region in the video frame to be processed is determined using the affine transformation matrix.

Optionally, the preset characteristic point amount threshold is 20.

Optionally, it is 1/1st to seven/15th size of reference video frame area that the pattern plate bolster, which includes: for area, Rectangle.

The application also provides a kind of method for realizing video implantation target occlusion, is characterized in that, comprising:

Video to be processed is obtained, the video frame to be processed in the video includes video implantation target area；

The foreground area for cutting acquisition to the subdivision operates, and the implantation target is blocked in realization.

The application also provides mesh calibration method in a kind of tracking video, is characterized in that, comprising:

It obtains video to be processed and determines reference video frame and video frame to be processed；

New video frame to be processed is set and updates reference video frame when meeting and updating preset renewal frequency；

It returns and obtains in reference video frame the location information in tracked target region and the step of video frame to be processed, until Video is disposed.

Optionally, the update reference video frame when meeting update preset renewal frequency includes:

The reference for being directed to video frame to be processed is updated according to the frequency of setting according to the movement speed of the camera lens of shooting video Video frame；Or

Using the former frame of video frame to be processed or the first two frame as reference video frame.

The application also provides a kind of storage equipment of sport foreground in video for identification, is characterized in that, is stored with instruction, Described instruction can be loaded by processor and execute following operation:

The application also provides a kind of for determining the storage equipment of target position in video, is characterized in that, is stored with instruction, Described instruction can be loaded by processor and execute following operation:

The application also provides a kind of storage equipment for realizing video implantation target occlusion, is characterized in that, is stored with finger It enables, described instruction can be loaded by processor and execute following operation:

The application also provides a kind of for tracking the storage equipment of target in video, is characterized in that, is stored with instruction, described Instruction can be loaded by processor and execute following operation:

The application also provides a kind of electronic equipment that can be used in identifying sport foreground in video, is characterized in that, including deposit Equipment and processor are stored up, the storage equipment is stored with the instruction of sport foreground in identification video, and described instruction is by the processing When device is loaded and executed, following operation is executed:

The application also provides a kind of electronic equipment that can determine target position in video, is characterized in that, including storage is set Standby and processor, the storage equipment are stored with the instruction of target position in determining video, and described instruction is added by the processor When carrying and executing, following operation is executed:

The application also provides a kind of electronic equipment for realizing video implantation target occlusion, is characterized in that, including storage equipment And processor, the storage equipment are stored with the instruction for realizing video implantation target occlusion, described instruction is added by the processor When carrying and executing, following operation is executed:

For the coarse segmentation as a result, by image edge processing, the sport foreground subdivision of the area-of-interest is cut；

The application also provides a kind of electronic equipment for tracking target in video, is characterized in that, including storage equipment and processing Device, the instruction that the storage equipment is stored with target in tracking video are held when described instruction is loaded and executed by the processor The following operation of row:

It returns and obtains the location information in tracked target region and the acquiring unit of video frame to be processed in reference video frame, Until video is disposed.

The application also provides a kind of method of target area position in determining video, is characterized in that, comprising:

Using the characteristic point and the match point determine reference video frame to video frame to be processed affine transformation matrix；

The application also provides a kind of method for tracking target area in video, is characterized in that, comprising:

The application also provides a kind of for determining the storage equipment of target area position in video, is characterized in that, is stored with Instruction, described instruction can be loaded by processor and execute following operation:

The application also provides a kind of for tracking the storage equipment of target area in video, is characterized in that, is stored with instruction, Described instruction can be loaded by processor and execute following operation:

The application also provides a kind of electronic equipment that can determine target area position in video, including storage equipment and place Device is managed, is characterized in that, the storage equipment is stored with the instruction of target area position in determining video, and described instruction is by the place When reason device is loaded and executed, following operation is executed:

The application also provides a kind of electronic equipment that can track target area in video, including storage equipment and processing Device is characterized in that the storage equipment is stored with the instruction of target area in tracking video, and described instruction is added by the processor When carrying and executing, following operation is executed:

Compared with prior art, the method for sport foreground has the advantage that in a kind of identification video provided by the present application It can play the role of quickly real-time and accurately identifying sport foreground in video frame.

Compared with prior art, the method for target position has the advantage that in a kind of determining video provided by the present application It can play the role of steadily determining target position in the video frame, achieve the effect that the accuracy for improving target following.

Compared with prior art, a kind of method for realizing video implantation target occlusion effect provided by the present application has following Advantage: the effect blocked by sport foreground implantation target area can be quickly realized in video in real time.Play raising video The effect of the efficiency and effect of implantation.

Compared with prior art, mesh calibration method has the advantage that in a kind of tracking video provided by the present application The effect and efficiency that improve video implantation are played the role of in the position for quickly determining tracked target in video in real time.

Detailed description of the invention

Fig. 1 is the flow diagram of the embodiment of the method for sport foreground in a kind of identification video of the application；

Fig. 2 is that a kind of process of the embodiment of optional way of the method for sport foreground in a kind of identification video of the application is shown Meaning schematic diagram；

Fig. 3 is the flow diagram of the embodiment of the method for target position in a kind of determining video of the application；

Fig. 4 is a kind of flow diagram of the embodiment for the method for realizing video implantation target occlusion of the application；

Fig. 5 is the flow diagram of the embodiment of mesh calibration method in a kind of tracking video of the application；

Fig. 6 is the flow diagram of the embodiment of the method for target area position in a kind of determining video of the application；

Fig. 7 is the flow diagram of the embodiment of the method for target area in a kind of tracking video of the application；

Fig. 8 is the video frame schematic diagram to be processed of the embodiment of the method for sport foreground in a kind of identification video of the application；

Fig. 9 is the coarse segmentation result schematic diagram of the embodiment of the method for sport foreground in a kind of identification video of the application；

Figure 10 is that result schematic diagram is cut in the subdivision of the embodiment of the method for sport foreground in a kind of identification video of the application.

Specific embodiment

In the following description, numerous specific details are set forth in order to facilitate a full understanding of the present invention.But the present invention can be with Much it is different from other way described herein to implement, those skilled in the art can be without prejudice to intension of the present invention the case where Under do similar popularization, therefore the present invention is not limited to the specific embodiments disclosed below.

The embodiment of the method for sport foreground, to be processed to a frame when being implanted into video in a kind of identification video of the application For video frame is handled, flow diagram is as shown in Figure 1, include following operation:

Step 101, acquisition includes the video frame to be processed of video implantation target area.

For into video, the case where the image of implantation object, the object being implanted is in the scene that video is embodied, When the object for having movement, which is located at, to be implanted between object and video lens in the scene, need to identify the moving object Body, the i.e. sport foreground relative to the video implantation target area where implantation object, can realize the object of implantation by it The effect that the object of preceding movements blocks.

Wherein, sport foreground described in the present embodiment refers to relative to the object for being implanted into target area movement in video, and For the object visually relative to target to be implanted closer to camera lens or video observer, the object of movement is passing through mesh to be implanted The target to be implanted can be blocked when mark.For example, video frame to be processed as shown in Figure 8, when needing the image by a secondary poster to put When setting on the metope in video scene between two already existing posters, if in video someone (man in black in Fig. 8) from If passing by before metope (between the poster i.e. on camera lens and metope), need to identify the image institute of poster in video frame The corresponding region of the people to pass by before metope in the zone is just able to achieve the people to be passed by the image of the poster in implantation video The effect blocked.That is the image overlay area of man in black is the sport foreground region for needing to identify, namely the movement for needing to identify Prospect.

The video frame of the video is obtained in this frame, includes the information of implantation target area in the video frame, including The data of its position in the video frame, such as when video implantation target area is rectangle, the coordinate etc. of each angle point.

Other than can the position of video implanted region be directly set, following manner (step 1 to step can also be used Four) location information of the video implantation target area in the video frame to be processed is obtained:

Step 1 obtains the location information in tracked target region and video frame to be processed, the quilt in reference video frame Tracking target area is that video is implanted into target area.

The reference video frame be same video in include video be implanted into target area and its position known to video Frame.Under normal conditions, the first frame of the image for the object for needing to be implanted into can be used as reference video frame.When needing implanted object mark When the scene at place has larger displacement relative to the frame, reference video frame corresponding to subsequent video frame to be processed can be updated.

The reference video frame and video frame to be processed are converted to single channel black white image by step 2.

For color video frequency image, above-mentioned video frame, which is converted to single channel black white image, to be helped to reduce calculation amount, is mentioned High arithmetic speed, it is ensured that the real-time of processing.

Step 3, reference video frame and video frame to be processed for the single channel black white image are obtained from the ginseng Examine video frame to the video frame to be processed affine transformation matrix.

It, can be in any manner after the reference video frame and video frame to be processed are converted to single channel black white image It obtains from the reference video frame to the affine transformation matrix of the video frame to be processed.

Affine transformation can be understood as by the scaling to reference axis, rotation, and the transformation such as translation, former coordinate is new after transformation There is corresponding value in coordinate domain.More succinct says: affine transformation=linear transformation+translation.

The present embodiment provides following manner (step 3 〇 mono- to step 3 〇 tri-) obtain affine transformation matrix:

Step 3 〇 mono-, determination meets the requirements quantity in preset characteristic point region of search in reference video frame Characteristic point.

The reference video frame is the reference video frame for being converted to single channel black white image.

The preset characteristic point region of search be the preset characteristic point region of search be include by Target area is tracked, and area is not less than the region of the two times of sizes in tracked target region.Spy can be rapidly and accurately determined in this way Sign point.

When the characteristic point quantity determined in features described above point search region is less than preset characteristic point amount threshold When, after expanding characteristic point region of search, characteristic point is determined in widened characteristic point region of search.

For example, for no feature or the unconspicuous characteristic point region of search of feature, such as the white wall of bulk, it frequently results in Characteristic point number it is less.We find the characteristic point of surrounding by expanding the range of signature search frame outward step by step, directly It is greater than the threshold value of setting to the number of characteristic point.

Preferably characteristic point amount threshold will can be set as 20, can ensured while guaranteeing accuracy in real time Property.

When characteristic point quantity is very few, more characteristic points, characteristic point quantity can be obtained by expanding characteristic point region of search When enough, more accurate affine transformation matrix can be obtained, and then obtain better tracking effect.Feature is counted simultaneously Amount, which is limited in preset threshold value, to be also beneficial to reduce calculation amount, improves arithmetic speed.

The characteristic point can be any characteristic point algorithm and obtain characteristic point, as Harris angle point, ShiTomasi angle point, SURF angle point, FAST characteristic point, SIFT feature etc. can be reached using Harris angle point as characteristic point in the present embodiment Preferable effect.

Step 3 〇 bis- is preset in the match point region of search of size, in the video frame to be processed using list Point template matching process determines the characteristic point corresponding match point in the video frame to be processed.

The video frame to be processed is to be converted to the video frame to be processed of single channel black white image.

Specifically, can be to each characteristic point determined in reference video frame, setting of the setting comprising the specified point is big Small region is as pattern plate bolster.The pattern plate bolster includes: that area is that reference video frame area 1/1st to seven/15th is big Small rectangle.At this point, effect is preferable.For example, the pattern plate bolster can be the rectangle for including the characteristic point field, picture Vegetarian noodles product is 1/10th of reference video frame elemental area.

Slided on the image of match point region of search by way of sliding window again, by comparing template with match point search The similarity of the subgraph of the image in region finds the maximum subgraph of similarity.Template image is covered on each in original image A position is stored in the metric after calculating in result images matrix (R), and each position (x, y) in R includes matching Metric (matched degree), finally selects most matched result.

It the case where for being the match point region of search for presetting size with entire video frame to be processed, can be with aforementioned Pattern plate bolster, by row or column point by point search and the most matched region of the pattern plate bolster in video frame to be processed, to determine reference video Each of frame characteristic point corresponding match point in video frame to be processed；

It, can be with or for the case where part with video frame to be processed is the match point region of search for presetting size In video frame to be processed, corresponding in the match point region of search of the setting of the characteristic point position of reference video frame, search with The most matched region of the pattern plate bolster, to determine each of reference video frame characteristic point corresponding in video frame to be processed With point.

When not searching match point in the match point region of search, then the range of matching area is spread；It is expanding Match point is determined in the match point region of search of range.

Such as: the scanning search in the match point region of search of the setting of the characteristic point position corresponding to reference video frame The pattern plate bolster of 200 × 200 pixels；If not finding match point, expands the range of match point region of search, continue to search.

The size of the match point region of search for presetting size can be different according to the mobile speed of video lens And different, the fast situation of camera lens movement speed, the size of the match point region of search can be more on the contrary then smaller.Usually It can be set to the region of 200 × 200 pixels, to obtain preferable search result and faster search speed.

Step 3 〇 tri- determines reference video frame to the imitative of video frame to be processed using the characteristic point and the match point Penetrate transformation matrix.

It has been determined that the tool of can use obtains in the characteristic point and video frame to be processed of reference video frame after corresponding match point It obtains from the characteristic point to the corresponding affine transformation matrix of the match point.Using the matrix as from the reference video frame to institute State the affine transformation matrix of video frame to be processed.

Step 4 determines the tracked target region in the video frame to be processed using the affine transformation matrix Position, also just obtain location information of the video implantation target area in the video frame to be processed.

After obtaining from the reference video frame to the affine transformation matrix of the video frame to be processed, can use by with Position and the affine transformation matrix of the track target area in reference video frame obtain the tracked target region described The location information of position namely video implantation target area in the video frame to be processed in video frame to be processed.

Such as when the video is implanted into target area, i.e., when tracked target region is rectangle in reference video frame, Location information is the coordinate of its each angle point, by after this step operation, can obtain its in video frame to be processed it is corresponding The coordinate of each angle point, the i.e. coordinate of each angle point of the video implantation target area in the video frame to be processed.

Position of the tracked target region in the video frame to be processed is obtained, i.e., the described video is implanted into target area Domain is behind the position in the video frame to be processed, when position of the tracked target region in the video frame to be processed When displacement relative to reference video frame is less than the threshold value of setting, it can also be tracked in the previous video frame of video frame to be processed Position of the position of target area as tracked target region in video frame to be processed.

The influence shaken to video can be effectively inhibited in this way.

For example, position of the tracked target region obtained in the video frame to be processed is relative in reference video frame When the displacement of the position in tracked target region is 1 pixel, it is believed that be shake, and use in former frame video and be tracked Position of the position of target area as tracked target region described in this video frame to be processed.

Step 102, it is implanted into target area according to the video to determine, corresponding area-of-interest.

The area-of-interest (Region Of Interest) is determined by the position of video implantation target area, can be Rectangle, round or other closed figures.The area-of-interest includes video implantation target area.

For example, the image for the object that need to be implanted into is to be posted in two seas on metope for video frame to be processed shown in Fig. 8 The image of poster between report, the region that wherein box fences up is area-of-interest.The area-of-interest includes poster Image where region.

It can be seen that, the region that the image of man in black is covered in the area-of-interest is subsequent step needs simultaneously The sport foreground region identified.

Can first determine the video implantation target area in video frame to be processed, for example, it is artificial determine or from other programs into Journey obtains.The minimum for calculating the implantation target area further according to the video implantation target area in the video frame to be processed is external Rectangle or triangle finally calculate resulting minimum circumscribed rectangle or triangle Shape definition area-of-interest according to described.

It, can be according to practical feelings when according to the resulting minimum circumscribed rectangle of the calculating or triangle Shape definition area-of-interest The minimum circumscribed rectangle or triangle are considered as area-of-interest by condition.Alternatively, in the minimum circumscribed rectangle or triangle When area size is not less than the threshold value set, to area size's scaled down, until meeting the requirement of the threshold value, with contracting Region after small is as area-of-interest.

Different tool software also provides the method that the area-of-interest is arranged, such as Halcon, OpenCV, Matlab Various operators (Operator) and function are commonly used on equal machine vision softwares to acquire area-of-interest.

Setting area-of-interest entire video frame can not calculated in subsequent processing, only by regional area into Row calculates to reduce calculation amount, to guarantee the real-time of this method.

When the video frame to be processed is too big or the target of video implantation is excessive in the video frame to be processed, lead to it Area-of-interest is also excessive when being unfavorable for handling in real time, can according to area-of-interest described in preset scale smaller, with It is further reduced calculation amount, improves the real-time entirely handled.

Such as when the area-of-interest size of acquisition is 960 × 640 pixels, and when being unfavorable for handling in real time, by its according to Length-width ratio can satisfy the needs handled in real time when respectively reducing half to 480 × 320 pixel, then by the size of 480 × 320 pixels As area-of-interest.

Step 103, the movement for the area-of-interest, through prospect compared with background, to the area-of-interest Prospect carries out coarse segmentation.

In the present embodiment, the prospect and background it is more specific refer to obtain background model and present frame model respectively, lead to Present frame model is crossed compared with background model, to obtain the sport foreground in present frame；Also in order to detecting area-of-interest Interior sport foreground needs that the model of video frame to be processed is then compared processing with background model first to background modeling, To detect sport foreground.

It is many that this reality can be used for the algorithm that the present embodiment carries out coarse segmentation to the background and prospect of image.Such as: k nearest neighbor (K Nearest Neighbors) background modeling algorithm, GMM (mixed Gaussian background modeling method) VIBE background difference method, MOG (adaptive GMM) etc..

The scheme of the application obtains prospect gray scale binary map using the background modeling algorithm based on k nearest neighbor.The algorithm combines Mixed Gauss model (Gaussian Mixture Model, GMM), the priori knowledge and k nearest neighbor thought of parameter Estimation complete picture The background modeling of plain spatial distribution, and then realize to static target judgement and the extraction of foreground moving object.

The unknown point phase of type in background modeling algorithm comparison image based on k nearest neighbor (K Nearest Neighbors) The classification information of neighbor point, using the classification of the classification of the point more than the quantity point unknown as type.For example, the point unknown when type Around closest point have 5, wherein 3 prospects, 2 when being background, be prospect by the type decision of the unknown point of the type. So as to identify sport foreground region.

By being based on the background modeling algorithm of k nearest neighbor (K Nearest Neighbors), it is formed a secondary prospect gray scale Binary map, wherein prospect can be assigned a value of 255, and background can be assigned a value of 0.

So far, the area-of-interest is divided into foreground and background.

After obtaining sport foreground, morphologic closed operation can also be used to the binary map, be smoothed.

Contain noise in prospect binary map due to closing on algorithm acquisition by K, so (i.e. using morphologic closed operation First expanding and corrode again) operation can eliminate disconnected small noise spot, while rebuild the marginal portion information of loss.

For example, image shown in Fig. 9 is processing of the area-of-interest of video frame to be processed shown in Fig. 8 Jing Guo this step Obtained coarse segmentation is as a result, wherein white is sport foreground region namely the sport foreground that coarse segmentation obtains.

Step 104, for the coarse segmentation as a result, by image edge processing, before the movement of the area-of-interest Scape subdivision is cut, and motion foreground segmentation result is obtained.

The prospect of image and background are finely divided on the basis of coarse segmentation the mode cut can there are many, such as Level Set (level-set segmentation), fuzzy C-mean algorithm (FCM) clustering algorithm, minimal cut (Min Cut) algorithm based on figure etc., the application's Scheme uses the Grab cut algorithm based on graph theory, carries out sport foreground subdivision to the video frame to be processed and cuts.It can reach Good segmentation effect.

The Grab cut algorithm specifically includes:

Using the coarse segmentation result as boundary condition, i.e., the prospect binary map for the area-of-interest that the above frame obtains is made For boundary condition, foreground and background region is marked to the video frame to be processed using non-fully labeling method, is forced prospect Gray value is that 255 corresponding pixels are set as prospect in binary map, other are set as possible background.

Gauss hybrid models are established respectively to the foreground area and background area color space again.By partitioning estimation and The interactive iteration process of model parameter study determines foreground area and the background area of the video frame to be processed.

So far, the moving foreground object region segmentation in video frame to be processed relative to implantation target area is come out, is obtained Obtain sport foreground region.

In order to further increase the integrality and accuracy of the sport foreground profile detected.We are in Grab Cut image It is post-processed on the basis of the prospect background binary map that segmentation obtains, using edge smoothing (such as Gaussian smoothing) and fills out hole operation, Obtain more accurate motion foreground segmentation result-sport foreground region.

For example, image shown in Fig. 10 is what coarse segmentation processing result shown in Fig. 9 obtained after the processing of this step Thin segmentation result.Wherein white area is sport foreground region, namely the sport foreground that subdivision is cut.

The embodiment of the optional way of the method for sport foreground provides a kind of identification view in a kind of identification video of the application The method of sport foreground in frequency, process simplified schematic diagram as shown in Fig. 2, successively the following steps are included:

Video frame to be processed is obtained, ROI region is chosen, KNN background modeling, closed operation, Grab Cut segmentation, and boundary is flat It is sliding, hole operation is filled out, moving foreground object region is obtained, terminates video frame to be processed.

Being described in detail for each step can refer to the method for sport foreground in a kind of identification video provided by the present application Corresponding description in embodiment, this will not be repeated here.

The embodiment of the method for target position provides target position in a kind of determining video in a kind of determining video of the application The method set, flow diagram is as shown in figure 3, include following operation:

Step 301, the location information in tracked target region and video frame to be processed in reference video frame are obtained.

Step 302, the reference video frame and video frame to be processed are converted into single channel black white image.

Step 303, the reference video frame for the single channel black white image and video frame to be processed are obtained from the ginseng Examine video frame to the video frame to be processed affine transformation matrix.

Step 304, determine the tracked target region in the video frame to be processed using the affine transformation matrix In position.

The above-mentioned detailed description respectively operated can refer to the reality of the method for sport foreground in a kind of identification video of the application Step 101 in example is applied, acquisition includes the corresponding explanation in the video frame to be processed of video implantation target area, is not done herein It repeats.

A kind of embodiment of method for realizing video implantation target occlusion of the application provides a kind of realization video implantation mesh The method blocked is marked, flow diagram is as shown in figure 4, include following each operations:

Step 401, video to be processed is obtained, the video frame to be processed in the video includes video implantation target area Domain.

It include video implantation target area in the video.

Step 402, target area is implanted into according to the video, determines corresponding area-of-interest.

Step 403, the movement for the area-of-interest, through prospect compared with background, to the area-of-interest Prospect carries out coarse segmentation.

Step 404, for the coarse segmentation as a result, by image edge processing, before the movement of the area-of-interest Scape subdivision is cut.

Step 405, the foreground area for cutting acquisition to the subdivision operates, and the implantation target is blocked in realization.

By each operation of front, the range for needing to realize the sport foreground region blocked in video frame can be obtained, it is right This region is operated, and mask is such as arranged: when there is sport foreground in video implantation target area, we can be according to acquisition Sport foreground region set the region mask (Mask), i.e. the mask regions image that only shows corresponding sport foreground, without aobvious Show the image of the object corresponding part of video implantation.

The above-mentioned detailed description respectively operated can be with reference to the implementation of the method for sport foreground in a kind of identification video of the application Associated description in example, this will not be repeated here.

The embodiment of mesh calibration method provides mesh calibration method in a kind of tracking video in a kind of tracking video of the application, Its flow diagram is as shown in figure 5, include following each operations:

Step 501, it obtains video to be processed and determines reference video frame.

The first frame of the video to be processed includes that the tracked target region of position has been determined, then first frame is arranged For reference frame, the video frame setting of in the subsequent video frame it needs to be determined that position in the tracked region is video to be processed Frame.

If the non-first frame of the video to be processed includes that the tracked target region of position has been determined, by this first A includes to be determined that the video frame in the tracked target region of position is set as reference video frame, is needed in subsequent video frame The video frame for determining the position in the tracked region is video frame to be processed.

Step 502, the location information in tracked target region and video frame to be processed in reference video frame are obtained.

The video frame to be processed is for first after the reference video frame it needs to be determined that the position in the tracked region Video frame or it is any it needs to be determined that the position in the tracked region video frame.

Step 503, the reference video frame and video frame to be processed are converted into single channel black white image.

Step 504, the reference video frame for the single channel black white image and video frame to be processed are obtained from the ginseng Examine video frame to the video frame to be processed affine transformation matrix.

Step 505, determine the tracked target region in the video frame to be processed using the affine transformation matrix In position.

Step 506, new video frame to be processed is set and updates reference when meeting and updating preset renewal frequency Video frame.

When include in subsequent video frame in video it needs to be determined that the tracked region position video frame when, will after In continuous video frame first it needs to be determined that the video frame of the position in the tracked region is set as new video frame to be processed.

The frequency that can be additionally updated according to the mobile speed setting reference video frame of video lens, for video lens Mobile fast video, the frequency that reference video frame updates is high, on the contrary then low.The most fast situation of renewal frequency can be every handled New reference video frame is set at frame video frame to be processed.The previous video frame of i.e. new video frame to be processed is For the reference video frame corresponding to it.

Updating reference video frame enables to the processing to video to be processed more quick, saves time and calculation amount, has Conducive to the real-time of processing.

Step 507, it returns and obtains the location information in tracked target region and video frame to be processed in reference video frame Step, until video is disposed.

After setting up new video frame to be processed or setting up new video frame to be processed and have updated reference video frame, It carries out step 502 to new video frame to be processed to handle accordingly to step 505, until all frames of entire video have all been handled Finish, i.e., the described video all it needs to be determined that in the video frame of tracked target frame position, the position of tracked target frame all by After determination, terminate this method.

Being described in detail for respectively operating in the present embodiment can also be with reference to the side of sport foreground in a kind of identification video of the application Associated description in the embodiment of method in step 101, this will not be repeated here.

The embodiment of the storage equipment of sport foreground in a kind of video for identification of the application, provides a kind of store and sets It is standby, it is stored with instruction, described instruction can be loaded by processor and execute following operation:

The application's is a kind of for determining the embodiment of the storage equipment of target position in video, provides a kind of store and sets It is standby, it is stored with instruction, described instruction can be loaded by processor and execute following operation:

The 8th embodiment of the application provides a kind of storage equipment for realizing video implantation target occlusion, is stored with finger It enables, described instruction can be loaded by processor and execute following operation:

For the coarse segmentation as a result, by image edge processing, the sport foreground subdivision of the area-of-interest is cut, Obtain motion foreground segmentation result；

The application's is a kind of for tracking the embodiment of the storage equipment of target in video, provides a kind of storage equipment, It is stored with instruction, described instruction can be loaded by processor and execute following operation:

It obtains video to be processed and determines reference video frame；

A kind of embodiment that can be used in identifying the electronic equipment of sport foreground in video of the application, provides a kind of electronics Equipment, including storage equipment and processor, the storage equipment are stored with the instruction of sport foreground in identification video, described instruction When being loaded and executed by the processor, following operation is executed:

The embodiment of the electronic equipment of target position, provides a kind of electronic equipment in a kind of determining video of the application,

Including storage equipment and processor, the storage equipment is stored with the instruction of target position in determining video, described When instruction is loaded and executed by the processor, following operation is executed:

A kind of embodiment of electronic equipment for realizing video implantation target occlusion of the application, provides a kind of electronic equipment, Including storage equipment and processor, the storage equipment is stored with the instruction for realizing video implantation target occlusion, described instruction quilt When the processor is loaded and executed, following operation is executed:

The electronic equipment of target, provides a kind of electronic equipment in a kind of tracking video of the application,

Including storage equipment and processor, the storage equipment is stored with the instruction of target in tracking video, described instruction When being loaded and executed by the processor, following operation is executed:

Flow diagram such as Fig. 6 of the embodiment of the method for target area position in a kind of determining video provided by the present application It is shown, comprising the following steps:

Step 601, the location information in tracked target region and video frame to be processed in reference video frame are obtained；

Step 602, the spy for meeting the requirements quantity is determined in preset characteristic point region of search in reference video frame Sign point；

Step 603, it in the video frame to be processed, presets in the match point region of search of size, using single-point Template matching method determines the characteristic point corresponding match point in the video frame to be processed；

Step 604, determine reference video frame to the affine of video frame to be processed using the characteristic point and the match point Transformation matrix；

Step 605, determine the tracked target region in the video frame to be processed using the affine transformation matrix In position.

Flow diagram such as Fig. 7 institute of the embodiment of the method for target area in a kind of tracking video provided by the present application Show, comprising:

Step 701, it obtains video to be processed and determines reference video frame and video frame to be processed；

Step 702, the location information in tracked target region and video frame to be processed in reference video frame are obtained；

Step 703, the spy for meeting the requirements quantity is determined in preset characteristic point region of search in reference video frame Sign point；

Step 704, it in the video frame to be processed, presets in the match point region of search of size, using single-point Template matching method determines the characteristic point corresponding match point in the video frame to be processed；

Step 705, determine reference video frame to the affine of video frame to be processed using the characteristic point and the match point Transformation matrix；

Step 706, determine the tracked target region in the video frame to be processed using the affine transformation matrix In position；

Step 707, new video frame to be processed is set and updates reference when meeting and updating preset renewal frequency Video frame；

Step 708, it returns and obtains the location information in tracked target region and video frame to be processed in reference video frame Step 702, until video is disposed.

The application's is a kind of for determining the embodiment of the storage equipment of target area position in video, provides a kind of storage Equipment is stored with instruction, and described instruction can be loaded by processor and execute following operation:

The application's is a kind of for tracking the embodiment of the storage equipment of target area in video, provides a kind of store and sets It is standby, it is stored with instruction, described instruction can be loaded by processor and execute following operation:

A kind of embodiment that can determine the electronic equipment of target area position in video of the application, provides a kind of electronics Equipment, including storage equipment and processor, the storage equipment is stored with the instruction of target area position in determining video, described When instruction is loaded and executed by the processor, following operation is executed:

One kind of the application can track the embodiment of the electronic equipment of target area in video, provide a kind of electronics and set It is standby, including storage equipment and processor, the instruction for storing equipment and being stored with target area in tracking video, described instruction quilt When the processor is loaded and executed, following operation is executed:

Although the present invention is disclosed as above with preferred embodiment, it is not for limiting the present invention, any this field skill Art personnel without departing from the spirit and scope of the present invention, can make possible variation and modification, therefore guarantor of the invention Shield range should be subject to the range that the claims in the present invention are defined.

Claims

1. the method for sport foreground, is characterized in that in a kind of identification video, comprising:

For the area-of-interest, through prospect compared with background, rough segmentation is carried out to the sport foreground of the area-of-interest It cuts；

For the coarse segmentation as a result, by image edge processing, the sport foreground subdivision of the area-of-interest is cut, is obtained Motion foreground segmentation result.

2. the method for sport foreground, is characterized in that in identification video according to claim 1, described to be planted according to the video Enter target area, determines that corresponding area-of-interest includes:

The minimum circumscribed rectangle or three of the implantation target area is calculated according to the implantation target area in the video frame to be processed It is angular；

3. the method for sport foreground, is characterized in that in identification video according to claim 2, described according to the calculating institute Minimum circumscribed rectangle or triangle Shape definition area-of-interest include:

Or:

In threshold value of the area size of the minimum circumscribed rectangle or triangle not less than setting, described area size etc. is compared Example reduces, until meeting the requirement of the threshold value, using the region after reducing as area-of-interest.

4. the method for sport foreground, is characterized in that in identification video according to claim 1, described for described interested Region, through prospect compared with background, carrying out coarse segmentation to the sport foreground in the area-of-interest includes:

5. the method for sport foreground in identification video according to claim 4, which is characterized in that further include:

6. the method for sport foreground, is characterized in that in identification video according to claim 1, described to be directed to the coarse segmentation As a result, by image edge processing, the sport foreground subdivision in the area-of-interest is cut includes:

The Grab cut algorithm specifically includes:

Using the coarse segmentation result as boundary condition,

By the interactive iteration process that partitioning estimation and model parameter learn determine the video frame to be processed foreground area and Background area.

7. the method for sport foreground in identification video according to claim 6, which is characterized in that further include: to the figure As the result after edge processing algorithm process carries out Gaussian smoothing and fills out hole operation.

8. the method for sport foreground, is characterized in that, the acquisition includes video in identification video according to claim 1 Implantation target area video frame to be processed include:

Reference video frame and video frame to be processed for the single channel black white image are obtained from the reference video frame to institute State the affine transformation matrix of video frame to be processed；

Wherein, the tracked target region is that video is implanted into target area.

9. the method for sport foreground, is characterized in that in identification video according to claim 8, described to be directed to single channel black and white The reference video frame of image and video frame to be processed are obtained from the reference video frame to the affine change of the video frame to be processed Changing matrix includes:

It in the video frame to be processed, presets in the match point region of search of size, using single-point template matching method Determine the characteristic point corresponding match point in the video frame to be processed；

10. it is according to claim 9 identification video in sport foreground method, which is characterized in that it is described described wait locate It manages in video frame, the characteristic point corresponding matching in the video frame to be processed is determined using single-point template matching method Point, including

To each characteristic point determined in reference video frame, the region that is sized of the setting comprising the specified point is as template Frame；

With aforementioned pattern plate bolster, search and the most matched region of the pattern plate bolster in video frame to be processed, to determine reference video frame Each of characteristic point corresponding match point in video frame to be processed.

11. the method for sport foreground in identification video according to claim 10, which is characterized in that described with aforementioned template Frame, search and the most matched region of the pattern plate bolster in video frame to be processed, to determine each of reference video frame feature Point corresponding match point in video frame to be processed；Including

By row or column point by point search and the most matched region of the pattern plate bolster in video frame to be processed, to determine in reference video frame Each characteristic point in video frame to be processed corresponding match point；

Or in video frame to be processed, corresponding to the match point region of search of the setting of the characteristic point position of reference video frame It is interior, search and the most matched region of the pattern plate bolster, to determine each of reference video frame characteristic point in video frame to be processed In corresponding match point.

12. the method for sport foreground in identification video according to claim 10, which is characterized in that described with aforementioned template Frame, search and the most matched region of the pattern plate bolster in video frame to be processed, to determine each of reference video frame feature Point corresponding match point in video frame to be processed；Specially

In video frame to be processed, corresponding in the match point region of search of the setting of the characteristic point position of reference video frame, search Rope and the most matched region of the pattern plate bolster, to determine that each of reference video frame characteristic point is corresponding in video frame to be processed Match point；

13. the method for sport foreground, is characterized in that, the preset spy in identification video according to claim 9 Sign point search region is includes tracked target region, and area is not less than the region of the two times of sizes in tracked target region.

14. the method for sport foreground, is characterized in that in identification video according to claim 13, described in reference video frame In determine that the characteristic point for meeting quantitative requirement includes: in preset characteristic point region of search

15. the method for sport foreground, is characterized in that, the preset spy in identification video according to claim 14 Sign point amount threshold is 20.

16. the method for sport foreground, is characterized in that, the pattern plate bolster includes: in identification video according to claim 10 Area is the rectangle of 1/1st to seven/15th size of reference video frame area.

17. the method for sport foreground, is characterized in that, the characteristic point includes in identification video according to claim 9 Harris angle point, ShiTomasi angle point, SURF angle point, FAST characteristic point or SIFT feature.

18. the method for sport foreground, is characterized in that, determines in reference video frame in identification video according to claim 8 The location information in tracked target region includes

19. the method for sport foreground, is characterized in that in identification video according to claim 8, described using described affine Transformation matrix determines the tracked target region behind the position in the video frame to be processed further include:

When position of the tracked target region in the video frame to be processed is less than relative to the displacement of reference video frame When the threshold value of setting, use the position in tracked target region in the previous video frame of video frame to be processed as tracked target region Position in video frame to be processed.

20. the method for target position, is characterized in that in a kind of determining video, comprising:

21. the method for target position, is characterized in that in determining video according to claim 20, described black for single channel The reference video frame of white image and video frame to be processed are obtained from the reference video frame to the affine of the video frame to be processed Transformation matrix includes:

22. the method for target position in determining video according to claim 21, which is characterized in that it is described described wait locate It manages in video frame, the characteristic point corresponding matching in the video frame to be processed is determined using single-point template matching method Point, including

23. the method for target position in determining video according to claim 22, which is characterized in that described with aforementioned template Frame, search and the most matched region of the pattern plate bolster in video frame to be processed, to determine each of reference video frame feature Point corresponding match point in video frame to be processed；Including

24. the method for target position in determining video according to claim 22, which is characterized in that described with aforementioned template Frame, search and the most matched region of the pattern plate bolster in video frame to be processed, to determine each of reference video frame feature Point corresponding match point in video frame to be processed；Specially

25. the method for target position, is characterized in that in determining video according to claim 21, the preset spy Sign point search region is includes tracked target region, and area is not less than the region of the two times of sizes in tracked target region.

26. the method for target position, is characterized in that in determining video according to claim 21, described in reference video frame In determine that the characteristic point for meeting quantitative requirement includes: in preset characteristic point region of search

27. the method for target position, is characterized in that in determining video according to claim 26, the preset spy Sign point amount threshold is 20.

28. the method for target position, is characterized in that in determining video according to claim 22, the pattern plate bolster includes: It is the rectangle of 1/1st to seven/15th size of reference video frame area for area.

29. the method for target position, is characterized in that in determining video according to claim 21, the characteristic point includes Harris angle point, ShiTomasi angle point, SURF angle point, FAST characteristic point or SIFT feature.

30. the method for target position, is characterized in that, determines in reference video frame in determining video according to claim 20 The location information in tracked target region includes

31. the method for target position, is characterized in that in determining video according to claim 20, described using described affine Transformation matrix determines the tracked target region behind the position in the video frame to be processed further include:

32. a kind of method for realizing video implantation target occlusion, is characterized in that, comprising:

33. mesh calibration method in a kind of tracking video, is characterized in that, comprising:

It returns and obtains in reference video frame the location information in tracked target region and the step of video frame to be processed, until video It is disposed.

34. mesh calibration method in tracking video according to claim 33, is characterized in that, described update in satisfaction sets in advance Reference video frame is updated when fixed renewal frequency includes:

The reference video for being directed to video frame to be processed is updated according to the frequency of setting according to the movement speed of the camera lens of shooting video Frame；Or

35. the storage equipment of sport foreground, is characterized in that, be stored with instruction, described instruction can in a kind of video for identification It is loaded by processor and executes following operation:

36. it is a kind of for determining the storage equipment of target position in video, it is characterized in that, be stored with instruction, described instruction can It is loaded by processor and executes following operation:

37. a kind of storage equipment for realizing video implantation target occlusion, is characterized in that, is stored with instruction, described instruction energy It is enough to be loaded by processor and execute following operation:

38. it is a kind of for tracking the storage equipment of target in video, it is characterized in that, be stored with instruction, described instruction can be located Reason device loads and executes following operation:

39. a kind of electronic equipment that can be used in identifying sport foreground in video, is characterized in that, including storage equipment and processing Device, the storage equipment are stored with the instruction of sport foreground in identification video, and described instruction is loaded and executed by the processor When, execute following operation:

40. a kind of electronic equipment that can determine target position in video, is characterized in that, including storage equipment and processor, institute It states storage equipment and is stored with the instruction of target position in determining video and held when described instruction is loaded and executed by the processor The following operation of row:

41. a kind of electronic equipment for realizing video implantation target occlusion, is characterized in that, including storage equipment and processor, described Storage equipment, which is stored with, realizes that the instruction of video implantation target occlusion is held when described instruction is loaded and executed by the processor The following operation of row:

42. the electronic equipment of target, is characterized in that in a kind of tracking video, including storage equipment and processor, the storage are set The standby instruction for being stored with target in tracking video when described instruction is loaded and executed by the processor, executes following operation:

43. the method for target area position, is characterized in that in a kind of determining video, comprising:

44. the method for target area, is characterized in that in a kind of tracking video, comprising:

45. it is a kind of for determining the storage equipment of target area position in video, it is characterized in that, is stored with instruction, described instruction It can be loaded by processor and execute following operation:

46. it is a kind of for tracking the storage equipment of target area in video, it is characterized in that, be stored with instruction, described instruction can It is loaded by processor and executes following operation:

47. a kind of electronic equipment that can determine target area position in video, including storage equipment and processor, feature exist In the storage equipment is stored with the instruction of target area position in determining video, and described instruction is loaded simultaneously by the processor When execution, following operation is executed:

48. one kind can track the electronic equipment of target area in video, including storage equipment and processor, it is characterized in that, institute It states storage equipment and is stored with the instruction of target area in tracking video and held when described instruction is loaded and executed by the processor The following operation of row: