CN107590795A

CN107590795A - Image processing method and device, electronic installation and computer-readable recording medium

Info

Publication number: CN107590795A
Application number: CN201710813594.1A
Authority: CN
Inventors: 张学勇
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2017-09-11
Filing date: 2017-09-11
Publication date: 2018-01-16

Abstract

It is used to handle the invention discloses a kind of image processing method and merges image.Merging image is merged by personage's area image of the predetermined three-dimensional background image with active user in the scene image under real scene to be formed.Image processing method includes：Identification merges the certain objects in image；By the predetermined sound model matched with certain objects with merging image co-registration to have exported acoustic image.The invention also discloses a kind of image processing apparatus, electronic installation and computer-readable recording medium.Image processing method, image processing apparatus, electronic installation and the computer-readable recording medium of embodiment of the present invention carry out the identification of certain objects to the predetermined three-dimensional background image for merging image, and the predetermined sound model for determining to match with certain objects according to the certain objects that recognize has acoustic image so that predetermined sound model is exported with merging image co-registration, so as to which user can also hear sound while viewing merges image, strengthen the interest of image co-registration, improve the usage experience of user.

Description

Image processing method and device, electronic installation and computer-readable recording medium

Technical field

The present invention relates to technical field of image processing, more particularly to a kind of image processing method and device, electronic installation and Computer-readable recording medium.

Background technology

Existing image co-registration is typically to be merged real person with background image, but the entertaining of such a amalgamation mode Property is relatively low.

The content of the invention

Can the embodiment provides a kind of image processing method, image processing apparatus, electronic installation and computer Read storage medium.

The image processing method of embodiment of the present invention, which is used to handle, merges image, and the merging image is carried on the back by predetermined three-dimensional Personage area image of the scape image with active user in the scene image under real scene is merged into described image processing Method includes：

Identify the certain objects in the merging image；With

Image co-registration will be merged with the predetermined sound model that the certain objects match with described to have exported acoustic image.

The image processing apparatus of embodiment of the present invention, which is used to handle, merges image, and the merging image is by described predetermined three Personage area image of the dimension background image with active user in the scene image under real scene, which merges, to be formed, at described image Reason device includes processor, and the processor is used to identifying the certain objects merged in image, and will with it is described specific The predetermined sound model of object matches merges image co-registration to have exported acoustic image with described.

The electronic installation of embodiment of the present invention includes one or more processors, memory and one or more programs. Wherein one or more of programs are stored in the memory, and are configured to by one or more of processors Perform.Described program includes the instruction for above-mentioned image processing method.

The computer-readable recording medium of embodiment of the present invention, including electronic installation with that can image are used in combination Computer program.The computer program can be executed by processor to complete upper described image processing method.

Image processing method, image processing apparatus, electronic installation and the computer-readable storage medium of embodiment of the present invention When matter merges personage's area image with predetermined three-dimensional background image to form merging image, the predetermined three-dimensional merged in image is carried on the back Scape image carries out the identification of certain objects, and the predetermined sound for determining to match with certain objects according to the certain objects recognized Model, predetermined sound model there is into acoustic image with merging image co-registration output, so as to which user is while viewing merges image Sound can be also heard, strengthens the interest of image co-registration, user is had sense on the spot in person, improves the usage experience of user.

The additional aspect and advantage of the present invention will be set forth in part in the description, and will partly become from the following description Obtain substantially, or recognized by the practice of the present invention.

Brief description of the drawings

Of the invention above-mentioned and/or additional aspect and advantage will become from the following description of the accompanying drawings of embodiments Substantially and it is readily appreciated that, wherein：

Fig. 1 is the schematic flow sheet of the image processing method of some embodiments of the present invention.

Fig. 2 is the schematic diagram of the image processing apparatus of some embodiments of the present invention.

Fig. 3 is the structural representation of the electronic installation of some embodiments of the present invention.

Fig. 4 is the schematic flow sheet of the image processing method of some embodiments of the present invention.

Fig. 5 is the schematic flow sheet of the image processing method of some embodiments of the present invention.

Fig. 6 is the schematic flow sheet of the image processing method of some embodiments of the present invention.

Fig. 7 (a) to Fig. 7 (e) is the schematic diagram of a scenario of structural light measurement according to an embodiment of the invention.

Fig. 8 (a) and Fig. 8 (b) is the schematic diagram of a scenario of structural light measurement according to an embodiment of the invention.

Fig. 9 is the schematic flow sheet of the image processing method of some embodiments of the present invention.

Figure 10 is the schematic flow sheet of the image processing method of some embodiments of the present invention.

Figure 11 is the schematic flow sheet of the image processing method of some embodiments of the present invention.

Figure 12 is the schematic flow sheet of the image processing method of some embodiments of the present invention.

Figure 13 is the schematic flow sheet of the image processing method of some embodiments of the present invention.

Figure 14 is the schematic diagram of the image processing apparatus of some embodiments of the present invention.

Figure 15 is the schematic diagram of the electronic installation of some embodiments of the present invention.

Embodiment

Embodiments of the invention are described below in detail, the example of the embodiment is shown in the drawings, wherein from beginning to end Same or similar label represents same or similar element or the element with same or like function.Below with reference to attached The embodiment of figure description is exemplary, it is intended to for explaining the present invention, and is not considered as limiting the invention.

Referring to Fig. 1, the image processing method of embodiment of the present invention, which is used to handle, merges image.Merge image by making a reservation for Three-dimensional background image is merged with personage's area image and formed, and personage's area image is what active user shot under real scene The image of active user region in scene image.Image processing method includes：

03：Identification merges the certain objects in image；

04：By the predetermined sound model matched with certain objects with merging image co-registration to have exported acoustic image.

Referring to Fig. 2, the image processing method of embodiment of the present invention can be by the image procossing of embodiment of the present invention Device 100.Image processing apparatus 100, which is used to handle, merges image.Merge image by predetermined three-dimensional background image and people's object area Image co-registration forms.In the scene image that personage's area image shoots for active user under real scene where active user The image in region.Image processing apparatus 100 includes processor 20.Step 03 and step 04 can be realized by processor 20. I other words processor 20 can be used for identification to merge the certain objects in image, and the predetermined sound that will be matched with certain objects Model is with merging image co-registration to have exported acoustic image.

Referring to Fig. 3, the image processing apparatus 100 of embodiment of the present invention can apply to the electricity of embodiment of the present invention Sub-device 1000.In other words, the electronic installation 1000 of embodiment of the present invention includes the image procossing of embodiment of the present invention Device 100.

In some embodiments, electronic installation 1000 includes mobile phone, tablet personal computer, notebook computer, Intelligent bracelet, intelligence Energy wrist-watch, intelligent helmet, intelligent glasses etc..

In some embodiments, predetermined three-dimensional background image can be that the predetermined three-dimensional for modeling to obtain by actual scene is carried on the back The predetermined three-dimensional background image that scape image or cartoon making obtain.Predetermined three-dimensional background image can be by processor 20 It is randomly assigned, or is voluntarily selected by active user.

Image processing method, image processing apparatus 100 and the electronic installation 1000 of embodiment of the present invention are by people's object area When image merges to form merging image with predetermined three-dimensional background image, the predetermined three-dimensional background image merged in image is carried out special The identification of earnest body, and the predetermined sound model for determining to match with certain objects according to the certain objects recognized, will be pre- Determine sound model has acoustic image with merging image co-registration output, so as to which user can also hear sound while viewing merges image Sound, strengthen the interest of image co-registration, user is had sense on the spot in person, improve the usage experience of user.

In some embodiments, certain objects include animal, plant, flowing water, raindrop, musical instrument, flame, sky, road, Automobile etc..For example, the predetermined three-dimensional background image merged in image includes trees, then processor 20 can pairing by the following method And the trees in image are identified：Processor 20 is primarily based on the color histogram of rgb space to merging image or predetermined three Tie up background image and carry out color feature extracted, be then based on Gabor filter and carry out texture feature extraction, it is finally special according to color The information that textural characteristics of seeking peace combine, which determines to merge in image, has trees.Then, processor 20 can choose one section of wind trees Afterwards, the predetermined sound model that the leaf of trees susurrates is merged with merging image.For another example merge in image comprising dynamic Thing, then processor 20 the merging image of rgb space can be converted to the merging image of HSV space, then the merging figure to HSV space As carrying out color histogram calculating, low order statistical moment numerical value is adjusted by color histogram characteristic and is used as feature description amount, finally The generic for merging image is judged by K neighbor methods again, that is, merge in image with the presence or absence of animal and animal be present When animal classification, the predetermined sound model that final choice matches with the animal recognized.

Referring to Fig. 4, in some embodiments, the image processing method of embodiment of the present invention also includes：

021：Obtain the scene image of active user；

022：Obtain the depth image of active user；

023：Processing scene image and depth image are obtained with extracting people's object area of the active user in scene image Personage's area image；With

024：Personage's area image is merged to obtain merging image with predetermined three-dimensional background image.

Referring again to Fig. 3, in some embodiments, image processing apparatus 100 is also including visible image capturing first 11 and deeply Spend image collection assembly 12.Step 021 can realize that step 022 can gather group by depth image by visible image capturing first 11 Part 12 is realized.Step 023 and step 024 can be realized by processor 20.

In other words, it is seen that light video camera head 11 can be used for the scene image for obtaining active user.Depth image acquisition component 12 can be used for the depth image of acquisition active user.It is current to extract that processor 20 can be used for processing scene image and depth image People object area of the user in scene image and obtain personage's area image, and by personage's area image and predetermined three-dimensional background Image co-registration with obtain merge image.

Wherein, for that can be gray level image or coloured image, depth image characterizes to be included residing for active user scene image Each personal or object depth information in real scene.The scene domain of scene image is consistent with the scene domain of depth image, And each pixel in scene image can be found in depth image to should pixel depth information.

The method of existing segmentation personage and background according to similitude of the adjacent pixel in terms of pixel value and does not connect mainly Continuous property carries out the segmentation of personage and background, but this dividing method is easily influenceed by environmental factors such as ambient light photographs.It is of the invention real The image processing method for applying mode is gone out personage's extracted region in scene image by obtaining the depth image of active user Come.Influenceed because the acquisition of depth image is not easy the factor such as COLOR COMPOSITION THROUGH DISTRIBUTION in by illumination, scene, therefore, pass through depth image The people's object area extracted is more accurate, it is particularly possible to which accurate calibration goes out the border of people's object area.Further, more accurately Personage's area image merged with predetermined three-dimensional background image after merging image it is better.

Referring to Fig. 5, in some embodiments, the depth image that step 022 obtains active user includes：

0221：To active user's projective structure light；

0222：The structure light image that shooting is modulated through active user；With

0223：Phase information corresponding to each pixel of demodulation structure light image is to obtain depth image.

Referring again to Fig. 2, in some embodiments, depth image acquisition component 12 includes the He of structured light projector 121 Structure light video camera head 122.Step 0221 can be realized that step 0222 and step 0223 can be by tying by structured light projector 121 Structure light video camera head 122 is realized.

In other words, structured light projector 121 be able to can be used to active user's projective structure light, structure light video camera head 122 In shooting through active user modulate structure light image, and phase information corresponding to each pixel of demodulation structure light image with Obtain depth image.

Specifically, structured light projector 121 is by the face and body of the project structured light of certain pattern to active user Afterwards, the structure light image after being modulated by active user can be formed in the face of active user and the surface of body.Structure light images Structure light image after first 122 shooting is modulated, then structure light image is demodulated to obtain depth image.Wherein, structure The pattern of light can be laser stripe, Gray code, sine streak, non-homogeneous speckle etc..

Referring to Fig. 6, in some real-time modes, phase corresponding to each pixel of step 0223 demodulation structure light image Information is included with obtaining depth image：

02231：Phase information corresponding to each pixel in demodulation structure light image；

02232：Phase information is converted into depth information；With

02233：Depth image is generated according to depth information.

Referring again to Fig. 2, in some embodiments, step 02231, step 02232 and step 02233 can be by tying Structure light video camera head 122 is realized.

In other words, structure light video camera head 122 can be additionally used in phase letter corresponding to each pixel in demodulation structure light image Breath, phase information is converted into depth information, and depth image is generated according to depth information.

Specifically, compared with non-modulated structure light, the phase information of the structure light after modulation is changed, and is being tied The structure light showed in structure light image is to generate the structure light after distortion, wherein, the phase information of change can characterize The depth information of object.Therefore, structure light video camera head 122 demodulates phase corresponding to each pixel in structure light image and believed first Breath, calculates depth information, so as to obtain final depth image further according to phase information.

In order that those skilled in the art be more apparent from according to structure light come gather active user face and The process of the depth image of body, illustrated below by taking a kind of widely used optical grating projection technology (fringe projection technology) as an example Its concrete principle.Wherein, optical grating projection technology belongs to sensu lato area-structure light.

As shown in Fig. 7 (a), when being projected using area-structure light, sine streak is produced by computer programming first, And sine streak is projected to measured object by structured light projector 121, recycle structure light video camera head 122 to shoot striped by thing Degree of crook after body modulation, then demodulates the curved stripes and obtains phase, then phase is converted into depth information to obtain Depth image.The problem of to avoid producing error or error coupler, needed before carrying out depth information collection using structure light to depth Image collection assembly 12 carries out parameter calibration, and demarcation includes geometric parameter (for example, structure light video camera head 122 and project structured light Relative position parameter between device 121 etc.) demarcation, the inner parameter and structured light projector 121 of structure light video camera head 122 The demarcation of inner parameter etc..

Specifically, the first step, computer programming produce sine streak.Need to obtain using the striped of distortion due to follow-up Phase, for example phase is obtained using four step phase-shifting methods, therefore the striped that four width phase differences are pi/2, then structure light are produced here The projector 121 projects the four spokes line timesharing on measured object (mask shown in Fig. 7 (a)), and structure light video camera head 122 gathers To the figure on such as Fig. 7 (b) left sides, while to read the striped of the plane of reference shown on the right of Fig. 7 (b).

Second step, carry out phase recovery.The bar graph that structure light video camera head 122 is modulated according to four width collected is (i.e. Structure light image) to calculate the phase diagram by phase modulation, now obtained be to block phase diagram.Because four step Phase-shifting algorithms obtain Result be that gained is calculated by arctan function, therefore the phase after structure light modulation is limited between [- π, π], that is, Say, the phase after modulation exceedes [- π, π], and it can restart again.Shown in the phase main value such as Fig. 7 (c) finally given.

Wherein, it is necessary to carry out the saltus step processing that disappears, it is continuous phase that will block phase recovery during phase recovery is carried out Position.As shown in Fig. 7 (d), the left side is the continuous phase bitmap modulated, and the right is to refer to continuous phase bitmap.

3rd step, subtract each other to obtain phase difference (i.e. phase information) by the continuous phase modulated and with reference to continuous phase, should Phase difference characterizes depth information of the measured object with respect to the plane of reference, then phase difference is substituted into the conversion formula (public affairs of phase and depth The parameter being related in formula is by demarcation), you can obtain the threedimensional model of the object under test as shown in Fig. 7 (e).

It should be appreciated that in actual applications, according to the difference of concrete application scene, employed in the embodiment of the present invention Structure light in addition to above-mentioned grating, can also be other arbitrary graphic patterns.

As a kind of possible implementation, the depth information of pattern light progress active user also can be used in the present invention Collection.

Specifically, the method that pattern light obtains depth information is that this spreads out using a diffraction element for being essentially flat board The relief diffraction structure that there are element particular phases to be distributed is penetrated, cross section is with two or more concavo-convex step embossment knots Structure.Substantially 1 micron of the thickness of substrate in diffraction element, each step it is highly non-uniform, the span of height can be 0.7 Micron~0.9 micron.Structure shown in Fig. 8 (a) is the local diffraction structure of the collimation beam splitting element of the present embodiment.Fig. 8 (b) is edge The unit of the cross sectional side view of section A-A, abscissa and ordinate is micron.The speckle pattern of pattern photogenerated has The randomness of height, and can with the difference of distance changing patterns.Therefore, depth information is being obtained using pattern light Before, it is necessary first to the speckle pattern in space is calibrated, for example, in the range of 0~4 meter of distance structure light video camera head 122, A reference planes are taken every 1 centimetre, then just save 400 width speckle images after demarcating, the spacing of demarcation is smaller, obtains Depth information precision it is higher.Then, structured light projector 121 is by pattern light projection to measured object (i.e. active user) On, the speckle pattern that the difference in height on measured object surface to project the pattern light on measured object changes.Structure light Camera 122 is shot project speckle pattern (i.e. structure light image) on measured object after, then by speckle pattern and demarcation early stage The 400 width speckle images preserved afterwards carry out computing cross-correlation one by one, and then obtain 400 width correlation chart pictures.Measured object in space Position where body can show peak value on correlation chart picture, above-mentioned peak value is superimposed and after interpolation arithmetic i.e. It can obtain the depth information of measured object.

Multi beam diffraction light is obtained after diffraction is carried out to light beam due to common diffraction element, but per beam diffraction light light intensity difference Greatly, it is also big to the risk of human eye injury.Re-diffraction even is carried out to diffraction light, the uniformity of obtained light beam is relatively low. Therefore, the effect projected using the light beam of common diffraction element diffraction to measured object is poor.Using collimation in the present embodiment Beam splitting element, the element not only have the function that to collimate uncollimated rays, also have the function that light splitting, i.e., through speculum The non-collimated light of reflection is emitted multi-beam collimation light beam, and the multi-beam collimation being emitted after collimating beam splitting element toward different angles The area of section approximately equal of light beam, flux of energy approximately equal, and then to carry out using the scatterplot light after the beam diffraction The effect of projection is more preferable.Meanwhile laser emitting light is dispersed to every light beam, the risk of injury human eye is reduce further, and dissipate Spot structure light is for other uniform structure lights of arrangement, when reaching same collection effect, the consumption of pattern light Electricity is lower.

Referring to Fig. 9, in some embodiments, step 023 handles scene image and depth image to extract current use People object area of the family in scene image and obtaining personage's area image includes：

0231：Identify the human face region in scene image；

0232：Depth information corresponding with human face region is obtained from depth image；

0233：The depth bounds of people's object area is determined according to the depth information of human face region；With

0234：The personage area for determining to be connected and fallen into depth bounds with human face region according to the depth bounds of people's object area Domain is to obtain personage's area image.

Referring again to Fig. 2, in some embodiments, step 0231, step 0232, step 0233 and step 0234 To be realized by processor 20.In other words, the human face region that processor 20 can be used in identification scene image, from depth image Depth information corresponding with human face region is obtained, the depth bounds of people's object area is determined according to the depth information of human face region, with And determine to be connected with human face region according to the depth bounds of people's object area and people's object area for falling into depth bounds is to obtain people Object area image.

Specifically, the human face region that the deep learning Model Identification trained can be used to go out in scene image first, with The depth information of human face region is can determine that according to the corresponding relation of scene image and depth image afterwards.Because human face region includes The features such as nose, eyes, ear, lip, therefore, depth number of each feature corresponding in depth image in human face region According to being different, for example, in face face depth image acquisition component 12, depth that depth image acquisition component 12 is shot In image, depth data corresponding to nose may be smaller, and depth data corresponding to ear may be larger.Therefore, above-mentioned people The depth information in face region may be a numerical value or a number range.Wherein, when the depth information of human face region is one During individual numerical value, the numerical value can be by averaging to obtain to the depth data of human face region；Or can be by human face region Depth data take in be worth to.

Because people's object area includes human face region, in other words, people's object area is in some depth together with human face region In the range of, therefore, after processor 20 determines the depth information of human face region, it can be set according to the depth information of human face region The depth bounds of people's object area, the depth bounds extraction further according to people's object area fall into the depth bounds and with human face region phase People's object area of connection is to obtain personage's area image.

In this way, personage's area image can be extracted from scene image according to depth information.Due to obtaining for depth information The image of the not factor such as illumination, colour temperature in by environment is taken to ring, therefore, the personage's area image extracted is more accurate.

Referring again to Fig. 9, in some embodiments, step 023 processing scene image and depth image are current to extract People object area of the user in scene image and obtaining personage's area image also includes：

0235：Scene image is handled to obtain the whole audience edge image of scene image；With

0236：According to whole audience edge image amendment personage's area image of scene image.

Referring again to Fig. 2, in some embodiments, step 0235 and step 0236 can be realized by processor 20. In other words, processor 20 can also be used to handle scene image to obtain the whole audience edge image of scene image, and according to field Whole audience edge image amendment personage's area image of scape image.

Processor 20 carries out edge extracting to obtain the whole audience edge image of scene image to scene image first, wherein, Edge lines in the whole audience edge image of scene image include background object in scene residing for active user and active user Edge lines.Specifically, edge extracting can be carried out to scene image by Canny operators.Canny operators carry out edge extracting The core of algorithm mainly include the following steps：First, convolution is carried out to scene image with 2D gaussian filterings template to make an uproar to eliminate Sound；Then, the Grad of the gray scale of each pixel is obtained using differential operator, and the gray scale of each pixel is calculated according to Grad Gradient direction, adjacent pixels of the respective pixel along gradient direction can be found by gradient direction；Then, each picture is traveled through Element, if the gray value of some pixel is not maximum compared with the gray value of former and later two adjacent pixels on its gradient direction, that It is not marginal point to think this pixel.In this way, the pixel that marginal position is in scene image is can determine that, so as to obtain The whole audience edge image of scene image after edge extracting.

Processor 20 obtain scene image whole audience edge image after, further according to scene image whole audience edge image to people Object area image is modified.It is appreciated that personage's area image is will to be connected and fall into set with human face region in scene image Obtained after all pixels progress merger of fixed depth bounds, in some scenarios, it is understood that there may be some and human face region connect The object for connecing and falling into depth bounds.Therefore, to cause personage's area image of extraction more accurate, scene image can be used The whole audience edge graph personage's area image is modified.

Further, processor 20 can also carry out second-order correction to revised personage's area image, for example, can be to amendment Personage's area image afterwards carries out expansion process, expands personage's area image to retain the edge details of personage's area image.

Referring to Fig. 10, in some embodiments, step 024 melts personage's area image and predetermined three-dimensional background image Close is included with obtaining merging image：

02411：Obtain the predetermined integration region in predetermined three-dimensional background image；

02412：The pixel region to be replaced of predetermined integration region is determined according to personage's area image；With

02413：The pixel region to be replaced of predetermined integration region is replaced with into personage's area image to obtain merging image.

Referring again to Fig. 2, in some embodiments, step 02411, step 02412 and step 02413 can be by handling Device 20 is realized.In other words, processor 20 can be used for obtaining the predetermined integration region in predetermined three-dimensional background image, according to personage Area image determines the pixel region to be replaced of predetermined integration region, and the pixel region to be replaced of predetermined integration region is replaced Personage's area image is changed to obtain merging image.

It is appreciated that when predetermined three-dimensional background image models to obtain by actual scene, in predetermined three-dimensional background image Depth data can be obtained directly in modeling process corresponding to each pixel；Pass through cartoon making in predetermined three-dimensional background image When obtaining, depth data corresponding to each pixel can be by producer's sets itself in predetermined three-dimensional background image；It is in addition, predetermined Each object present in three-dimensional background image is also known, therefore, is melted carrying out image using predetermined three-dimensional background image Before processing is closed, personage's area image first can be calibrated according to depth data and the object being present in predetermined three-dimensional background image Fusion position, i.e., predetermined integration region.The size of the personage's area image collected due to visible image capturing first 11 is gathered The influence of distance, when gathering closer to the distance, personage's area image is larger, and when gathering distant, personage's area image is smaller, because This, processor 20 need to according to first 11 actual acquisition of visible image capturing to the size of personage's area image determine predetermined integration region In pixel region to be replaced.Then, the pixel region to be replaced in predetermined integration region is replaced with into personage's area image i.e. Merging image after being merged.In this way, realize merging for personage's area image and predetermined three-dimensional background image.

Figure 11 is referred to, in some embodiments, step 024 melts personage's area image and predetermined three-dimensional background image Close is included with obtaining merging image：

02421：Predetermined three-dimensional background image is handled to obtain the whole audience edge image of predetermined three-dimensional background image；

02422：Obtain the depth data of predetermined three-dimensional background image；

02423：Predetermined three-dimensional Background is determined according to the whole audience edge image and depth data of predetermined three-dimensional background image The calculating integration region of picture；

02424：Determined to calculate the pixel region to be replaced of integration region according to personage's area image；With

02425：The pixel region to be replaced for calculating integration region is replaced with into personage's area image to obtain merging image.

Referring again to Fig. 2, in some embodiments, step 02421, step 02422, step 02423, step 02424 It can be realized with step 02425 by processor 20.In other words, processor 20 can be used for processing predetermined three-dimensional background image with The whole audience edge image of predetermined three-dimensional background image is obtained, obtains the depth data of predetermined three-dimensional background image, according to predetermined three Tie up the whole audience edge image of background image and depth data determines the calculating integration region of predetermined three-dimensional background image, according to personage Area image is determined to calculate the pixel region to be replaced of integration region, and the pixel region to be replaced for calculating integration region is replaced Personage's area image is changed to obtain merging image.

It is appreciated that when if predetermined three-dimensional background image merges with personage's area image, the fusion position of personage's area image Not demarcation in advance is put, then processor 20 needs to determine fusion position of personage's area image in predetermined three-dimensional background image first. Specifically, processor 20 first carries out edge extracting to obtain whole audience edge image to predetermined three-dimensional background image, and obtains predetermined The depth data of three-dimensional background image, wherein, depth data obtains in the modeling of predetermined three-dimensional background image or animation process Take.Then, processor 20 determines predetermined three-dimensional background according to the whole audience edge image and depth data of predetermined three-dimensional background image Calculating integration region in image.Because the size of personage's area image is influenceed by the collection distance of visible image capturing first 11, Therefore, the size of personage's area image need to be calculated, and determines to calculate in integration region according to the size of personage's area image Pixel region to be replaced.Finally, the pixel region to be replaced calculated in integration region image is replaced with into personage's area image, from And obtain merging image.In this way, realize merging for personage's area image and predetermined three-dimensional background image.

In some embodiments, personage's area image can be the personage's area image or three-dimensional of two dimension Personage's area image.Wherein, the depth information that processor 20 can be combined in depth image extracts from scene image obtains two dimension Personage's area image, processor 20 can also establish the 3-D view of people's object area according to the depth information in depth image, then Color is carried out with reference to the color information in scene image to people's object area of three-dimensional to fill up to obtain the colored personage area of three-dimensional Area image.

In some embodiments, the predetermined integration region in predetermined three-dimensional background image or calculating integration region can be One or more.When predetermined integration region is one, personage's area image of two-dimentional personage's area image or three-dimensional exists Fusion position in predetermined three-dimensional background image is set to as an above-mentioned unique predetermined integration region；It is when calculating integration region At one, the fusion position of two-dimentional personage's area image or three-dimensional personage's area image in predetermined three-dimensional background image is set to As above-mentioned unique calculating integration region；When predetermined integration region is multiple, two-dimentional personage's area image or three Fusion position of the personage's area image of dimension in predetermined three-dimensional background image can be any one in multiple predetermined integration regions It is individual, further, because personage's area image of three-dimensional has depth information, therefore it can be sought in multiple predetermined integration regions The predetermined integration region that the depth information with three dimensional character area image matches is looked for as position is merged, preferably to be melted Close effect；When it is multiple to calculate integration region, personage's area image of two-dimentional personage's area image or three-dimensional is calculating three It can be any one in multiple calculating integration regions to tie up the fusion position in background image, further, due to three-dimensional Personage's area image has depth information, therefore the depth with three dimensional character area image can be found in multiple calculating integration regions The calculating integration region of information match is spent as fusion position, to obtain more preferable syncretizing effect.

Processor 20 is obtained after merging image, you can is handled to identify certain objects merging image, so as to match Go out predetermined sound model.Most predetermined sound model has acoustic image with merging image co-registration output at last.

Wherein, image can be merged by a frame by acoustic image to form with predetermined sound model, also image can be merged by multiframe Formed with predetermined sound model, the acoustic image that has now is sound video.

Figure 12 is referred to, in some embodiments, the image processing method of embodiment of the present invention also includes：

011：Judge the predetermined sound model for merging the whether relevant storage of predetermined three-dimensional background image in image；

012：When predetermined three-dimensional background image has the predetermined sound model of associated storage, image and predetermined sound will be merged Sound Model Fusion is to have exported acoustic image；When predetermined three-dimensional background image does not have the predetermined sound model of associated storage, enter Enter the certain objects in step 03 identification merging image.

Referring again to Fig. 2, in some embodiments, step 011 and step 012 can be realized by processor 20. I other words processor 20 can also be used to judge the predetermined sound for merging the whether relevant storage of predetermined three-dimensional background image in image Sound model, when predetermined three-dimensional background image has the predetermined sound model of associated storage, image and predetermined sound mould will be merged Type is merged to have exported acoustic image, and when predetermined three-dimensional background image does not have the predetermined sound model of associated storage, is entered Enter the certain objects in step 03 identification merging image.

Specifically, merge predetermined three-dimensional background image in image and, when early stage builds or chooses, appear in each predetermined three Certain objects in dimension background image be it is known, now can directly by the predetermined sound model matched with certain objects with it is pre- Determine three-dimensional background image and be associated storage.In this way, can be with when processor 20 uses a certain width predetermined three-dimensional background image Directly the predetermined sound model of associated storage is merged with merging image has acoustic image to export.Certainly, if making a reservation for three Dimension background image is not associated storage with predetermined sound model, then when processor 20 is using the width predetermined three-dimensional background image It is then more from what is prestored according to the certain objects identified firstly the need of the certain objects in identification predetermined three-dimensional background image The predetermined sound model that selection matches with certain objects in bent predetermined sound model, the predetermined sound model most chosen at last with Merging image is merged has acoustic image to export.

Figure 13 is referred to, in some embodiments, the image processing method of embodiment of the present invention also includes：

05：Broadcasting has acoustic image, and broadcasting has acoustic image to be played out for automatic broadcasting or according to triggering playing request.

Figure 14 is referred to, in some embodiments, image processing apparatus also includes electroacoustic component 70 and display 50.Step Rapid 05 can be realized jointly by electroacoustic component 70 and display 50.Wherein, image is shown that sound is by electroacoustic by display 50 Element 70 plays out.Electroacoustic component 70 can be loudspeaker, earphone, microphone, pickup etc..

Specifically, can be that acquiescence plays image without playing sound when thering is the acoustic image to play out.But active user can Played while carrying out image and sound with selection triggering playing request.When active user does not trigger playing request, only play Image is without playing sound.In addition, it is multiframe when there is the merging image in acoustic image, for example, active user and good friend are in video During chat, the image for the active user that good friend sees is to merge image, and now, active user and good friend can trigger to play and ask Ask and played while image and sound.In this way, the process for the Video chat of user increases interest.In addition, current use Family and good friend trigger playing request again in the image and sound for having acoustic image while in the state of playing, then display 50 continues Display merges image, and electroacoustic component 70 stops playing sound.

During sound image player or default image and sound play simultaneously.Now, active user can select to touch Playing request is sent out so as to stop the broadcasting of sound.

In some embodiments, it is bent or more bent that the predetermined sound model matched with certain objects includes one.

When predetermined sound model is bent including one, during sound image player, predetermined sound model plays once or more It is secondary.In other words, in broadcasting when having acoustic image to include frame merging image and a bent predetermined sound model, display 50 continues The merging image of a frame is shown, electroacoustic component 70 plays a predetermined sound model, or circulation repeatedly plays predetermined sound mould Type.In broadcasting when having acoustic image to include multiframe merging image and a bent predetermined sound model, display 50 is with certain frame per second Show that above-mentioned multiframe merges image, during which electroacoustic component 70 plays a predetermined sound model, or repeatedly broadcasting is pre- for circulation Determine sound model.

When how bent predetermined sound model include, predetermined sound model sequence is stored into list, in the sound image player phase Between, multiple predetermined sound models are in a manner of any one in played in order, shuffle, single circulation, list circulation Play.In other words, in broadcasting when having acoustic image to include frame merging image and how bent predetermined sound model, how bent makes a reservation for Sound model sequential storage can be by how bent predetermined into list, the merging image of the continuously display frame of display 50, electroacoustic component 70 The storage order of the list of sound model, played in order once how bent predetermined sound model, or to circulate the tab sequential multiple More bent predetermined sound models are played, either the predetermined sound model in shuffle playlist or the how bent predetermined sound of selection A first progress loop play in model.There is acoustic image to include multiframe merging image and a bent predetermined sound model in broadcasting When, display 50 shows that above-mentioned multiframe merges image with certain frame per second, and during which electroacoustic component 70 can be by how bent predetermined sound The storage order of the list of model, played in order once how bent predetermined sound model, or circulate the tab sequential and repeatedly play More bent predetermined sound models, either the predetermined sound model in shuffle playlist or choose how bent predetermined sound model In one first carry out loop play.

Also referring to Fig. 3 and Figure 14, embodiment of the present invention also proposes a kind of electronic installation 1000.Electronic installation 1000 Including image processing apparatus 100.Image processing apparatus 100 can utilize hardware and/or software to realize.Image processing apparatus 100 Including imaging device 10 and processor 20.

Imaging device 10 includes visible image capturing first 11 and depth image acquisition component 12.

Specifically, it is seen that light video camera head 11 includes imaging sensor 111 and lens 112, it is seen that light video camera head 11 can be used for The colour information of active user is caught to obtain scene image, wherein, imaging sensor 111 includes color filter lens array (such as Bayer filter arrays), the number of lens 112 can be one or more.Visible image capturing first 11 is obtaining scene image process In, each imaging pixel in imaging sensor 111 senses luminous intensity and wavelength information in photographed scene, generation one Group raw image data；Imaging sensor 111 sends this group of raw image data into processor 20, and processor 20 is to original View data obtains colored scene image after carrying out the computings such as denoising, interpolation.Processor 20 can be in various formats to original Each image pixel in view data is handled one by one, for example, each image pixel can have the locating depth of 8,10,12 or 14 bits Degree, processor 20 can be handled each image pixel by identical or different bit depth.

Depth image acquisition component 12 includes structured light projector 121 and structure light video camera head 122, depth image collection group The depth information that part 12 can be used for catching active user is to obtain depth image.Structured light projector 121 is used to throw structure light Active user is incident upon, wherein, structured light patterns can be the speckle of laser stripe, Gray code, sine streak or random alignment Pattern etc..Structure light video camera head 122 includes imaging sensor 1221 and lens 1222, and the number of lens 1222 can be one or more It is individual.Imaging sensor 1221 is used for the structure light image that capturing structure light projector 121 is projected on active user.Structure light figure As can be sent by depth acquisition component 12 to processor 20 be demodulated, the processing such as phase recovery, phase information calculate to be to obtain The depth information of active user.

In some embodiments, it is seen that the function of light video camera head 11 and structure light video camera head 122 can be by a camera Realize, in other words, imaging device 10 only includes a camera and a structured light projector 121, and above-mentioned camera is not only Structure light image can also be shot with photographed scene image.

Except using structure light obtain depth image in addition to, can also by binocular vision method, based on differential time of flight (Time Of Flight, TOF) even depth obtains the depth image of active user as acquisition methods.

Processor 20 is used for the personage's area image that will be extracted from scene image and depth image and predetermined three-dimensional background Image co-registration obtains merging image, and handles and merge image to determine predetermined sound model, most merges image and predetermined sound at last The output of sound Model Fusion has acoustic image.

In addition, image processing apparatus 100 also includes memory 30.Memory 30 can be embedded in electronic installation 1000, The memory that can be independently of outside electronic installation 1000, and may include direct memory access (DMA) (Direct Memory Access, DMA) feature.The knot that the raw image data or depth image acquisition component 12 of first 11 collection of visible image capturing gather Structure light image related data, which can transmit, to be stored or is cached into memory 30.Predetermined sound model may be alternatively stored in storage In device 30.Processor 20 can read raw image data to be handled to obtain scene image from memory 30, also can be from depositing Structure light image related data is read in reservoir 30 to be handled to obtain depth image, can also be read from memory 30 predetermined Sound model is to merge the further processing of image.In addition, scene image and depth image are also storable in memory 30 In, calling is handled device 20 for processing at any time, for example, processor 20 calls scene image and depth image to carry out people's object area and carry Take, and obtained personage's area image after carrying carries out fusion treatment to obtain merging image with predetermined three-dimensional background image, It is followed by subsequent processing and merges image recognition certain objects, then find the predetermined sound model matched with certain objects, most merges figure at last As having exported acoustic image with predetermined sound Model Fusion.Wherein, predetermined three-dimensional background image, merge image and have acoustic image It is storable in memory 30.

Image processing apparatus 100 may also include display 50.Display 50 can obtain directly from processor 20 sound spectrogram The merging image of picture, the merging image for having acoustic image can be also obtained from memory 30.Display 50 is shown in acoustic image Merge image so that user watches, or by graphics engine or graphics processor (Graphics Processing Unit, GPU) It is further processed.Image processing apparatus 100 also includes encoder/decoder 60, and encoder/decoder 60 can encoding and decoding Scene image, depth image and the view data for merging image etc., the view data of coding can be saved in memory 30, and It can be shown before image is shown on display 50 by decoder decompresses.Encoder/decoder 60 can be in Central processor (Central Processing Unit, CPU), GPU or coprocessor are realized.In other words, encoder/decoder 60 can be in central processing unit (Central Processing Unit, CPU), GPU and coprocessor any one or It is a variety of.

Image processing apparatus 100 also includes control logic device 40.Imaging device 10 imaging when, processor 20 can according into As the data that equipment obtains are analyzed to determine one or more control parameters of imaging device 10 (for example, time for exposure etc.) Image statistics.Processor 20 sends image statistics to control logic device 40, the control imaging of control logic device 40 Equipment 10 is imaged with the control parameter determined.Control logic device 40 may include to perform one or more routines (such as firmware) Processor and/or microcontroller.One or more routines can determine imaging device 10 according to the image statistics of reception Control parameter.

Image processing apparatus 100 also includes electroacoustic component 70, and electroacoustic component 70 is used to play the predetermined sound having in acoustic image Sound model.Electroacoustic component 70 is generally made up of vibrating membrane, voice coil loudspeaker voice coil, permanent magnet, support etc..When the voice coil loudspeaker voice coil of electroacoustic component 70 is passed through After tone currents, alternating magnetic field is produced in the presence of electric current, it is constant that permanent magnet also produces a size and Orientation simultaneously Constant magnetic field.Because the size and Orientation in magnetic field caused by voice coil loudspeaker voice coil is constantly changing with the change of tone currents, so The interaction in two magnetic fields makes voice coil loudspeaker voice coil make the motion of the sense of current in voice coil loudspeaker voice coil, because voice coil loudspeaker voice coil is connected with vibrating membrane, from And drive vibrating membrane to produce vibration, the vibrations of air are caused by vibrating membrane and send sound.Electroacoustic component 70 can be from processor 20 Acquisition has the predetermined sound model in acoustic image to play out, and the predetermined sound having in acoustic image can be also obtained from memory 30 Model plays out.

Figure 15 is referred to, the electronic installation 1000 of embodiment of the present invention includes one or more processors 20, memory 30 and one or more programs 31.Wherein one or more programs 31 are stored in memory 30, and are configured to by one Individual or multiple processors 20 perform.Program 31 includes being used to perform the finger of the image processing method of above-mentioned any one embodiment Order.

For example, program 31 includes being used for the instruction for performing the image processing method described in following steps：

03：Identification merges the certain objects in image；

For another example program 31 also includes being used for the instruction for performing the image processing method described in following steps：

0231：Identify the human face region in scene image；

The computer-readable recording medium of embodiment of the present invention includes being combined with the electronic installation 1000 that can be imaged making Computer program.Computer program can be performed by processor 20 to complete the image procossing of above-mentioned any one embodiment Method.

For example, computer program can be performed by processor 20 to complete the image processing method described in following steps：

03：Identification merges the certain objects in image；

For another example computer program can be also performed by processor 20 to complete the image processing method described in following steps：

0231：Identify the human face region in scene image；

In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show The description of example " or " some examples " etc. means specific features, structure, material or the spy for combining the embodiment or example description Point is contained at least one embodiment or example of the present invention.In this manual, to the schematic representation of above-mentioned term not Identical embodiment or example must be directed to.Moreover, specific features, structure, material or the feature of description can be with office Combined in an appropriate manner in one or more embodiments or example.In addition, in the case of not conflicting, the skill of this area Art personnel can be tied the different embodiments or example and the feature of different embodiments or example described in this specification Close and combine.

In addition, term " first ", " second " are only used for describing purpose, and it is not intended that instruction or hint relative importance Or the implicit quantity for indicating indicated technical characteristic.Thus, define " first ", the feature of " second " can be expressed or Implicitly include at least one this feature.In the description of the invention, " multiple " are meant that at least two, such as two, three It is individual etc., unless otherwise specifically defined.

Any process or method described otherwise above description in flow chart or herein is construed as, and represents to include Module, fragment or the portion of the code of the executable instruction of one or more the step of being used to realize specific logical function or process Point, and the scope of the preferred embodiment of the present invention includes other realization, wherein can not press shown or discuss suitable Sequence, including according to involved function by it is basic simultaneously in the way of or in the opposite order, carry out perform function, this should be of the invention Embodiment person of ordinary skill in the field understood.

Expression or logic and/or step described otherwise above herein in flow charts, for example, being considered use In the order list for the executable instruction for realizing logic function, may be embodied in any computer-readable medium, for Instruction execution system, device or equipment (such as computer based system including the system of processor or other can be held from instruction The system of row system, device or equipment instruction fetch and execute instruction) use, or combine these instruction execution systems, device or set It is standby and use.For the purpose of this specification, " computer-readable medium " can any can be included, store, communicate, propagate or pass Defeated program is for instruction execution system, device or equipment or the dress used with reference to these instruction execution systems, device or equipment Put.The more specifically example (non-exhaustive list) of computer-readable medium includes following：Electricity with one or more wiring Connecting portion (electronic installation), portable computer diskette box (magnetic device), random access memory (RAM), read-only storage (ROM), erasable edit read-only storage (EPROM or flash memory), fiber device, and portable optic disk is read-only deposits Reservoir (CDROM).In addition, computer-readable medium, which can even is that, to print the paper of described program thereon or other are suitable Medium, because can then enter edlin, interpretation or if necessary with it for example by carrying out optical scanner to paper or other media His suitable method is handled electronically to obtain described program, is then stored in computer storage.

It should be appreciated that each several part of the present invention can be realized with hardware, software, firmware or combinations thereof.Above-mentioned In embodiment, software that multiple steps or method can be performed in memory and by suitable instruction execution system with storage Or firmware is realized.If, and in another embodiment, can be with well known in the art for example, realized with hardware Any one of row technology or their combination are realized：With the logic gates for realizing logic function to data-signal Discrete logic, have suitable combinational logic gate circuit application specific integrated circuit, programmable gate array (PGA), scene Programmable gate array (FPGA) etc..

Those skilled in the art are appreciated that to realize all or part of step that above-described embodiment method carries Suddenly it is that by program the hardware of correlation can be instructed to complete, described program can be stored in a kind of computer-readable storage medium In matter, the program upon execution, including one or a combination set of the step of embodiment of the method.

In addition, each functional unit in each embodiment of the present invention can be integrated in a processing module, can also That unit is individually physically present, can also two or more units be integrated in a module.Above-mentioned integrated mould Block can both be realized in the form of hardware, can also be realized in the form of software function module.The integrated module is such as Fruit is realized in the form of software function module and as independent production marketing or in use, can also be stored in a computer In read/write memory medium.

Storage medium mentioned above can be read-only storage, disk or CD etc..Although have been shown and retouch above Embodiments of the invention are stated, it is to be understood that above-described embodiment is exemplary, it is impossible to be interpreted as the limit to the present invention System, one of ordinary skill in the art can be changed to above-described embodiment, change, replace and become within the scope of the invention Type.

Claims

1. a kind of image processing method, merge image for handling, the merging image is by predetermined three-dimensional background image and currently Personage area image fusion of the user in the scene image under real scene forms, it is characterised in that described image processing side Method includes：

Identify the certain objects in the merging image；With

2. image processing method according to claim 1, it is characterised in that described image processing method also includes：

Judge the predetermined sound model of the whether relevant storage of the predetermined three-dimensional background image in the merging image；

When the predetermined three-dimensional background image has the predetermined sound model of the associated storage, by the merging image and institute Predetermined sound Model Fusion is stated to have exported acoustic image；With

When the predetermined three-dimensional background image does not have the predetermined sound model of the associated storage, described in the identification The step of merging the certain objects in image.

3. image processing method according to claim 1, it is characterised in that described image processing method also includes：

Obtain the scene image of the active user；

Obtain the depth image of the active user；

The scene image and the depth image are handled to extract personage area of the active user in the scene image Domain and obtain personage's area image；With

Personage's area image is merged to obtain merging image with the predetermined three-dimensional background image.

4. image processing method according to claim 3, it is characterised in that the depth map for obtaining the active user The step of picture, includes：

To active user's projective structure light；

The structure light image that shooting is modulated through the active user；With

Phase information corresponding to each pixel of the structure light image is demodulated to obtain the depth image.

5. image processing method according to claim 4, it is characterised in that described to demodulate each of the structure light image The step of phase information corresponding to pixel is to obtain the depth image includes：

Demodulate phase information corresponding to each pixel in the structure light image；

The phase information is converted into depth information；With

The depth image is generated according to the depth information.

6. image processing method according to claim 3, it is characterised in that it is described by personage's area image with it is described Predetermined three-dimensional background image is merged to include the step of obtaining and merge image：

Obtain the predetermined integration region in the predetermined three-dimensional background image；

The pixel region to be replaced of the predetermined integration region is determined according to personage's area image；With

The pixel region to be replaced of the predetermined integration region is replaced with into personage's area image to obtain the merging figure Picture.

7. image processing method according to claim 3, it is characterised in that it is described by personage's area image with it is described Predetermined three-dimensional background image is merged to include the step of obtaining and merge image：

The predetermined three-dimensional background image is handled to obtain the whole audience edge image of the predetermined three-dimensional background image；

Obtain the depth data of the predetermined three-dimensional background image；

The predetermined three-dimensional background is determined according to the whole audience edge image of the predetermined three-dimensional background image and the depth data The calculating integration region of image；

The pixel region to be replaced of the calculating integration region is determined according to personage's area image；With

The pixel region to be replaced of the calculating integration region is replaced with into personage's area image to obtain the merging figure Picture.

8. image processing method according to claim 1, it is characterised in that it is described matched with the certain objects it is predetermined It is bent or more bent that sound model includes one；

When the predetermined sound model matched with the certain objects is bent including one, during the sound image player, The predetermined sound model plays one or many；

When how bent the predetermined sound model matched with the certain objects include, the predetermined sound model sequence storage Into list, during the sound image player, multiple predetermined sound models are followed with played in order, shuffle, single Any one mode in ring, list circulation plays.

9. image processing method according to claim 1, it is characterised in that described image processing method also includes：

There is acoustic image described in broadcasting, there is acoustic image to be broadcast for automatic broadcasting or according to triggering playing request described in the broadcasting Put.

10. a kind of image processing apparatus, merge image for handling, the merging image by the predetermined three-dimensional background image with Personage area image fusion of the active user in the scene image under real scene forms, it is characterised in that at described image Reason device includes processor, and the processor is used for：

Identify the certain objects in the merging image；With

11. image processing apparatus according to claim 10, it is characterised in that the processor is additionally operable to：

12. image processing apparatus according to claim 10, it is characterised in that described image processing unit also includes：

Visible image capturing head, the visible image capturing head are used for the scene image for obtaining the active user；

Depth image acquisition component, the depth image acquisition component are used for the depth image for obtaining the active user；

The processor is additionally operable to：

13. image processing apparatus according to claim 12, it is characterised in that the depth image acquisition component includes knot Structure light projector and structure light video camera head, the structured light projector are used for active user's projective structure light；

The structure light video camera head is used for：

14. image processing apparatus according to claim 13, it is characterised in that the structure light video camera head is additionally operable to：

The phase information is converted into depth information；With

The depth image is generated according to the depth information.

15. image processing apparatus according to claim 12, it is characterised in that the processor is additionally operable to：

16. image processing apparatus according to claim 12, it is characterised in that the processor is additionally operable to：

17. image processing apparatus according to claim 10, it is characterised in that it is described matched with the certain objects it is pre- It is bent or more bent including one to determine sound model；

18. image processing apparatus according to claim 10, it is characterised in that described image processing unit also includes：

Electroacoustic component and display, the electroacoustic component and the display, which are used to playing, described has acoustic image, the broadcasting institute Acoustic image has been stated to play out for automatic broadcasting or according to triggering playing request.

19. a kind of electronic installation, it is characterised in that the electronic installation includes：

One or more processors；

Memory；With

One or more programs, wherein one or more of programs are stored in the memory, and be configured to by One or more of computing devices, described program include being used at the image that perform claim is required described in 1 to 9 any one The instruction of reason method.

A kind of 20. computer-readable recording medium, it is characterised in that the meter being used in combination including the electronic installation with that can image Calculation machine program, the computer program can be executed by processor to complete the image procossing described in claim 1 to 9 any one Method.