CN107623823A - Video communication background display methods and device - Google Patents
Video communication background display methods and device Download PDFInfo
- Publication number
- CN107623823A CN107623823A CN201710812050.3A CN201710812050A CN107623823A CN 107623823 A CN107623823 A CN 107623823A CN 201710812050 A CN201710812050 A CN 201710812050A CN 107623823 A CN107623823 A CN 107623823A
- Authority
- CN
- China
- Prior art keywords
- image
- active user
- scene
- depth
- targeted customer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The present invention proposes a kind of video communication background display methods and device, wherein, method includes：Obtain the scene image of active user；Obtain the depth image of active user；Scene image and depth image are handled, personage's area image is obtained to extract people object area of the active user in scene image；Personage's area image is shown to the targeted customer that video communication is carried out with active user with default solid background image co-registration to obtain merging image；Determine the familiarity between active user and targeted customer；Corresponding component element is obtained in scene according to where familiarity from active user, and to targeted customer's display module element in solid background image.Thus, in video communication, according to the familiarity with targeted customer, by the scene information of user, gradually opening is shown to targeted customer, protects the privacy of user, realizes communication security.
Description
Technical field
The present invention relates to technical field of image processing, more particularly to a kind of video communication background display methods and device.
Background technology
With the development of Internet technology, increasing communication function is developed and applied, wherein, video communication work(
Can be due to that can realize, the visualization of strange land user is linked up and is used widely.
However, in correlation technique, when user carries out Video chat, the environmental information for being presented to the other user depends on taking the photograph
As the acquisition range of first-class hardware unit, so that environmental information where user are presented directly into the other user, to working as
The privacy information of preceding user can not be protected effectively.
The content of the invention
The present invention provides a kind of video communication background display methods and device, and to solve in the prior art, user video is chatted
It when, the technical problem of scene information where user can not be shielded.
The embodiment of the present invention provides a kind of video communication background display methods, for electronic installation, including：Obtain current use
The scene image at family；Obtain the depth image of the active user；The scene image and the depth image are handled, with extraction
People object area of the active user in the scene image and obtain personage's area image；By personage's area image with
Default solid background image co-registration is shown to the target use that video communication is carried out with the active user to obtain merging image
Family；Determine the familiarity between the active user and the targeted customer；According to where the familiarity from active user
Corresponding component element is obtained in scene, and the component metadata is shown to the targeted customer in the solid background image
Element.
Another embodiment of the present invention provides a kind of video communication background display device, for electronic installation, including：Visible ray
Camera, for obtaining the scene image of active user；Depth image acquisition component, for obtaining the depth of the active user
Image；Processor, for handling the scene image and the depth image, to extract the active user in the scene graph
As in people's object area and obtain personage's area image；By personage's area image and default solid background image co-registration with
Obtain merging image, be shown to the targeted customer that video communication is carried out with the active user；Determine the active user and institute
State the familiarity between targeted customer；Corresponding component metadata is obtained from the scene where active user according to the familiarity
Element, and show the component element to the targeted customer in the solid background image.
Further embodiment of this invention provides a kind of electronic installation, including：One or more processors；Memory；With one
Or multiple programs, wherein one or more of programs are stored in the memory, and it is configured to by one
Or multiple computing devices, described program include being used to perform the video communication background display methods described in above-described embodiment.
A further embodiment of the present invention provides a kind of computer-readable recording medium, including the electronic installation knot with that can image
The computer program used is closed, the computer program can be executed by processor to complete the video communication described in above-described embodiment
Background display methods.
Technical scheme provided in an embodiment of the present invention can include the following benefits：
The scene image of active user is obtained, obtains the depth image of active user, handles scene image and depth image,
Obtain personage's area image to extract people object area of the active user in scene image, by personage's area image with it is default
Solid background image co-registration is shown to the targeted customer that video communication is carried out with active user, it is determined that working as to obtain merging image
Familiarity between preceding user and targeted customer, and then, obtained in the scene according to where familiarity from active user corresponding
Component element, and to targeted customer's display module element in solid background image.Thus, in video communication, according to and mesh
The familiarity of user is marked, gradually opening is shown to targeted customer by the scene information of user, protects the privacy of user, realizes
Communication security.
The additional aspect of the present invention and advantage will be set forth in part in the description, and will partly become from the following description
Obtain substantially, or recognized by the practice of the present invention.
Brief description of the drawings
Of the invention above-mentioned and/or additional aspect and advantage will become from the following description of the accompanying drawings of embodiments
Substantially and it is readily appreciated that, wherein：
Fig. 1 is the schematic flow sheet of the video communication background display methods of some embodiments of the present invention；
Fig. 2 is the module diagram of the video communication background display device of some embodiments of the present invention；
Fig. 3 is the structural representation of the electronic installation of some embodiments of the present invention；
Fig. 4 is the schematic flow sheet of the video communication background display methods of some embodiments of the present invention；
Fig. 5 is the schematic flow sheet of the video communication background display methods of some embodiments of the present invention；
Fig. 6 (a) to Fig. 6 (e) is the schematic diagram of a scenario of structural light measurement according to an embodiment of the invention；
Fig. 7 (a) and Fig. 7 (b) structural light measurements according to an embodiment of the invention schematic diagram of a scenario；
Fig. 8 is the schematic flow sheet of the video communication background display methods of some embodiments of the present invention；
Fig. 9 is the schematic flow sheet of the video communication background display methods of some embodiments of the present invention；
Figure 10 is the schematic flow sheet of the video communication background display methods of some embodiments of the present invention；
Figure 11 is the schematic flow sheet of the video communication background display methods of some embodiments of the present invention；
Figure 12 is the schematic flow sheet of the video communication background display methods of some embodiments of the present invention；
Figure 13 is the module diagram of the electronic installation of some embodiments of the present invention；And
Figure 14 is the module diagram of the electronic installation of some embodiments of the present invention.
Embodiment
Embodiments of the invention are described below in detail, the example of the embodiment is shown in the drawings, wherein from beginning to end
Same or similar label represents same or similar element or the element with same or like function.Below with reference to attached
The embodiment of figure description is exemplary, it is intended to for explaining the present invention, and is not considered as limiting the invention.
Below with reference to the accompanying drawings the video communication background display methods and device of the embodiment of the present invention are described.
Fig. 1 is the flow chart of video communication background display methods according to an embodiment of the invention, as shown in figure 1, should
Method includes：
Step 101, the scene image of active user is obtained.
Step 102, the depth image of active user is obtained.
Step 103, scene image and depth image are handled, to extract people object area of the active user in scene image
Obtain personage's area image.
Step 104, by personage's area image with default solid background image co-registration to obtain merging image, be shown to
Active user carries out the targeted customer of video communication.Wherein, referring to Fig. 2 and Fig. 3, the video communication background of embodiment of the present invention
Display methods can be realized by the video communication background display device 100 of embodiment of the present invention.Embodiment of the present invention regards
Frequency communication background shows that 100 are used for electronic installation 1000.As shown in figure 3, video communication background display device 100 includes visible ray
Camera 11, depth image acquisition component 12 and processor 20.Step 101 can be by first 11 realization of visible image capturing, step 102
It can be realized by depth image acquisition component 12, step 103 and step 104 are realized by processor 20.
In other words, it is seen that light video camera head 11 can be used for the scene image for obtaining active user；Depth image acquisition component
12 can be used for the depth image of acquisition active user；It is current to extract that processor 20 can be used for processing scene image and depth image
People object area of the user in scene image and obtain personage's area image, and personage's area image and default pure color are carried on the back
Scape image co-registration with obtain merge image.
Wherein, it can be gray level image or coloured image that scene image, which is, and depth image characterizes the field for including active user
Each personal or object depth information in scape.The scene domain of scene image is consistent with the scene domain of depth image, and scene
Each pixel in image can be found in depth image to should pixel depth information.
The video communication background display device 100 of embodiment of the present invention can apply to the electronics of embodiment of the present invention
Device 1000.In other words, the electronic installation 1000 of embodiment of the present invention includes the video communication back of the body of embodiment of the present invention
Scape display device 100.
In some embodiments, electronic installation 1000 includes mobile phone, tablet personal computer, notebook computer, Intelligent bracelet, intelligence
Energy wrist-watch, intelligent helmet, intelligent glasses etc..
The method of existing segmentation personage and background according to similitude of the adjacent pixel in terms of pixel value and does not connect mainly
Continuous property carries out the segmentation of personage and background, but this dividing method is easily influenceed by environmental factors such as ambient light photographs.It is of the invention real
Video communication background display methods video communication background display device 100 and the electronic installation 1000 for applying mode are current by obtaining
The depth image of user is so that personage's extracted region in scene image to be come out.Due to the acquisition of depth image be not easy by illumination,
The influence of the factor such as COLOR COMPOSITION THROUGH DISTRIBUTION in scene, therefore, the people's object area extracted by depth image is more accurate, it is particularly possible to
Accurate calibration goes out the border of people's object area.Further, more accurately personage's area image merges with default solid background
Afterwards merging image it is better.
Fig. 4 is referred to, as a kind of possible implementation, the depth image of active user is obtained in above-mentioned steps 102
The step of include：
Step 201, to active user's projective structure light.
Step 202, the structure light image modulated through active user is shot.
Step 203, phase information corresponding to each pixel of demodulation structure light image is to obtain depth image.
In this example, structured light projector 121 and structure light are included with continued reference to Fig. 3, depth image acquisition component 12
Camera 122.Step 201 can be realized that step 202 and step 203 can be by structure light video camera heads by structured light projector 121
122 realize.
In other words, structured light projector 121 can be used for active user's transmittance structure light；Structure light video camera head 122 can
For shooting the structure light image modulated through active user, and phase information corresponding to each pixel of demodulation structure light image
To obtain depth image.
Specifically, structured light projector 121 is by the face and body of the project structured light of certain pattern to active user
Afterwards, the structure light image after being modulated by active user can be formed in the face of active user and the surface of body.Structure light images
Structure light image after first 122 shooting is modulated, then structure light image is demodulated to obtain depth image.Wherein, structure
The pattern of light can be laser stripe, Gray code, sine streak, non-homogeneous speckle etc..
Referring to Fig. 5, in some embodiments, phase corresponding to each pixel of step 203 demodulation structure light image
The step of information is to obtain depth image includes：
Step 301, phase information corresponding to each pixel in demodulation structure light image.
Step 302, phase information is converted into depth information.
Step 303, depth image is generated according to depth information.
Please continue to refer to Fig. 2, in some embodiments, step 301, step 302 and step 303 can be by structure lights
Camera 122 is realized.
In other words, structure light video camera head 122 can be further used in demodulation structure light image phase corresponding to each pixel
Position information, phase information is converted into depth information, and depth image is generated according to depth information.
Specifically, compared with non-modulated structure light, the phase information of the structure light after modulation is changed, and is being tied
The structure light showed in structure light image is to generate the structure light after distortion, wherein, the phase information of change can characterize
The depth information of object.Therefore, structure light video camera head 122 demodulates phase corresponding to each pixel in structure light image and believed first
Breath, calculates depth information, so as to obtain final depth image further according to phase information.
In order that those skilled in the art is more apparent from gathering the face of active user and body according to structure
The process of the depth image of body, illustrate it by taking a kind of widely used optical grating projection technology (fringe projection technology) as an example below
Concrete principle.Wherein, optical grating projection technology belongs to sensu lato area-structure light.
As shown in Fig. 6 (a), when being projected using area-structure light, sine streak is produced by computer programming first,
And sine streak is projected to measured object by structured light projector 121, recycle structure light video camera head 122 to shoot striped by thing
Degree of crook after body modulation, then demodulates the curved stripes and obtains phase, then phase is converted into depth information to obtain
Depth image.The problem of to avoid producing error or error coupler, needed before carrying out depth information collection using structure light to depth
Image collection assembly 12 carries out parameter calibration, and demarcation includes geometric parameter (for example, structure light video camera head 122 and project structured light
Relative position parameter between device 121 etc.) demarcation, the inner parameter and structured light projector 121 of structure light video camera head 122
The demarcation of inner parameter etc..
Specifically, the first step, computer programming produce sine streak.Need to obtain using the striped of distortion due to follow-up
Phase, for example phase is obtained using four step phase-shifting methods, therefore produce four width phase differences here and beStriped, then structure light throw
Emitter 121 projects the four spokes line timesharing on measured object (mask shown in Fig. 6 (a)), and structure light video camera head 122 collects
Such as the figure on Fig. 6 (b) left sides, while to read the striped of the plane of reference shown on the right of Fig. 6 (b).
Second step, carry out phase recovery.The bar graph that structure light video camera head 122 is modulated according to four width collected is (i.e.
Structure light image) to calculate the phase diagram by phase modulation, now obtained be to block phase diagram.Because four step Phase-shifting algorithms obtain
Result be that gained is calculated by arctan function, therefore the phase after structure light modulation is limited between [- π, π], that is,
Say, the phase after modulation exceedes [- π, π], and it can restart again.Shown in the phase main value such as Fig. 6 (c) finally given.
Wherein, it is necessary to carry out the saltus step processing that disappears, it is continuous phase that will block phase recovery during phase recovery is carried out
Position.As shown in Fig. 6 (d), the left side is the continuous phase bitmap modulated, and the right is to refer to continuous phase bitmap.
3rd step, subtract each other to obtain phase difference (i.e. phase information) by the continuous phase modulated and with reference to continuous phase, should
Phase difference characterizes depth information of the measured object with respect to the plane of reference, then phase difference is substituted into the conversion formula (public affairs of phase and depth
The parameter being related in formula is by demarcation), you can obtain the threedimensional model of the object under test as shown in Fig. 6 (e).
It should be appreciated that in actual applications, according to the difference of concrete application scene, employed in the embodiment of the present invention
Structure light in addition to above-mentioned grating, can also be other arbitrary graphic patterns.
As a kind of possible implementation, the depth information of pattern light progress active user also can be used in the present invention
Collection.
Specifically, the method that pattern light obtains depth information is that this spreads out using a diffraction element for being essentially flat board
The relief diffraction structure that there are element particular phases to be distributed is penetrated, cross section is with two or more concavo-convex step embossment knots
Structure.Substantially 1 micron of the thickness of substrate in diffraction element, each step it is highly non-uniform, the span of height can be 0.7
Micron~0.9 micron.Structure shown in Fig. 7 (a) is the local diffraction structure of the collimation beam splitting element of the present embodiment.Fig. 7 (b) is edge
The unit of the cross sectional side view of section A-A, abscissa and ordinate is micron.The speckle pattern of pattern photogenerated has
The randomness of height, and can with the difference of distance changing patterns.Therefore, depth information is being obtained using pattern light
Before, it is necessary first to the speckle pattern in space is calibrated, for example, in the range of 0~4 meter of distance structure light video camera head 122,
A reference planes are taken every 1 centimetre, then just save 400 width speckle images after demarcating, the spacing of demarcation is smaller, obtains
Depth information precision it is higher.Then, structured light projector 121 is by pattern light projection to measured object (i.e. active user)
On, the speckle pattern that the difference in height on measured object surface to project the pattern light on measured object changes.Structure light
Camera 122 is shot project speckle pattern (i.e. structure light image) on measured object after, then by speckle pattern and demarcation early stage
The 400 width speckle images preserved afterwards carry out computing cross-correlation one by one, and then obtain 400 width correlation chart pictures.Measured object in space
Position where body can show peak value on correlation chart picture, above-mentioned peak value is superimposed and after interpolation arithmetic i.e.
It can obtain the depth information of measured object.
Most diffraction lights are obtained after diffraction is carried out to light beam due to common diffraction element, but per beam diffraction light light intensity difference
Greatly, it is also big to the risk of human eye injury.Re-diffraction even is carried out to diffraction light, the uniformity of obtained light beam is relatively low.
Therefore, the effect projected using the light beam of common diffraction element diffraction to measured object is poor.Using collimation in the present embodiment
Beam splitting element, the element not only have the function that to collimate uncollimated rays, also have the function that light splitting, i.e., through speculum
The non-collimated light of reflection is emitted multi-beam collimation light beam, and the multi-beam collimation being emitted after collimating beam splitting element toward different angles
The area of section approximately equal of light beam, flux of energy approximately equal, and then to carry out using the scatterplot light after the beam diffraction
The effect of projection is more preferable.Meanwhile laser emitting light is dispersed to every light beam, the risk of injury human eye is reduce further, and dissipate
Spot structure light is for other uniform structure lights of arrangement, when reaching same collection effect, the consumption of pattern light
Electricity is lower.
Referring to Fig. 8, as a kind of possible implementation, step 103 handles scene image and depth image to extract
People object area of the active user in scene image and obtain personage's area image, including：
Step 401, the human face region in scene image is identified.
Step 402, depth information corresponding with human face region is obtained from depth image.
Step 403, the depth bounds of people's object area is determined according to the depth information of human face region.
Step 404, the people for being connected and being fallen into depth bounds with human face region is determined according to the depth bounds of people's object area
Object area is to obtain personage's area image.
Referring again to Fig. 2, in some embodiments, step 401, step 402, step 403 and step 404 can be by
Reason device 20 is realized.
In other words, processor 20 can be further used for identifying the human face region in scene image, be obtained from depth image
Depth information corresponding with human face region is taken, the depth bounds of people's object area is determined according to the depth information of human face region, and
Determine to be connected with human face region according to the depth bounds of people's object area and people's object area for falling into depth bounds is to obtain personage
Area image.
Specifically, the human face region that the deep learning Model Identification trained can be used to go out in scene image first, with
The depth information of human face region is can determine that according to the corresponding relation of scene image and depth image afterwards.Because human face region includes
The features such as nose, eyes, ear, lip, therefore, depth number of each feature corresponding in depth image in human face region
According to being different, for example, in face face depth image acquisition component 12, depth that depth image acquisition component 12 is shot
In image, depth data corresponding to nose may be smaller, and depth data corresponding to ear may be larger.Therefore, above-mentioned people
The depth information in face region may be a numerical value or a number range.Wherein, when the depth information of human face region is one
During individual numerical value, the numerical value can be by averaging to obtain to the depth data of human face region；Or can be by human face region
Depth data take in be worth to.
Because people's object area includes human face region, in other words, people's object area is in some depth together with human face region
In the range of, therefore, after processor 20 determines the depth information of human face region, it can be set according to the depth information of human face region
The depth bounds of people's object area, the depth bounds extraction further according to people's object area fall into the depth bounds and with human face region phase
People's object area of connection is to obtain personage's area image.
In this way, personage's area image can be extracted from scene image according to depth information.Due to obtaining for depth information
The image of the not factor such as illumination, colour temperature in by environment is taken to ring, therefore, the personage's area image extracted is more accurate.
Referring to Fig. 9, in some embodiments, video communication background display methods is further comprising the steps of：
Step 501, scene image is handled to obtain the whole audience edge image of scene image.
Step 502, according to whole audience edge image amendment personage's area image.
Referring again to Fig. 2, in some embodiments, step 501 and step 502 can be realized by processor 20.
In other words, processor 20 can also be used to handle scene image to obtain the whole audience edge image of scene image, with
And according to whole audience edge image amendment personage's area image.
Processor 20 carries out edge extracting to obtain whole audience edge image to scene image first, wherein, whole audience edge graph
Edge lines as in include the edge lines of background object in scene residing for active user and active user.Specifically, may be used
Edge extracting is carried out to scene image by Canny operators.The core that Canny operators carry out the algorithm of edge extracting mainly includes
The following steps：First, convolution is carried out to scene image to eliminate noise with 2D gaussian filterings template；Then, differential operator is utilized
The Grad of the gray scale of each pixel, and the gradient direction of the gray scale according to each pixel of Grad calculating are obtained, passes through gradient
Direction can find adjacent pixels of the respective pixel along gradient direction；Then, each pixel is traveled through, if the gray scale of some pixel
Value is not maximum compared with the gray value of former and later two adjacent pixels on its gradient direction, then it is not side to think this pixel
Edge point.In this way, the pixel that marginal position is in scene image is can determine that, so as to obtain the whole audience edge after edge extracting
Image.
After processor 20 obtains whole audience edge image, personage's area image is modified further according to whole audience edge image.
It is appreciated that personage's area image is will to be connected and fall into all pictures of the depth bounds of setting in scene image with human face region
Obtained after element progress merger, in some scenarios, it is understood that there may be some are connected and fallen into depth bounds with human face region
Object.Therefore, to cause personage's area image of extraction more accurate, whole audience edge graph can be used to carry out personage's area image
Amendment.
Further, processor 20 can also carry out second-order correction to revised personage's area image, for example, can be to amendment
Personage's area image afterwards carries out expansion process, expands personage's area image to retain the edge details of personage's area image.
After processor 20 obtains personage's area image, you can carry out personage's area image and default solid background image
Fusion, and then obtain merging image.In some embodiments, the color of default solid background image can be by processor
20 randomly select, or are voluntarily selected by active user.Merging image after fusion can be on the display screen of electronic installation 1000
It has been shown that, can also be printed by the printer being connected with electronic installation 1000.
In one embodiment of the invention, active user carries out wishing to hide the current back of the body during video with other people
Scape, now, you can using the video communication background display methods of embodiment of the present invention by people's object area corresponding to active user
Image and default solid background image co-registration, then show the merging image after fusion to targeted customer.Due to active user just
With other side's video calling, therefore, it is seen that light video camera head 11 needs the scene image of captured in real-time active user, depth image collection group
Part 12 is also required to gather depth image corresponding to active user in real time, and by the scene image to gathering in real time in time of processor 20
Carry out being processed so that other side the video pictures combined by multiframe merging image it can be seen that smooth with depth image.
Step 105, the familiarity between active user and targeted customer is determined.
Step 106, corresponding component element is obtained in the scene according to where familiarity from active user, and is carried on the back in pure color
To targeted customer's display module element in scape image.
It is appreciated that under application scenes, the user of video communication is familiar with, then may have its institute
Other side is showed in the real scene image real scene of environment, now, determines to be familiar between active user and targeted customer
Degree, and then, corresponding component element is obtained in the scene according to where familiarity from active user, and in solid background image
To targeted customer's display module element.
It should be noted that according to the difference of concrete application scene, active user can be determined using various ways
Familiarity between targeted customer：
As a kind of possible implementation, as shown in Figure 10, step 105 includes：
Step 601, according to the video interactive information of default matching Indexs measure active user and targeted customer.
Step 602, if detecting, video interactive information meets default match information, inquire about default match information with
The corresponding relation of familiarity, determine the familiarity between active user and targeted customer.
In this example, it should be appreciated that be familiar between user, its topic talked about or the language used are got over
It is random, or, the information content that it is talked about is bigger, is also familiar between user, so as to be examined according to default matching index
The content keyword of voice messaging and text message is surveyed, wherein, the keyword is demarcated according to lot of experimental data, Ke Yishi
The word of familiarity between user is defined, for example comprising relation address word (" mother ", " partner ", " father ") between user, is compared again
Such as colloquial word " you go extremely ", " quick genius believes you ", and/or, the information content of voice messaging and text message.
And then, it will be understood that the corresponding relation of match information and familiarity is pre-set, for example is familiar with corresponding to " mother "
Degree is higher, and familiarity corresponding to " Prof Lee " is relatively low, and 10 corresponding familiarities of voice messaging amount are relatively low, voice messaging amount 1000
Familiarity corresponding to bar is higher etc., so as to which if detecting, video interactive information meets default match information, inquires about default
The corresponding relation of match information and familiarity, determine the familiarity between active user and targeted customer.
As alternatively possible implementation, as shown in figure 11, step 105 includes：
Step 701, to targeted customer's transmission and checking request corresponding to different familiarities.
Step 702, the request fed back according to targeted customer responds to be verified with default standard information, is tied according to checking
Fruit determines the familiarity between active user and targeted customer.
In this example, user is pre-set for checking request corresponding to different familiarities, and for each checking
Standard information corresponding to request setting, the checking request set such as high familiarity is " there are several mouthfuls of people in our families ", for
The standard information that the request is set may be " five mouthfuls ", and the checking request for the setting of low familiarity is " I is that man is female ", pin
The standard information set to the request may be " woman " etc., so as in Video chat, be sent to targeted customer ripe with difference
Checking request corresponding to degree of knowing.
Now, targeted customer responds according to the checking request feedback request, and the targeted customer being now only familiar with is
Standard information corresponding to checking request corresponding with higher familiarity can be fed back, more strange targeted customer is only capable of feeding back
Go out standard information corresponding to checking request corresponding with relatively low familiarity, thus, the request fed back according to targeted customer responds
Verified with default standard information, the familiarity between active user and targeted customer is determined according to the result.
As another possible implementation, as shown in figure 12, step 105 includes：
Step 801, the user images of targeted customer are obtained, extract the face feature information of user images.Step 802, root
The identity information of default Image Database acquisition targeted customer is inquired about according to face feature information.
Step 803, default identity information and the corresponding relation of familiarity are inquired about, determine active user and targeted customer it
Between familiarity.
In this example, the identity information of other users and the corresponding relation of familiarity are pre-established, such as, establish household
Identity information and higher familiarity corresponding relation, establish the identity information of friend and the corresponding relation of medium familiarity, build
Corresponding relation of vertical stranger and relatively low familiarity etc..
And then the user images of targeted customer are obtained, the face feature information of user images is extracted, the user images can be
The facial sectional drawing extraction of targeted customer in video calling is intercepted, and then, default image is inquired about according to face feature information and believed
The identity information that storehouse obtains targeted customer is ceased, default identity information and the corresponding relation of familiarity is inquired about, determines active user
Familiarity between targeted customer.
Further, corresponding component element is obtained in the scene according to where familiarity from active user, wherein, component
Element include user Item Information in the scene, ambient light etc., and shown in solid background image to targeted customer
Component element, such as, according to number of articles corresponding with familiarity, obtain the group of respective numbers in the scene from active user
Part element, such as familiarity are higher, and the component element being shown in the current scene of targeted customer is more, and for example, according to
Type of items corresponding to familiarity, it (for example is daily necessities, office appliance to obtain respective type in the scene from active user
Deng) component element, such as familiarity is higher, and the type for the component element being shown in the current scene of targeted customer is quicker
Sense.
In summary, the video communication background display methods of the embodiment of the present invention, obtains the scene image of active user, obtains
The depth image of active user is taken, scene image and depth image are handled, to extract personage of the active user in scene image
Region and obtain personage's area image, by personage's area image with default solid background image co-registration to obtain merging image,
The targeted customer that video communication is carried out with active user is shown to, the familiarity between active user and targeted customer is determined, enters
And corresponding component element is obtained in the scene according to where familiarity from active user, and to mesh in solid background image
Mark user's display module element.Thus, in video communication, according to the familiarity with targeted customer, by the scene information of user
Gradually opening is shown to targeted customer, protects the privacy of user, realizes communication security.
Also referring to Fig. 3 and Figure 13, embodiment of the present invention also proposes a kind of electronic installation 1000.Electronic installation 1000
Including video communication background display device 100.Video communication background display device 100 can utilize hardware and/or software to realize.
Video communication background display device 100 includes imaging device 10 and processor 20.
Imaging device 10 includes visible image capturing first 11 and depth image acquisition component 12.
Specifically, it is seen that light video camera head 11 includes imaging sensor 111 and lens 112, it is seen that light video camera head 11 can be used for
The colour information of active user is caught to obtain scene image, wherein, imaging sensor 111 includes color filter lens array (such as
Bayer filter arrays), the number of lens 112 can be one or more.Visible image capturing first 11 is obtaining scene image process
In, each imaging pixel in imaging sensor 111 senses luminous intensity and wavelength information in photographed scene, generation one
Group raw image data；Imaging sensor 111 sends this group of raw image data into processor 20, and processor 20 is to original
View data obtains colored scene image after carrying out the computings such as denoising, interpolation.Processor 20 can be in various formats to original
Each image pixel in view data is handled one by one, for example, each image pixel can have the locating depth of 8,10,12 or 14 bits
Degree, processor 20 can be handled each image pixel by identical or different bit depth.
Depth image acquisition component 12 includes structured light projector 121 and structure light video camera head 122, depth image collection group
The depth information that part 12 can be used for catching active user is to obtain depth image.Structured light projector 121 is used to throw structure light
Active user is incident upon, wherein, structured light patterns can be the speckle of laser stripe, Gray code, sine streak or random alignment
Pattern etc..Structure light video camera head 122 includes imaging sensor 1221 and lens 1222, and the number of lens 1222 can be one or more
It is individual.Imaging sensor 1221 is used for the structure light image that capturing structure light projector 121 is projected on active user.Structure light figure
As can be sent by depth acquisition component 12 to processor 20 be demodulated, the processing such as phase recovery, phase information calculate to be to obtain
The depth information of active user.
In some embodiments, it is seen that the function of light video camera head 11 and structure light video camera head 122 can be by a camera
Realize, in other words, imaging device 10 only includes a camera and a structured light projector 121, and above-mentioned camera is not only
Structure light image can also be shot with photographed scene image.
Except using structure light obtain depth image in addition to, can also by binocular vision method, based on differential time of flight (Time
Of Flight, TOF) even depth obtains the depth image of active user as acquisition methods.
Processor 20 is further used for personage's area image by being extracted from scene image and depth image and preset
Solid background image co-registration, by merge image be shown to active user carry out video communication targeted customer, and then, it is determined that
Familiarity between active user and targeted customer, corresponding component is obtained in the scene according to where familiarity from active user
Element, and to targeted customer's display module element in solid background image.When extracting personage's area image, processor 20 can
, can also be according to depth to combine personage's area image that the depth information in depth image extracts two dimension from scene image
Depth information in image establishes the graphics of people's object area, the personage area in conjunction with the color information in scene image to three-dimensional
Domain carries out color and filled up to obtain colored personage's area image of three-dimensional.Therefore, fusion treatment personage area image and default
Solid background image when can be by two dimension personage's area image merged with default solid background image to obtain
Merge image or merged colored personage's area image of three-dimensional to obtain with default solid background image
Merge image.
In addition, video communication background display device 100 also includes video memory 30.Video memory 30 can be embedded in electricity
In sub-device 1000 or independently of the memory outside electronic installation 1000, and it may include direct memory access (DMA)
(Direct Memory Access, DMA) feature.The raw image data of first 11 collection of visible image capturing or depth image collection
The structure light image related data that component 12 gathers, which can transmit, to be stored or is cached into video memory 30.Processor 20
Raw image data can be read from video memory 30 to be handled to obtain scene image, also can be from video memory 30
Structure light image related data is read to be handled to obtain depth image.In addition, scene image and depth image can also store
In video memory 30, calling is handled device 20 for processing at any time, for example, processor 20 calls scene image and depth image
Personage's extracted region is carried out, and obtained personage's area image after carrying carries out fusion treatment with default solid background image
To obtain merging image.Wherein, default solid background image and merging image may be alternatively stored in video memory 30.
Video communication background display device 100 may also include display 50.Display 50 can be obtained directly from processor 20
Merging image is taken, can also be obtained from video memory 30 and merge image.The display of display 50 merges image for targeted customer
Viewing, or further located by graphics engine or graphics processor (Graphics Processing Unit, GPU)
Reason.Video communication background display device 100 also includes encoder/decoder 60, and encoder/decoder 60 can encoding and decoding scene graph
Picture, depth image and the view data for merging image etc., the view data of coding can be stored in video memory 30, and can
To be shown before image is shown on display 50 by decoder decompresses.Encoder/decoder 60 can be by center
Processor (Central Processing Unit, CPU), GPU or coprocessor are realized.In other words, encoder/decoder 60
Can be in central processing unit (Central Processing Unit, CPU), GPU and coprocessor any one or it is more
Kind.
Video communication background display device 100 also includes control logic device 40.Imaging device 10 is in imaging, processor 20
It can be analyzed according to the data that imaging device obtains to determine one or more control parameters of imaging device 10 (for example, exposing
Between light time etc.) image statistics.Processor 20 sends image statistics to control logic device 40, control logic device 40
Control imaging device 10 is imaged with the control parameter determined.Control logic device 40 may include to perform one or more routines
The processor and/or microcontroller of (such as firmware).One or more routines can determine to be imaged according to the image statistics of reception
The control parameter of equipment 10.
Figure 14 is referred to, the electronic installation 1000 of embodiment of the present invention includes one or more processors 200, memory
300 and one or more programs 310.Wherein one or more programs 310 are stored in memory 300, and are configured to
Performed by one or more processors 200.Program 310 includes being used for the video communication back of the body for performing above-mentioned any one embodiment
The instruction of scape display methods.
For example, program 310 includes being used for the instruction for performing the video communication background display methods described in following steps：
Step 01, the scene image of active user is obtained.
Step 02, the depth image of active user is obtained.
Step 03, scene image and depth image are handled, to extract people object area of the active user in scene image
Obtain personage's area image.
Step 04, by personage's area image with default solid background image co-registration to obtain merging image, be shown to
Active user carries out the targeted customer of video communication.
Step 05, the familiarity between active user and targeted customer is determined.
Step 06, corresponding component element is obtained in the scene according to where familiarity from active user, and is carried on the back in pure color
To targeted customer's display module element in scape image.
For another example program 310 also includes being used for the instruction for performing the video communication background display methods described in following steps：
0331：Phase information corresponding to each pixel in demodulation structure light image；
0332：Phase information is converted into depth information；With
0333：Depth image is generated according to depth information.
The computer-readable recording medium of embodiment of the present invention includes being combined with the electronic installation 1000 that can be imaged making
Computer program.Computer program can be performed by processor 200 to be led to completing the video of above-mentioned any one embodiment
Believe background display methods.
Shown for example, computer program can be performed by processor 200 with completing the video communication background described in following steps
Method：
Step 01, the scene image of active user is obtained.
Step 02, the depth image of active user is obtained.
Step 03, scene image and depth image are handled, to extract people object area of the active user in scene image
Obtain personage's area image.
Step 04, by personage's area image with default solid background image co-registration to obtain merging image, be shown to
Active user carries out the targeted customer of video communication.
Step 05, the familiarity between active user and targeted customer is determined.
Step 06, corresponding component element is obtained in the scene according to where familiarity from active user, and is carried on the back in pure color
To targeted customer's display module element in scape image.
For another example computer program can be also performed by processor 200 to complete the video communication background described in following steps
Display methods：
0331：Phase information corresponding to each pixel in demodulation structure light image；
0332：Phase information is converted into depth information；With
0333：Depth image is generated according to depth information.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or the spy for combining the embodiment or example description
Point is contained at least one embodiment or example of the present invention.In this manual, to the schematic representation of above-mentioned term not
Identical embodiment or example must be directed to.Moreover, specific features, structure, material or the feature of description can be with office
Combined in an appropriate manner in one or more embodiments or example.In addition, in the case of not conflicting, the skill of this area
Art personnel can be tied the different embodiments or example and the feature of different embodiments or example described in this specification
Close and combine.
In addition, term " first ", " second " are only used for describing purpose, and it is not intended that instruction or hint relative importance
Or the implicit quantity for indicating indicated technical characteristic.Thus, define " first ", the feature of " second " can be expressed or
Implicitly include at least one this feature.In the description of the invention, " multiple " are meant that at least two, such as two, three
It is individual etc., unless otherwise specifically defined.
Any process or method described otherwise above description in flow chart or herein is construed as, and represents to include
Module, fragment or the portion of the code of the executable instruction of one or more the step of being used to realize custom logic function or process
Point, and the scope of the preferred embodiment of the present invention includes other realization, wherein can not press shown or discuss suitable
Sequence, including according to involved function by it is basic simultaneously in the way of or in the opposite order, carry out perform function, this should be of the invention
Embodiment person of ordinary skill in the field understood.
Expression or logic and/or step described otherwise above herein in flow charts, for example, being considered use
In the order list for the executable instruction for realizing logic function, may be embodied in any computer-readable medium, for
Instruction execution system, device or equipment (such as computer based system including the system of processor or other can be held from instruction
The system of row system, device or equipment instruction fetch and execute instruction) use, or combine these instruction execution systems, device or set
It is standby and use.For the purpose of this specification, " computer-readable medium " can any can be included, store, communicate, propagate or pass
Defeated program is for instruction execution system, device or equipment or the dress used with reference to these instruction execution systems, device or equipment
Put.The more specifically example (non-exhaustive list) of computer-readable medium includes following：Electricity with one or more wiring
Connecting portion (electronic installation), portable computer diskette box (magnetic device), random access memory (RAM), read-only storage
(ROM), erasable edit read-only storage (EPROM or flash memory), fiber device, and portable optic disk is read-only deposits
Reservoir (CDROM).In addition, computer-readable medium, which can even is that, to print the paper of described program thereon or other are suitable
Medium, because can then enter edlin, interpretation or if necessary with it for example by carrying out optical scanner to paper or other media
His suitable method is handled electronically to obtain described program, is then stored in computer storage.
It should be appreciated that each several part of the present invention can be realized with hardware, software, firmware or combinations thereof.Above-mentioned
In embodiment, software that multiple steps or method can be performed in memory and by suitable instruction execution system with storage
Or firmware is realized.Such as, if realized with hardware with another embodiment, following skill well known in the art can be used
Any one of art or their combination are realized：With the logic gates for realizing logic function to data-signal from
Logic circuit is dissipated, the application specific integrated circuit with suitable combinational logic gate circuit, programmable gate array (PGA), scene can compile
Journey gate array (FPGA) etc..
Those skilled in the art are appreciated that to realize all or part of step that above-described embodiment method carries
Suddenly it is that by program the hardware of correlation can be instructed to complete, described program can be stored in a kind of computer-readable storage medium
In matter, the program upon execution, including one or a combination set of the step of embodiment of the method.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing module, can also
That unit is individually physically present, can also two or more units be integrated in a module.Above-mentioned integrated mould
Block can both be realized in the form of hardware, can also be realized in the form of software function module.The integrated module is such as
Fruit is realized in the form of software function module and as independent production marketing or in use, can also be stored in a computer
In read/write memory medium.
Storage medium mentioned above can be read-only storage, disk or CD etc..Although have been shown and retouch above
Embodiments of the invention are stated, it is to be understood that above-described embodiment is exemplary, it is impossible to be interpreted as the limit to the present invention
System, one of ordinary skill in the art can be changed to above-described embodiment, change, replace and become within the scope of the invention
Type.
Claims (15)
- A kind of 1. video communication background display methods, for electronic installation, it is characterised in that including：Obtain the scene image of active user；Obtain the depth image of the active user；The scene image and the depth image are handled, to extract personage area of the active user in the scene image Domain and obtain personage's area image；By personage's area image with default solid background image co-registration to obtain merging image, be shown to it is described current User carries out the targeted customer of video communication；Determine the familiarity between the active user and the targeted customer；Corresponding component element is obtained from the scene where active user according to the familiarity, and in the solid background figure As in the component element is shown to the targeted customer.
- 2. according to the method for claim 1, it is characterised in that the depth image for obtaining the active user, including：To active user's projective structure light；The structure light image that shooting is modulated through the active user；WithPhase information corresponding to each pixel of the structure light image is demodulated to obtain the depth image.
- 3. according to the method for claim 2, it is characterised in that each pixel of the demodulation structure light image is corresponding Phase information to obtain the depth image, including：Demodulate phase information corresponding to each pixel in the structure light image；The phase information is converted into depth information；WithThe depth image is generated according to the depth information.
- 4. according to the method for claim 1, it is characterised in that it is described processing the scene image and the depth image with Extract people object area of the active user in the scene image and obtain personage's area image, including：Identify the human face region in the scene image；Depth information corresponding with the human face region is obtained from the depth image；The depth bounds of people's object area is determined according to the depth information of the human face region；WithThe people for determining to be connected and fall into the depth bounds with the human face region according to the depth bounds of people's object area Object area is to obtain personage's area image.
- 5. according to the method for claim 4, it is characterised in that also include：The scene image is handled to obtain the whole audience edge image of the scene image；WithAccording to personage's area image described in the whole audience edge image amendment.
- 6. according to the method for claim 1, it is characterised in that it is described determine the active user and the targeted customer it Between familiarity, including：According to the video interactive information of active user and the targeted customer described in default matching Indexs measure；If detecting, the video interactive information meets default match information, inquires about the default match information with being familiar with The corresponding relation of degree, determine the familiarity between the active user and the targeted customer.
- 7. according to the method for claim 6, it is characterised in that described currently to be used according to default matching Indexs measure Family and the video interactive information of the targeted customer, including：According to default matching Indexs measure voice messaging and the content keyword of text message, and/or, voice messaging and text The information content of information.
- 8. according to the method for claim 1, it is characterised in that it is described determine the active user and the targeted customer it Between familiarity, including：To targeted customer transmission and checking request corresponding to different familiarities；The request fed back according to the targeted customer responds to be verified with default standard information, and institute is determined according to the result State the familiarity between active user and the targeted customer.
- 9. according to the method for claim 1, it is characterised in that it is described determine the active user and the targeted customer it Between familiarity, including：The user images of the targeted customer are obtained, extract the face feature information of the user images；The identity information of the default Image Database acquisition targeted customer is inquired about according to the face feature information；Inquire about the default identity information and the corresponding relation of familiarity, determine the active user and the targeted customer it Between familiarity.
- 10. according to the method for claim 1, it is characterised in that described according to where the familiarity from active user Corresponding component element is obtained in scene, including：According to number of articles corresponding with the familiarity, the component metadata of respective numbers is obtained in the scene from active user Element；And/orAccording to type of items corresponding with the familiarity, the component metadata of respective type is obtained in the scene from active user Element.
- A kind of 11. video communication background display device, it is characterised in that for electronic installation, including：Visible image capturing head, for obtaining the scene image of active user；Depth image acquisition component, for obtaining the depth image of the active user；Processor, for handling the scene image and the depth image, to extract the active user in the scene graph As in people's object area and obtain personage's area image；By personage's area image with default solid background image co-registration to obtain merging image, be shown to it is described current User carries out the targeted customer of video communication；Determine the familiarity between the active user and the targeted customer；Corresponding component element is obtained from the scene where active user according to the familiarity, and in the solid background figure As in the component element is shown to the targeted customer.
- 12. device as claimed in claim 11, it is characterised in that the depth image acquisition component includes structured light projector With structure light video camera head, the structured light projector is used for active user's projective structure light；The structure light video camera head is used for：The structure light image that shooting is modulated through the active user；WithPhase information corresponding to each pixel of the structure light image is demodulated to obtain the depth image.
- 13. device as claimed in claim 11, it is characterised in that characterized in that, the processor is additionally operable to：The scene image is handled to obtain the whole audience edge image of the scene image；WithAccording to personage's area image described in the whole audience edge image amendment.
- 14. a kind of electronic installation, it is characterised in that the electronic installation includes：One or more processors；Memory；WithOne or more programs, wherein one or more of programs are stored in the memory, and be configured to by One or more of computing devices, described program include being used for any described video communication back ofs the body of perform claim requirement 1-10 Scape display methods.
- A kind of 15. computer-readable recording medium, it is characterised in that the meter being used in combination including the electronic installation with that can image Calculation machine program, the computer program can be executed by processor to complete any described video communication backgrounds of claim 1-10 Display methods.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710812050.3A CN107623823B (en) | 2017-09-11 | 2017-09-11 | Video communication background display method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710812050.3A CN107623823B (en) | 2017-09-11 | 2017-09-11 | Video communication background display method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107623823A true CN107623823A (en) | 2018-01-23 |
| CN107623823B CN107623823B (en) | 2020-12-18 |
Family
ID=61088501
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710812050.3A Active CN107623823B (en) | 2017-09-11 | 2017-09-11 | Video communication background display method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107623823B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1411277A (en) * | 2001-09-26 | 2003-04-16 | Lg电子株式会社 | Video-frequency communication system |
| CN102663810A (en) * | 2012-03-09 | 2012-09-12 | 北京航空航天大学 | Full-automatic modeling approach of three dimensional faces based on phase deviation scanning |
| US20160006772A1 (en) * | 2014-07-07 | 2016-01-07 | Nintendo Co., Ltd. | Information-processing device, communication system, storage medium, and communication method |
| CN105893562A (en) * | 2016-03-31 | 2016-08-24 | 北京小米移动软件有限公司 | Conversation message processing method and device and terminal |
| CN106878588A (en) * | 2017-02-27 | 2017-06-20 | 努比亚技术有限公司 | A kind of video background blurs terminal and method |
| CN106909911A (en) * | 2017-03-09 | 2017-06-30 | 广东欧珀移动通信有限公司 | Image processing method, image processing apparatus and electronic installation |
| CN106954034A (en) * | 2017-03-28 | 2017-07-14 | 宇龙计算机通信科技(深圳)有限公司 | A kind of image processing method and device |
-
2017
- 2017-09-11 CN CN201710812050.3A patent/CN107623823B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1411277A (en) * | 2001-09-26 | 2003-04-16 | Lg电子株式会社 | Video-frequency communication system |
| CN102663810A (en) * | 2012-03-09 | 2012-09-12 | 北京航空航天大学 | Full-automatic modeling approach of three dimensional faces based on phase deviation scanning |
| US20160006772A1 (en) * | 2014-07-07 | 2016-01-07 | Nintendo Co., Ltd. | Information-processing device, communication system, storage medium, and communication method |
| CN105893562A (en) * | 2016-03-31 | 2016-08-24 | 北京小米移动软件有限公司 | Conversation message processing method and device and terminal |
| CN106878588A (en) * | 2017-02-27 | 2017-06-20 | 努比亚技术有限公司 | A kind of video background blurs terminal and method |
| CN106909911A (en) * | 2017-03-09 | 2017-06-30 | 广东欧珀移动通信有限公司 | Image processing method, image processing apparatus and electronic installation |
| CN106954034A (en) * | 2017-03-28 | 2017-07-14 | 宇龙计算机通信科技(深圳)有限公司 | A kind of image processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107623823B (en) | 2020-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107610077A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107707839A (en) | Image processing method and device | |
| CN107734267A (en) | Image processing method and device | |
| CN107509045A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107707831A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107707838A (en) | Image processing method and device | |
| CN107707835A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107610080A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107610078A (en) | Image processing method and device | |
| CN107734264A (en) | Image processing method and device | |
| CN107613239A (en) | Video communication background display methods and device | |
| CN107592491A (en) | Video communication background display methods and device | |
| CN107509043A (en) | Image processing method and device | |
| CN107705278A (en) | The adding method and terminal device of dynamic effect | |
| CN107644440A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107527335A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107705277A (en) | Image processing method and device | |
| CN107613228A (en) | The adding method and terminal device of virtual dress ornament | |
| CN107610076A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107613223A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107622496A (en) | Image processing method and device | |
| CN107454336A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107705243A (en) | Image processing method and device, electronic installation and computer-readable recording medium | |
| CN107682656A (en) | Background image processing method, electronic equipment and computer-readable recording medium | |
| CN107707833A (en) | Image processing method and device, electronic installation and computer-readable recording medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18 Applicant after: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS Corp.,Ltd. Address before: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18 Applicant before: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS Corp.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |