CN108154465B - Image processing method and device - Google Patents
Image processing method and device Download PDFInfo
- Publication number
- CN108154465B CN108154465B CN201711377564.7A CN201711377564A CN108154465B CN 108154465 B CN108154465 B CN 108154465B CN 201711377564 A CN201711377564 A CN 201711377564A CN 108154465 B CN108154465 B CN 108154465B
- Authority
- CN
- China
- Prior art keywords
- image
- depth
- rgb image
- rgb
- background
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 17
- 238000000034 method Methods 0.000 claims abstract description 20
- 230000011218 segmentation Effects 0.000 claims description 49
- 238000013527 convolutional neural network Methods 0.000 claims description 41
- 238000012545 processing Methods 0.000 claims description 33
- 238000013528 artificial neural network Methods 0.000 claims description 30
- 238000012549 training Methods 0.000 claims description 28
- 238000003709 image segmentation Methods 0.000 claims description 16
- 230000000694 effects Effects 0.000 abstract description 24
- 230000011514 reflex Effects 0.000 abstract description 10
- 238000010586 diagram Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 9
- 230000003287 optical effect Effects 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/174—Segmentation; Edge detection involving the use of two or more images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
Abstract
The present disclosure relates to an image processing method and apparatus. The image processing method comprises the following steps: acquiring an RGB image and a depth image of a shooting target; segmenting the RGB image according to the depth image, and determining a portrait area and a background area in the RGB image; and carrying out background blurring operation on the background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image. According to the method and the device, the portrait area and the background area in the RGB image can be accurately segmented based on the depth information provided by the depth image, so that the background blurring effect and the shooting quality are guaranteed, the portrait background blurring effect of a single-lens reflex camera is simulated to the greatest extent, and the user experience is improved.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method and apparatus.
Background
The single lens reflex camera can shoot portrait photos with background blurring effect and has extremely strong visual impact. The background blurring effect has the following characteristics: 1) the focused foreground portrait needs to be imaged clearly; 2) background scene imaging outside the portrait is blurred; 3) the farther the background scenery is away from the portrait, the larger the fuzzy degree is, otherwise, the smaller the fuzzy degree is, namely the fuzzy degree is different along with different depths of field; 4) out-of-focus imaging is a linear problem.
In the related art, a red-green-blue (RGB) camera is configured in a mobile phone, a portrait segmentation algorithm based on an RGB image is adopted to segment a human body part and a surrounding background part in the RGB image, and after the segmentation is completed, a background blurring operation is performed to obtain a portrait background blurring effect similar to that of a single lens reflex camera.
Disclosure of Invention
To overcome the problems in the related art, embodiments of the present disclosure provide an image processing method and apparatus. The technical scheme is as follows:
according to a first aspect of embodiments of the present disclosure, there is provided an image processing method, including:
acquiring a red, green and blue (RGB) image and a depth image of a shooting target;
segmenting the RGB image according to the depth image, and determining a portrait area and a background area in the RGB image;
and carrying out background blurring operation on the background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image.
In one embodiment, segmenting the RGB image according to the depth image, and determining a portrait region and a background region in the RGB image includes:
using the depth image and a first depth convolution neural network obtained through pre-training to carry out portrait segmentation on the RGB image to obtain a portrait segmentation image; the portrait segmentation image is used for segmenting the RGB image into a portrait area and a background area;
and determining a portrait area and a background area in the RGB image according to the portrait segmentation image.
In one embodiment, the input of the first deep convolutional neural network is the depth image and the RGB image, and the output of the first deep convolutional neural network is the human image segmentation image.
In one embodiment, performing a background blurring operation on a background region in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image includes:
and performing background blurring operation on the background area in the RGB image by using the depth image and a second depth convolution neural network obtained by pre-training to obtain a background blurring image corresponding to the RGB image.
In one embodiment, the input of the second deep convolutional neural network is the depth image and the RGB image, and the output of the second deep convolutional neural network is a background blurring image corresponding to the RGB image.
According to a second aspect of the embodiments of the present disclosure, there is provided an image processing apparatus including:
the acquisition module is used for acquiring a red, green and blue RGB image and a depth image of a shooting target;
the portrait segmentation module is used for segmenting the portrait of the RGB image according to the depth image and determining a portrait area and a background area in the RGB image;
and the background blurring module is used for performing background blurring operation on a background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image.
In one embodiment, the portrait segmentation module includes:
the portrait segmentation submodule is used for segmenting the RGB image by using the depth image and a first depth convolution neural network obtained by pre-training to obtain a portrait segmentation image; the portrait segmentation image is used for segmenting the RGB image into a portrait area and a background area;
and the determining submodule is used for determining a portrait area and a background area in the RGB image according to the portrait segmentation image.
In one embodiment, the input of the first deep convolutional neural network is the depth image and the RGB image, and the output of the first deep convolutional neural network is the human image segmentation image.
In one embodiment, the background blurring module performs a background blurring operation on a background area in the RGB image by using the depth image and a second depth convolutional neural network obtained by pre-training, so as to obtain a background blurring image corresponding to the RGB image.
In one embodiment, the input of the second deep convolutional neural network is the depth image and the RGB image, and the output of the second deep convolutional neural network is a background blurring image corresponding to the RGB image.
According to a third aspect of the embodiments of the present disclosure, there is provided an image processing apparatus including:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
acquiring a red, green and blue (RGB) image and a depth image of a shooting target;
segmenting the RGB image according to the depth image, and determining a portrait area and a background area in the RGB image;
and carrying out background blurring operation on the background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of any of the method embodiments of the first aspect described above.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects: according to the technical scheme, the portrait area and the background area in the RGB image are accurately segmented based on the depth information provided by the depth image, so that the background blurring effect and the shooting quality are guaranteed, the portrait background blurring effect of a single-lens reflex camera can be simulated to the greatest extent, and the user experience is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
FIG. 1 is a flow diagram illustrating an image processing method according to an exemplary embodiment.
FIG. 2 is a flow diagram illustrating an image processing method according to an exemplary embodiment.
FIG. 3 is a flow diagram illustrating an image processing method according to an exemplary embodiment.
Fig. 4 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.
Fig. 5 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.
Fig. 6 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.
Fig. 7 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.
Fig. 8 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
In the related art, an RGB camera is configured in a mobile phone, a portrait segmentation algorithm based on an RGB image is adopted to segment a human body part and a surrounding background part in the RGB image, and after the segmentation is completed, a background blurring operation is performed to obtain a portrait background blurring effect similar to that of a single lens reflex camera. However, under the condition that the fusion degree of the portrait clothes texture and the surrounding background is high, for example, when a photographed object wears camouflage clothes to take a picture in the field, it is difficult to accurately segment the portrait and the surrounding background by using a portrait segmentation algorithm based on RGB images, which often causes errors in boundary segmentation of the portrait and the background, which seriously affects the blurring effect of background blurring operation, reduces the photographing quality, and affects user experience.
In order to solve the above problem, an embodiment of the present disclosure provides an image processing method, including: acquiring an RGB image and a depth image of a shooting target; segmenting the RGB image according to the depth image, and determining a portrait area and a background area in the RGB image; and carrying out background blurring operation on the background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image. According to the technical scheme, the portrait area and the background area in the RGB image can be accurately segmented based on the depth information provided by the depth image, so that the background blurring effect and the shooting quality are guaranteed, the portrait background blurring effect of a single-lens reflex camera is simulated to the greatest extent, and the user experience is improved.
Based on the above analysis, the following specific examples are proposed.
Fig. 1 is a flowchart illustrating an image processing method according to an exemplary embodiment, where an execution subject of the method may be a terminal, such as a smart phone, a tablet computer, a desktop computer, a notebook computer, etc.; as shown in fig. 1, the method comprises the following steps 101-103:
in step 101, an RGB image and a depth image of a photographic subject are acquired.
For example, a terminal equipped with a 3-dimensional (3D) structured light camera may capture not only RGB images of a photographic subject but also a depth image of the photographic subject. The depth information provided by the depth image means that each pixel in the depth image represents the distance between a point in the shooting target and the camera of the terminal. The shooting target comprises a person selected by a camera viewfinder of the terminal and a background around the person.
In step 102, the RGB image is segmented according to the depth image, and a portrait area and a background area in the RGB image are determined.
For example, in the case that the fusion degree of the portrait clothes texture and the surrounding background is high, for example, when a shot object wears camouflage clothes to take a picture in the field, considering that the difference between the depth information of the portrait area and the depth information of the background area in the depth image is relatively large, the present disclosure combines the depth information of the depth image and the color characteristics of the RGB image to accurately segment the portrait area and the background area.
Illustratively, a large number of training samples are collected in advance, each sample including: a depth image, an RGB image and a portrait segmentation image; the portrait segmentation image is an image representing a result of the portrait segmentation, and is used to segment the RGB image into a portrait region and a background region. Training using the training samples results in a first deep convolutional neural network. The input of the first depth convolution neural network is a depth image and an RGB image, and the output of the first depth convolution neural network is a human image segmentation image.
In an example, after an RGB image and a depth image of a shooting target are obtained, a depth image and a first depth convolution neural network obtained through pre-training are used for carrying out portrait segmentation on the RGB image to obtain a portrait segmentation image; the portrait segmentation image is used for segmenting the RGB image into a portrait area and a background area. For example, the depth image and the RGB image are used as input of a first depth convolution neural network, and the output of the first depth convolution neural network is a human image segmentation image, for example, in a human image segmentation image representing the human image segmentation result, a human image region is represented by a pixel value 1, and a background region is represented by a pixel value 0, so that the boundary of the pixel value 1 and the pixel value 0 represents the human image segmentation edge of the human image region and the background region. And segmenting the image according to the portrait, and determining the portrait area and the background area in the RGB image.
In step 103, a background blurring operation is performed on the background area in the RGB image according to the depth image, so as to obtain a background blurring image corresponding to the RGB image.
According to the technical scheme, the portrait area and the background area in the RGB image can be accurately segmented based on the depth information provided by the depth image, so that the background blurring effect and the shooting quality are guaranteed, the portrait background blurring effect of a single-lens reflex camera can be simulated to the greatest extent, and therefore user experience can be improved.
In the related technology, a background blurring algorithm based on an RGB image is adopted, and the defects that the background is uniformly blurred and the background blurring cannot present two-linearity exist. In view of this problem, fig. 2 proposes an image processing method that solves the problem. FIG. 2 is a flow diagram illustrating a method of image processing according to an exemplary embodiment; as shown in fig. 2, on the basis of the embodiment shown in fig. 1, the image processing method according to the present disclosure includes the following steps 201 and 203:
in step 201, an RGB image and a depth image of a photographic subject are acquired.
In step 202, the RGB image is segmented according to the depth image, and a portrait area and a background area in the RGB image are determined.
It should be noted that, for the description of step 201 and step 202, refer to the description of step 101 and step 102 in the embodiment shown in fig. 1, and are not described herein again.
In step 203, a background blurring operation is performed on the background area in the RGB image by using the depth image and a second depth convolutional neural network obtained by pre-training, so as to obtain a background blurring image corresponding to the RGB image.
For example, when the background blurring effect of the single lens reflex is simulated, two problems need to be paid attention to, namely, the background blurring degree is related to the distance of a background object, and the background blurring can generate a second linear phenomenon. In the related technology, a filter with fixed parameters is used for blurring and filtering the background part of the image, and only a blurring effect with uniform distribution can be obtained. The method trains a second depth convolution network to carry out background blurring, wherein the input of the second depth convolution network is a pair of depth images and RGB images of a determined portrait area and a background area, and the output of the second depth convolution neural network is a background blurring image corresponding to the RGB images. The depth image provides depth information, and a blurring effect changing along with the depth can be realized; the deep convolutional neural network has strong nonlinear modeling capability and can realize a linear blurring effect. To train the second deep convolutional neural network, a large number of training samples are collected in advance, each sample including: the depth image, the RGB image of the determined portrait area and the background area, and the background blurring image; and training by using the training samples to obtain a second deep convolutional neural network.
It should be noted that the second deep convolutional neural network related to the present disclosure is very different from the deep convolutional network in the related art: (1) the second deep convolutional neural network related to the present disclosure has two inputs, which are a depth image and an RGB image; whereas the input to the deep convolutional network in the related art has only one RGB image. (2) The output of the deep convolutional network in the correlation technique is the category of the output RGB image, and the loss function is based on the cross entropy; while the output of the second deep convolutional neural network to which the present disclosure relates is a blurred RGB image, the loss function is a loss function based on the mean square error.
In an example, after a portrait segmentation is performed on an RGB image according to a depth image, and a portrait region and a background region in the RGB image are determined, a background blurring operation is performed on the background region in the RGB image by using the depth image and a second depth convolution neural network obtained through pre-training, so as to obtain a background blurring image corresponding to the RGB image. For example, the depth image and the RGB image with the determined portrait area and background area are used as the input of the second depth convolution neural network, and the output of the second depth convolution neural network is the background blurring image corresponding to the RGB image. By using the depth information in the depth image, in the background blurring image corresponding to the RGB image, the farther the background scenery in the background region is from the portrait, the greater the degree of blur, and the background blurring effect varying with the depth of field can be obtained.
According to the technical scheme provided by the embodiment of the disclosure, the depth convolution neural network is trained to perform background blurring operation by combining the depth image and the RGB image, the background blurring effect changing along with the depth of field can be obtained through the depth information in the depth image, the defect of uniform blurring of the background in the related technology is overcome, meanwhile, through the powerful nonlinear modeling capability of the depth convolution network, the second linear blurring effect can be realized, the background blurring effect and the shooting quality are ensured, and the portrait background blurring effect of a single-lens reflex camera is simulated to the maximum extent.
FIG. 3 is a flow diagram illustrating a method of image processing according to an exemplary embodiment; as shown in fig. 3, on the basis of the embodiment shown in fig. 1, the image processing method according to the present disclosure includes the following steps 301-304:
in step 301, an RGB image and a depth image of a photographic subject are acquired.
In step 302, performing portrait segmentation on the RGB image by using the depth image and a first depth convolutional neural network obtained by pre-training to obtain a portrait segmentation image; the portrait segmentation image is used for segmenting the RGB image into a portrait area and a background area.
For example, the input of the first deep convolutional neural network is a depth image and an RGB image, and the output of the first deep convolutional neural network is a human image segmentation image.
In step 303, the image is segmented according to the portrait, and a portrait area and a background area in the RGB image are determined.
In step 304, a background blurring operation is performed on the background area in the RGB image by using the depth image and a second depth convolutional neural network obtained by pre-training, so as to obtain a background blurring image corresponding to the RGB image.
For example, the input of the second deep convolutional neural network is a depth image and an RGB image, and the output of the second deep convolutional neural network is a background blurring image corresponding to the RGB image.
According to the technical scheme provided by the embodiment of the disclosure, the portrait segmentation is carried out on the RGB image by using the first deep convolution neural network obtained through pre-training, and the background blurring operation is carried out on the background area in the RGB image by using the second deep convolution neural network obtained through pre-training, so that the portrait area and the background area in the RGB image can be accurately segmented, meanwhile, the farther the background scenery in the background area is away from the portrait, the larger the blurring degree is, the background blurring effect and shooting quality are ensured, the portrait background blurring effect of a single-lens reflex camera is simulated to the maximum extent, and the user experience is improved.
The following are embodiments of the disclosed apparatus that may be used to perform embodiments of the disclosed methods.
FIG. 4 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment; the apparatus may be implemented in various ways, for example, with all of the components of the apparatus being implemented in a terminal, or with components of the apparatus being implemented in a coupled manner on the terminal side; the apparatus may implement the method related to the present disclosure by software, hardware, or a combination of both, as shown in fig. 4, the image processing apparatus includes: an obtaining module 401, a portrait segmentation module 402, and a background blurring module 403, wherein:
the acquisition module 401 is configured to acquire an RGB image and a depth image of a photographic target;
the portrait segmentation module 402 is configured to perform portrait segmentation on the RGB image according to the depth image, and determine a portrait region and a background region in the RGB image;
the background blurring module 403 is configured to perform a background blurring operation on the background area in the RGB image according to the depth image, so as to obtain a background blurring image corresponding to the RGB image.
The device provided by the embodiment of the disclosure can be used for executing the technical scheme of the embodiment shown in fig. 1, and the execution mode and the beneficial effect are similar, and are not described again here.
In one possible implementation, as shown in fig. 5, the image processing apparatus shown in fig. 4 may further include a portrait segmentation module 402 configured to include: a portrait segmentation sub-module 501 and a determination sub-module 502, wherein:
the portrait segmentation sub-module 501 is configured to perform portrait segmentation on the RGB image by using the depth image and a first depth convolutional neural network obtained through pre-training to obtain a portrait segmentation image; the human image segmentation image is used for segmenting the RGB image into a human image area and a background area;
the determination submodule 502 is configured to segment the image from the portrait, determining a portrait area and a background area in the RGB image.
In one possible embodiment, the input of the first deep convolutional neural network is a depth image and an RGB image, and the output of the first deep convolutional neural network is a human image segmentation image.
In a possible implementation manner, the background blurring module 403 performs a background blurring operation on a background area in the RGB image by using the depth image and a second depth convolutional neural network obtained by pre-training, so as to obtain a background blurring image corresponding to the RGB image.
In one possible embodiment, the input of the second deep convolutional neural network is a depth image and an RGB image, and the output of the second deep convolutional neural network is a background blurring image corresponding to the RGB image.
FIG. 6 is a block diagram illustrating an image processing device that may be implemented in various ways, such as implementing all of the components of the device in a terminal or implementing the components of the device in a coupled manner on the terminal side, according to an example embodiment; referring to fig. 6, the image processing apparatus 600 includes:
a processor 601;
a memory 602 for storing processor-executable instructions;
wherein the processor 601 is configured to:
acquiring an RGB image and a depth image of a shooting target;
segmenting the RGB image according to the depth image, and determining a portrait area and a background area in the RGB image;
and carrying out background blurring operation on the background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image.
In one embodiment, the processor 601 may be further configured to:
using the depth image and a first depth convolution neural network obtained by pre-training to carry out portrait segmentation on the RGB image to obtain a portrait segmentation image; the human image segmentation image is used for segmenting the RGB image into a human image area and a background area;
and segmenting the image according to the portrait, and determining the portrait area and the background area in the RGB image.
In one embodiment, the input of the first deep convolutional neural network is a depth image and an RGB image, and the output of the first deep convolutional neural network is a human image segmentation image.
In one embodiment, the processor 601 may be further configured to:
and performing background blurring operation on the background area in the RGB image by using the depth image and a second depth convolution neural network obtained by pre-training to obtain a background blurring image corresponding to the RGB image.
In one embodiment, the input of the second deep convolutional neural network is a depth image and an RGB image, and the output of the second deep convolutional neural network is a background blurring image corresponding to the RGB image.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Fig. 7 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. For example, the apparatus 700 may be a terminal, such as a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, or an exercise device, among others.
Referring to fig. 7, apparatus 700 may include one or more of the following components: a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 710, an input/output (I/O) interface 712, a sensor component 714, and a communication component 716.
The processing component 702 generally controls overall operation of the device 700, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 702 may include one or more processors 720 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 702 may include one or more modules that facilitate interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate interaction between the multimedia component 708 and the processing component 702.
The memory 704 is configured to store various types of data to support operations at the apparatus 700. Examples of such data include instructions for any application or method operating on device 700, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 704 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
The power supply component 706 provides power to the various components of the device 700. The power components 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the apparatus 700.
The multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the device 700 is in an operation mode, such as a photographing mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component 710 is configured to output and/or input audio signals. For example, audio component 710 includes a Microphone (MIC) configured to receive external audio signals when apparatus 700 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the memory 704 or transmitted via the communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.
The I/O interface 712 provides an interface between the processing component 702 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor assembly 714 includes one or more sensors for providing status assessment of various aspects of the apparatus 700. For example, sensor assembly 714 may detect an open/closed state of device 700, the relative positioning of components, such as a display and keypad of device 700, sensor assembly 714 may also detect a change in position of device 700 or a component of device 700, the presence or absence of user contact with device 700, orientation or acceleration/deceleration of device 700, and a change in temperature of device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 716 is configured to facilitate wired or wireless communication between the apparatus 700 and other devices. The apparatus 700 may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the apparatus 700 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.
In an exemplary embodiment, a non-transitory computer readable storage medium comprising instructions, such as the memory 704 comprising instructions, executable by the processor 720 of the device 700 to perform the above-described method is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
Fig. 8 is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. For example, the apparatus 800 may be provided as a server. The apparatus 800 comprises a processing component 802 that further comprises one or more processors, and memory resources, represented by memory 803, for storing instructions, e.g., applications, that are executable by the processing component 802. The application programs stored in the memory 803 may include one or more modules that each correspond to a set of instructions. Further, the processing component 802 is configured to execute instructions to perform the above-described methods.
The device 800 may also include a power component 806 configured to perform power management of the image processing device 800, a wired or wireless network interface 805 configured to connect the image processing device 800 to a network, and an input output (I/O) interface 808. The apparatus 800 may operate based on an operating system stored in the memory 803, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.
A non-transitory computer readable storage medium in which instructions, when executed by a processor of a device 700 or 800, enable the device 700 or 800 to perform an image processing method comprising:
acquiring an RGB image and a depth image of a shooting target;
segmenting the RGB image according to the depth image, and determining a portrait area and a background area in the RGB image;
and carrying out background blurring operation on the background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image.
In one embodiment, segmenting the RGB image according to the depth image, and determining a portrait region and a background region in the RGB image, includes:
using the depth image and a first depth convolution neural network obtained by pre-training to carry out portrait segmentation on the RGB image to obtain a portrait segmentation image; the human image segmentation image is used for segmenting the RGB image into a human image area and a background area;
and segmenting the image according to the portrait, and determining the portrait area and the background area in the RGB image.
In one embodiment, the input of the first deep convolutional neural network is a depth image and an RGB image, and the output of the first deep convolutional neural network is a human image segmentation image.
In one embodiment, performing a background blurring operation on a background region in an RGB image according to a depth image to obtain a background blurring image corresponding to the RGB image includes:
and performing background blurring operation on the background area in the RGB image by using the depth image and a second depth convolution neural network obtained by pre-training to obtain a background blurring image corresponding to the RGB image.
In one embodiment, the input of the second deep convolutional neural network is a depth image and an RGB image, and the output of the second deep convolutional neural network is a background blurring image corresponding to the RGB image.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (8)
1. An image processing method, comprising:
acquiring a red, green and blue (RGB) image and a depth image of a shooting target;
segmenting the RGB image according to the depth image, and determining a portrait area and a background area in the RGB image;
performing background blurring operation on a background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image;
performing background blurring operation on the background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image, including:
performing background blurring operation on a background area in the RGB image by using the depth image and a second depth convolution neural network obtained by pre-training to obtain a background blurring image corresponding to the RGB image;
the input of the second depth convolution neural network is the depth image and the RGB image, and the output of the second depth convolution neural network is a background blurring image corresponding to the RGB image.
2. The method of claim 1, wherein segmenting the RGB image into the portrait regions according to the depth image, and determining the portrait region and the background region in the RGB image comprises:
using the depth image and a first depth convolution neural network obtained through pre-training to carry out portrait segmentation on the RGB image to obtain a portrait segmentation image; the portrait segmentation image is used for segmenting the RGB image into a portrait area and a background area;
and determining a portrait area and a background area in the RGB image according to the portrait segmentation image.
3. The method of claim 2, wherein the input to the first deep convolutional neural network is the depth image and the RGB image, and wherein the output of the first deep convolutional neural network is the human image segmentation image.
4. An image processing apparatus characterized by comprising:
the acquisition module is used for acquiring a red, green and blue RGB image and a depth image of a shooting target;
the portrait segmentation module is used for segmenting the portrait of the RGB image according to the depth image and determining a portrait area and a background area in the RGB image;
the background blurring module is used for performing background blurring operation on a background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image;
the background blurring module performs background blurring operation on a background area in the RGB image by using the depth image and a second depth convolution neural network obtained by pre-training to obtain a background blurring image corresponding to the RGB image;
the input of the second depth convolution neural network is the depth image and the RGB image, and the output of the second depth convolution neural network is a background blurring image corresponding to the RGB image.
5. The apparatus of claim 4, wherein the portrait segmentation module comprises:
the portrait segmentation submodule is used for segmenting the RGB image by using the depth image and a first depth convolution neural network obtained by pre-training to obtain a portrait segmentation image; the portrait segmentation image is used for segmenting the RGB image into a portrait area and a background area;
and the determining submodule is used for determining a portrait area and a background area in the RGB image according to the portrait segmentation image.
6. The apparatus of claim 5, wherein the input of the first deep convolutional neural network is the depth image and the RGB image, and the output of the first deep convolutional neural network is the human image segmentation image.
7. An image processing apparatus characterized by comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to:
acquiring a red, green and blue (RGB) image and a depth image of a shooting target;
segmenting the RGB image according to the depth image, and determining a portrait area and a background area in the RGB image;
performing background blurring operation on a background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image;
performing background blurring operation on the background area in the RGB image according to the depth image to obtain a background blurring image corresponding to the RGB image, including:
performing background blurring operation on a background area in the RGB image by using the depth image and a second depth convolution neural network obtained by pre-training to obtain a background blurring image corresponding to the RGB image;
the input of the second depth convolution neural network is the depth image and the RGB image, and the output of the second depth convolution neural network is a background blurring image corresponding to the RGB image.
8. A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, carry out the steps of the method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711377564.7A CN108154465B (en) | 2017-12-19 | 2017-12-19 | Image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711377564.7A CN108154465B (en) | 2017-12-19 | 2017-12-19 | Image processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108154465A CN108154465A (en) | 2018-06-12 |
| CN108154465B true CN108154465B (en) | 2022-03-01 |
Family
ID=62463982
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711377564.7A Active CN108154465B (en) | 2017-12-19 | 2017-12-19 | Image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108154465B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109242763B (en) * | 2018-08-27 | 2023-09-01 | Oppo广东移动通信有限公司 | Picture processing method, picture processing device and terminal equipment |
| CN109102460A (en) * | 2018-08-28 | 2018-12-28 | Oppo广东移动通信有限公司 | A kind of image processing method, image processing apparatus and terminal device |
| EP3853772A4 (en) * | 2018-09-21 | 2022-06-22 | Position Imaging, Inc. | Machine-learning-assisted self-improving object-identification system and method |
| CN110956576B (en) * | 2018-09-27 | 2024-03-22 | 北京小米移动软件有限公司 | Image processing method, device, equipment and storage medium |
| CN109961453B (en) | 2018-10-15 | 2021-03-12 | 华为技术有限公司 | Image processing method, device and equipment |
| CN109636814A (en) * | 2018-12-18 | 2019-04-16 | 联想(北京)有限公司 | A kind of image processing method and electronic equipment |
| CN110276767B (en) | 2019-06-28 | 2021-08-31 | Oppo广东移动通信有限公司 | Image processing method and device, electronic equipment and computer readable storage medium |
| CN112614057A (en) * | 2019-09-18 | 2021-04-06 | 华为技术有限公司 | Image blurring processing method and electronic equipment |
| CN111064895B (en) * | 2019-12-31 | 2022-02-01 | 维沃移动通信有限公司 | Virtual shooting method and electronic equipment |
| CN112184586A (en) * | 2020-09-29 | 2021-01-05 | 中科方寸知微(南京)科技有限公司 | Method and system for rapidly blurring monocular visual image background based on depth perception |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9542626B2 (en) * | 2013-09-06 | 2017-01-10 | Toyota Jidosha Kabushiki Kaisha | Augmenting layer-based object detection with deep convolutional neural networks |
| CN106327448A (en) * | 2016-08-31 | 2017-01-11 | 上海交通大学 | Picture stylization processing method based on deep learning |
| CN106683147A (en) * | 2017-01-23 | 2017-05-17 | 浙江大学 | Method of image background blur |
| CN106952222A (en) * | 2017-03-17 | 2017-07-14 | 成都通甲优博科技有限责任公司 | A kind of interactive image weakening method and device |
| CN106993112A (en) * | 2017-03-09 | 2017-07-28 | 广东欧珀移动通信有限公司 | Background-blurring method and device and electronic installation based on the depth of field |
| CN107085825A (en) * | 2017-05-27 | 2017-08-22 | 成都通甲优博科技有限责任公司 | Image weakening method, device and electronic equipment |
| CN107403430A (en) * | 2017-06-15 | 2017-11-28 | 中山大学 | A kind of RGBD image, semantics dividing method |
| CN107426493A (en) * | 2017-05-23 | 2017-12-01 | 深圳市金立通信设备有限公司 | A kind of image pickup method and terminal for blurring background |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104751406B (en) * | 2015-03-11 | 2018-05-01 | 百度在线网络技术（北京）有限公司 | A kind of method and apparatus for being blurred to image |
| KR101745406B1 (en) * | 2015-09-03 | 2017-06-12 | 한국과학기술연구원 | Apparatus and method of hand gesture recognition based on depth image |
| CN107172346B (en) * | 2017-04-28 | 2020-02-07 | 维沃移动通信有限公司 | Virtualization method and mobile terminal |
-
2017
- 2017-12-19 CN CN201711377564.7A patent/CN108154465B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9542626B2 (en) * | 2013-09-06 | 2017-01-10 | Toyota Jidosha Kabushiki Kaisha | Augmenting layer-based object detection with deep convolutional neural networks |
| CN106327448A (en) * | 2016-08-31 | 2017-01-11 | 上海交通大学 | Picture stylization processing method based on deep learning |
| CN106683147A (en) * | 2017-01-23 | 2017-05-17 | 浙江大学 | Method of image background blur |
| CN106993112A (en) * | 2017-03-09 | 2017-07-28 | 广东欧珀移动通信有限公司 | Background-blurring method and device and electronic installation based on the depth of field |
| CN106952222A (en) * | 2017-03-17 | 2017-07-14 | 成都通甲优博科技有限责任公司 | A kind of interactive image weakening method and device |
| CN107426493A (en) * | 2017-05-23 | 2017-12-01 | 深圳市金立通信设备有限公司 | A kind of image pickup method and terminal for blurring background |
| CN107085825A (en) * | 2017-05-27 | 2017-08-22 | 成都通甲优博科技有限责任公司 | Image weakening method, device and electronic equipment |
| CN107403430A (en) * | 2017-06-15 | 2017-11-28 | 中山大学 | A kind of RGBD image, semantics dividing method |
Non-Patent Citations (1)
| Title |
|---|
| 基于深度信息的目标分割及跟踪技术的研究;段炼;《中国优秀硕士学位论文全文数据库 信息科技辑》;20140815(第08期);I138-1203 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108154465A (en) | 2018-06-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108154465B (en) | Image processing method and device | |
| CN108182730B (en) | Virtual and real object synthesis method and device | |
| US9674395B2 (en) | Methods and apparatuses for generating photograph | |
| CN109889724B (en) | Image blurring method and device, electronic equipment and readable storage medium | |
| CN109345485B (en) | Image enhancement method and device, electronic equipment and storage medium | |
| CN106408603B (en) | Shooting method and device | |
| CN106331504B (en) | Shooting method and device | |
| CN108154466B (en) | Image processing method and device | |
| CN110580688B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN109784164B (en) | Foreground identification method and device, electronic equipment and storage medium | |
| CN110569822A (en) | image processing method and device, electronic equipment and storage medium | |
| CN111340731A (en) | Image processing method and device, electronic equipment and storage medium | |
| CN111586296B (en) | Image capturing method, image capturing apparatus, and storage medium | |
| CN112614064B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN105528765A (en) | Method and device for processing image | |
| CN111741187B (en) | Image processing method, device and storage medium | |
| CN113706421A (en) | Image processing method and device, electronic equipment and storage medium | |
| US11222235B2 (en) | Method and apparatus for training image processing model, and storage medium | |
| CN107730443B (en) | Image processing method and device and user equipment | |
| CN106469446B (en) | Depth image segmentation method and segmentation device | |
| CN113315903B (en) | Image acquisition method and device, electronic equipment and storage medium | |
| CN113592733A (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| CN107707819B (en) | Image shooting method, device and storage medium | |
| CN114390189A (en) | Image processing method, device, storage medium and mobile terminal | |
| CN112016595A (en) | Image classification method and device, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |