WO2018068420A1

WO2018068420A1 - Image processing method and apparatus

Info

Publication number: WO2018068420A1
Application number: PCT/CN2016/113270
Authority: WO
Inventors: 杨铭
Original assignee: 广州视源电子科技股份有限公司
Priority date: 2016-10-12
Filing date: 2016-12-29
Publication date: 2018-04-19
Also published as: CN106447677A

Abstract

Disclosed are an image processing method and apparatus, relating to the technical field of image processing. The image processing method comprises: acquiring a depth image and an RGB image under the same scene; according to a depth value in the depth image, segmenting the RGB image to obtain at least two area images, and processing at least one of the area images; and fusing the processed area image and the RGB image. In the technical solution, an RGB image is segmented by means of a depth value of a depth image under the same scene. Compared with the solution of separately using an RGB image to detect a target area, higher precision is achieved. Moreover, by processing at least one area image among at least two area images obtained through the segmentation of an RGB image, an area to be processed can be selected according to a self-requirement, different processing can be respectively performed on different areas, and a personalized requirement of a user can be satisfied. A processed area image and an original RGB image are fused, and at the same time, both the processing efficiency and a fusion result are taken into consideration.

Description

Image processing method and device

Technical field

Embodiments of the present invention relate to the field of image processing technologies, and in particular, to an image processing method and apparatus.

Background technique

With the popularity of mobile terminals, more and more users are accustomed to using the camera function of mobile terminals to record life and retain memories. In order to make the picture display according to the user's preference, a large number of beautification image software or a terminal configured to beautify the image function has been appeared.

In many image beautification scenarios, the focus of beautification is often the foreground part of the image. For example, when a face is beautified, the main beautification object is the face in the foreground. At present, there are two main types of existing Mito technology solutions: First, the entire image is processed. The problem with this type of method is that it does not distinguish between regions and does not distinguish the key points from the whole map. It not only wastes a lot of unnecessary calculation time, but also has low efficiency, and can not satisfy the user's partial beautification, highlight the image focus, and pass different beautifications. The effect enhances the individual needs of the image hierarchy and the like. Secondly, the target area detection method such as face detection and skin color detection is used to beautify based on the detection result of the target area. Although the method reduces the calculation time, the result of image processing often depends on the accuracy of target area detection, and the target area detection methods such as face detection and skin color detection have limited target area and omission result due to limited robustness. Not quite satisfactory.

Summary of the invention

The invention provides an image processing method and device, which can solve the problem that image processing cannot be divided into regions or objects. High-precision image segmentation and efficient image processing are achieved by inaccurate detection of standard areas.

In a first aspect, an embodiment of the present invention provides an image processing method, where the method includes:

Obtaining depth images and RGB images in the same scene;

Dividing the RGB image according to a depth value in the depth image to obtain at least two area images, and processing at least one of the area images;

The processed area image is fused with the RGB image.

In a second aspect, an embodiment of the present invention further provides an image processing apparatus, where the apparatus includes:

An original image acquisition module, configured to acquire a depth image and an RGB image in the same scene;

An area image processing module, configured to divide the RGB image according to a depth value in the depth image to obtain at least two area images, and process at least one of the area images;

An image fusion module is configured to fuse the processed area image with the RGB image.

According to the technical solution of the present invention, the RGB image is segmented by the depth value of the depth image in the same scene, and the precision is higher than that of the target region detected by using the RGB image alone. Moreover, it is possible to process at least one of the at least one of the at least two area images by dividing the RGB image, that is, to select the area to be processed according to the needs of the user, and to perform the same or different processing on different areas to satisfy User's personalized needs. The fusion of the processed area image and the original RGB image can effectively reduce the distortion after image processing, preserve the integrity of the image information, and at the same time take into consideration the processing efficiency and the fusion effect. In the above technical solution, the method for processing the entire image is relatively indistinguishable, and by dividing the RGB image into at least two area images and processing only the area image selected by the user, the unnecessary calculation time can be greatly reduced. Real-time calculation; relative to the method based on specific target area detection, directly using depth information to segment RGB images is more accurate and robust.

DRAWINGS

1 is a schematic flowchart of an image processing method according to Embodiment 1 of the present invention;

2A is a schematic flowchart of an image processing method according to Embodiment 2 of the present invention;

2B is a schematic diagram of splitting an RGB image into two area images according to Embodiment 2 of the present invention;

FIG. 3 is a schematic structural diagram of an image processing apparatus according to Embodiment 3 of the present invention.

detailed description

The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. It should also be noted that, for ease of description, only some, but not all, of the structures related to the present invention are shown in the drawings.

Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as a process or method depicted as a flowchart. Although the flowcharts describe the various steps as a sequential process, many of the steps can be implemented in parallel, concurrently, or concurrently. In addition, the order of the steps can be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the figures. The processing may correspond to methods, functions, procedures, subroutines, subroutines, and the like.

Embodiment 1

FIG. 1 is a flowchart of an image processing method according to Embodiment 1 of the present invention. The method may be implemented by an image processing apparatus, and the apparatus may be implemented by hardware and/or software, and is generally independent. The configuration of the embodiment is implemented in the terminal.

The method of this embodiment specifically includes:

S110. Obtain a depth image and an RGB image in the same scene.

Among them, the RGB image can be understood as a color image, and the color of the RGB image is changed by three reference color channels of red (Red, R), green (Green), blue (Blue, B) and their mutual Superimposed to get a variety of colors. The depth image can represent the color resolution of the image by the depth of the image, ie the number of bits used to store each pixel. The size of the pixel values in the depth image reflects the depth of the depth of field. In this embodiment, by acquiring the depth image and the RGB image in the same scene, more and more scene information can be acquired, so that more accurate image processing is performed by the image data.

Generally, the acquired image is related to factors such as shooting distance, shooting angle, shooting environment, and shooting time. Therefore, the depth image and the RGB image in the same scene may be obtained by acquiring the depth image and the RGB image of the same scene at the same time. Specifically, the depth image may be acquired by a depth camera, and the RGB image may be acquired by an ordinary camera; of course, the depth image and the RGB image may also be acquired by the same camera.

S120. Divide the RGB image according to the depth value in the depth image to obtain at least two area images, and process at least one of the area images.

Specifically, dividing the original RGB image according to the depth value in the depth image to obtain the at least two area images may be: determining a second pixel value of the original RGB image according to the first pixel value of the depth image, and further dividing the original RGB according to the second pixel value. The image gets at least two area images. Further, the segmented region image may also be labeled. For example, each area image is labeled based on the pixel value of each pixel of the area image. Alternatively, the pixel values of the pixels of the same image area may be labeled with the same value.

In the technical solution, processing at least one area image may be processing one, two or more area images. The user can select the area to be processed according to the actual segmentation situation and his own individual needs. Of course, it is also possible to process the corresponding area image according to the default setting.

For example, the original RGB image may be segmented according to the depth value in the depth image to obtain two regional images, wherein the pixel values of each pixel of the regional image may be represented by 1 or 0 respectively; wherein 1 represents the foreground region; Background area. The user can process the foreground and/or background regions based on pixel values of 1 or 0, as well as their own personalization needs. For example, a foreground area with a pixel value of 1 can be set by default.

S130: merging the processed area image with the RGB image.

Since the processed area image may be for each area image of the full image or for the foreground area, in order to simultaneously calculate the calculation efficiency and the fusion effect, the processed area image and the unprocessed area image may be selected. Fusion. This technical solution is particularly suitable for the case of processing only individual region images after RGB image segmentation.

Image fusion can include pixel level fusion, feature level fusion, and decision level fusion. Among them, there are spatial domain algorithm and transform domain algorithm in pixel level fusion. There are multiple fusion rule methods in spatial domain algorithm, such as logic filtering method, gray weighted average method, contrast modulation method, etc.; pyramid domain decomposition fusion in transform domain algorithm Method, wavelet transform method, etc. In this embodiment, the merging of the processed area image and the RGB image may be performed by combining pixel values of the processed area image with pixel values of the RGB image to preserve the original as much as possible. data.

According to the technical solution of the present invention, the RGB image is segmented by the depth value of the depth image in the same scene, and the precision is higher than that of the target region detected by using the RGB image alone. Moreover, it is possible to perform the division of the RGB image to obtain at least one of the at least two area images. The reason is that the area to be processed can be selected according to the needs of the user, and the same or different processing can be performed on different areas to meet the personalized needs of the user. The fusion of the processed area image and the original RGB image can effectively reduce the distortion after image processing, preserve the integrity of the image information, and at the same time take into consideration the processing efficiency and the fusion effect. In the above technical solution, the method for processing the entire image is relatively indistinguishable, and by dividing the RGB image into at least two area images and processing only the area image selected by the user, the unnecessary calculation time can be greatly reduced. Real-time calculation; relative to the method based on specific target area detection, directly using depth information to segment RGB images is more accurate and robust.

Embodiment 2

2A is a flowchart of an image processing method according to Embodiment 2 of the present invention. As shown in FIG. 2A, in this embodiment, based on the foregoing embodiment, optionally, the RGB image is divided according to the depth value in the depth image to obtain at least two area images, and processing at least one of the area images includes: Determining, according to the first depth value of each pixel point of the depth image, a second depth value corresponding to each pixel point of the RGB image; and dividing the RGB image according to the second depth value and a preset rule to obtain at least two The area image; processing at least one of the area images corresponding to the preset processing model.

On the basis of the second depth value corresponding to each pixel of the RGB image, the method further includes: calculating the depth image before determining the second depth value corresponding to each pixel of the RGB image according to the first depth value of each pixel of the depth image. a mapping matrix of a first coordinate of each pixel and a second coordinate of each pixel of the RGB image; determining according to the second coordinate, the mapping matrix, and a first depth value of each pixel of the depth image a second depth value corresponding to each pixel point of the RGB image.

In addition, the merging the processed area image with the RGB image may further include: performing Gaussian smoothing on the processed area image to obtain a smooth image of each pixel of the area image. a prime value; calculating a fused pixel value of the fused image according to the smoothed pixel value and a pixel value of each pixel of the RGB image; and outputting the fused image according to the fused pixel value.

Specifically, the method in this embodiment may include:

S201. Acquire a depth image and an RGB image in the same scene.

S202. Calculate a mapping matrix of a first coordinate of each pixel of the depth image and a second coordinate of each pixel of the RGB image.

In this embodiment, the mapping matrix of the first coordinate and the second coordinate may be calculated by using a method of the prior art, for example, according to parameters of a camera that acquires a depth image and an RGB image, or may acquire a depth image and an RGB image. The part in which the features overlap, the correspondence between the depth image and the features of the RGB image is obtained based on the feature overlapping portion, and the mapping matrix is calculated according to the correspondence between the features.

S203. Determine a second depth value corresponding to each pixel point of the RGB image according to the second coordinate, the mapping matrix, and a first depth value of each pixel of the depth image.

Specifically, the pixel values of the depth image corresponding to each pixel point of the RGB image may be determined according to the second coordinate and the mapping matrix; and the first depth value of each pixel point of the acquired depth image is used as the pixel point of the depth image. a second depth value corresponding to each pixel of the corresponding RGB image. For example, determining the pixel values of the depth image corresponding to each pixel point of the RGB image according to the first coordinate and the mapping matrix of each pixel of the RGB image may include: multiplying the coordinates of each pixel point of the RGB image by the mapping matrix. The coordinates of each pixel point of the corresponding depth image. The depth image is aligned with the RGB image, and the depth value of each pixel of the RGB image is obtained by the depth value of each pixel of the depth image.

S204. Segment the RGB image according to the second depth value and a preset rule to obtain at least two area images.

The preset rule may be: setting a depth threshold, and dividing the RGB image to obtain at least two area images according to the comparison result of the second depth value and the depth threshold. Specifically, according to the second depth value of each pixel point in the RGB image, and the preset depth threshold (or depth range), each pixel point in the RGB image is traversed, and the second depth value and the depth threshold of each pixel point are Or the depth range is compared, and the RGB image is divided into two area images according to the comparison result. The depth threshold can be a specific point value or a range. The specific numerical value or numerical interval of the depth threshold may be set by the user according to the actual image of the acquired RGB image, which is not limited herein.

In this embodiment, the segmentation of the original RGB image according to the second depth value and the preset rule to obtain the at least two region images may specifically determine whether the second depth value is greater than a preset depth threshold, and if so, then The position of the pixel corresponding to the depth value is used as the foreground area. If not, the position of the pixel corresponding to the second depth value is used as the background area; and the original RGB image is divided according to the foreground area and the background area to obtain two area images. Alternatively, the position of the pixel corresponding to the second depth value within the depth threshold range may be acquired, and as the foreground region, the set of positions of the remaining pixels is used as the background region.

For example, as shown in FIG. 2B, if the avatar part is used as the foreground area and the other part is used as the background area, the depth threshold range may be set according to actual conditions, and then judged. Whether the second depth value of each pixel in the RGB image is within a preset depth threshold range, and if so, the pixel point is used as the foreground area, that is, the pixel part of the avatar part, if not within the preset depth threshold range, Then the pixel is used as the pixel of the background area. The pixel values of each pixel in the foreground area may be marked as 1 (black area in FIG. 2B); the set of the remaining pixels is used as the background area, and the pixel values of each pixel in the background area may be marked as 0 (FIG. 2B) Medium grid area). Further, the foreground area and/or the background area may be respectively determined according to the pixel values of the respective pixel points. Process it. It should be noted that the grid in FIG. 2B is only used to indicate that the pixel values of the background area are all marked as 0, and the grid area does not exist.

In the actual operation process, the obtained image of the area to be processed may be affected by the noise of the depth image to present some loop-like areas, and these areas may be filled by the expansion etching operation to obtain the final area image.

It can be understood that the at least two area images include two, three and more area images. The advantage of such setting can realize the partition processing or partial processing of the image, which can improve the efficiency of image processing and enrich the effect of image processing.

S205. Process at least one of the area images corresponding to the preset processing model.

The preset processing model may include a model for enhancing, restoring, segmenting, extracting features, removing noise, and the like, or two or more processing methods and techniques. In this embodiment, the preset processing model may include a beautification processing algorithm. For example, in a real-time smear application scenario, a bilateral filtering beautification processing algorithm may be used to implement a real-time dermabrasion image effect.

It should be noted that processing at least one area image corresponding to the preset processing model includes processing one, two or more area images. Further, the processing models of the image of each region may be the same or different. Specifically, at least one area image may be acquired according to a selected instruction input by the user; and the selected at least one area image is processed based on the preset processing model.

S206. Perform Gaussian smoothing on the processed area image to obtain smooth pixel values of each pixel of the area image.

Specifically, in the RGB image segmentation, the size of the divided RGB image may be consistent with the size of the original RGB image, and the pixel points of the segmented RGB image are in one-to-one correspondence with the pixel points of the original RGB image, and further, after the segmentation The pixel value of each pixel in the RGB image can be only 1 or 0. Indicates whether the position of the RGB image corresponding to the pixel is the area image to be processed, respectively. As above, if divided into two area images, 1 can be used to indicate the foreground area; 0 is the background area. On the basis of this, Gaussian smoothing is performed on the segmented image, and the pixel value a(x, y) of the segmented image thus obtained may take an arbitrary value between 0-1. This operation can effectively remove noise in the image and ensure image quality.

S207. Calculate a fused pixel value of the fused image according to the smoothed pixel value and a pixel value of each pixel of the RGB image.

For example, the smoothed pixel value and the pixel value of each pixel of the RGB image may be weighted and summed to calculate a fused pixel value of the fused image. For example, suppose that the smoothed pixel value of the processed area image is I _B (x, y), and the pixel value of each pixel of the original RGB image is I ₀ (x, y). In this case, I _B (x) , y) is weighted with I ₀ (x, y), summed to obtain the fused pixel value I _R (x, y) of the final fused image, ie I _R (x, y) = I _B (x, y ) *a(x,y)+I ₀ (x,y)*{1-a(x,y)}. Where a(x, y) represents the weight of the smoothed pixel value. It can be understood that the specific value of a(x, y) can be set according to the actual situation, and can be a fixed value or calculated by other parameters, which is not limited herein.

S208. Output a fused image according to the fused pixel value.

The technical solution of the embodiment determines the second depth value of each pixel of the RGB image by using the first depth value of each pixel of the depth image in the same scene, and can acquire more scene information by using the depth image and the RGB image. By dividing the RGB image by dividing the depth image, the accuracy of the region image segmentation can be improved; and then the RGB image is segmented according to the second depth value and the preset rule, and at least one region image corresponding to the preset processing model is processed, Dividing the image into at least one area image according to its own needs, and selecting one or more area images to be processed in at least one area image, and then processing the corresponding area image based on the preset processing model, not only enabling the user to The local processing of the image is realized according to the requirements of the user, and the user can meet the individualized requirements for different processing of the image of different regions; finally, the pixel value of the processed region image is merged with the pixel value of the original RGB image to generate the final fusion. The image is fused with pixel values, and the image is output, taking into account both computational efficiency and fusion effects.

Embodiment 3

FIG. 3 is a schematic structural diagram of an image processing apparatus according to Embodiment 3 of the present invention. The device can be implemented by means of hardware and/or software, and can generally implement the method of the embodiment in the terminal independently. As shown in FIG. 3, the image processing apparatus specifically includes an original image acquiring module 310, an area image processing module 320, and an image fusion module 330.

The original image obtaining module 310 is configured to acquire the depth image and the RGB image in the same scene, and the area image processing module 320 is configured to divide the RGB image according to the depth value in the depth image to obtain at least two regional images. Processing at least one of the area images; the image fusion module 330 is configured to fuse the processed area image with the RGB image.

Based on the foregoing embodiments, the area image processing module 320 may include: a depth value determining sub-module, an area image generating sub-module, and an area image processing sub-module.

The depth value determining sub-module is configured to determine, according to the first depth value of each pixel point of the depth image, a second depth value corresponding to each pixel point of the RGB image; the area image generating sub-module is configured to be used according to the The second depth value and the preset rule divide the RGB image to obtain at least two area images; the area image processing sub-module is configured to process at least one of the area images corresponding to the preset processing model.

Based on the foregoing embodiments, the depth value determining submodule may further include a mapping matrix calculating unit and a second depth value calculating unit.

The mapping matrix calculation unit is configured to calculate a mapping matrix of a first coordinate of each pixel of the depth image and a second coordinate of each pixel of the RGB image; a second depth value calculating unit, configured to And determining, by the second coordinate, the mapping matrix, and the first depth value of each pixel of the depth image, a second depth value corresponding to each pixel of the RGB image.

On the basis of the foregoing embodiments, the area image generation sub-module is specifically configured to: determine whether the second depth value is greater than a preset depth threshold, and if yes, the pixel point corresponding to the second depth value The position is the foreground area, and if not, the position of the pixel corresponding to the second depth value is used as the background area; and the RGB image is divided according to the foreground area and the background area to obtain two area images.

Based on the above embodiments, the pixel values of the pixels of the area image are respectively represented by 1 or 0; wherein 1 represents the foreground area; 0 represents the background area.

On the basis of the foregoing embodiments, the image fusion module 330 is specifically configured to: perform Gaussian smoothing processing on the processed region image, and acquire smooth pixel values of each pixel of the region image; according to the smoothed pixel value and a pixel value of each pixel of the RGB image, calculating a fused pixel value of the fused image; and outputting the fused image according to the fused pixel value.

In the above embodiments, the preset processing model may include a beautification processing algorithm. For example, in a real-time application scenario, a bilateral filtering beautification processing algorithm may be used to implement a real-time microdermabrasion image effect.

The embodiment further provides a terminal, which includes the image processing apparatus according to any embodiment of the present invention. Illustratively, the terminal may include a device having a photographing function such as a mobile phone, a tablet computer, a smart watch, and a camera.

The image processing apparatus and the mobile terminal provided in the above embodiments may perform any embodiment of the present invention. The provided image processing method has the corresponding functional modules and beneficial effects for performing the method. For details of the techniques not described in detail in the above embodiments, reference may be made to the image processing method provided by any embodiment of the present invention.

Note that the above are only the preferred embodiments of the present invention and the technical principles applied thereto. Those skilled in the art will appreciate that the invention is not limited to the specific embodiments described herein, and that various modifications, changes and substitutions may be made without departing from the scope of the invention. Therefore, the present invention has been described in detail by the above embodiments, but the present invention is not limited to the above embodiments, and other equivalent embodiments may be included without departing from the inventive concept. The scope is determined by the scope of the appended claims.

Claims

An image processing method, comprising:

Obtaining depth images and RGB images in the same scene;

Dividing the RGB image according to a depth value in the depth image to obtain at least two area images, and processing at least one of the area images;

The processed area image is fused with the RGB image.
The image processing method according to claim 1, wherein the dividing the RGB image according to a depth value in the depth image to obtain at least two area images, and processing at least one of the area images comprises:

Determining, according to the first depth value of each pixel point of the depth image, a second depth value corresponding to each pixel point of the RGB image;

Dividing the RGB image according to the second depth value and a preset rule to obtain at least two area images;

Processing at least one of the area images corresponding to the preset processing model.
The image processing method according to claim 2, wherein the determining, according to the first depth value of each pixel of the depth image, the second depth value corresponding to each pixel of the RGB image comprises:

Calculating a mapping matrix of a first coordinate of each pixel of the depth image and a second coordinate of each pixel of the RGB image;

Determining, according to the second coordinate, the mapping matrix, and the first depth value of each pixel of the depth image, a second depth value corresponding to each pixel of the RGB image.
The image processing method according to claim 2 or 3, wherein the dividing the RGB image according to the second depth value and a preset rule to obtain at least two area images comprises:

Determining whether the second depth value is greater than a preset depth threshold,

If yes, the position of the pixel corresponding to the second depth value is used as the foreground area.

If not, the position of the pixel corresponding to the second depth value is used as the background area;

Dividing the RGB image according to the foreground region and the background region results in two region images.
The image processing method according to claim 4, wherein the pixel values of the respective pixel points of the area image are respectively represented by 1 or 0; wherein 1 indicates the foreground area; and 0 indicates the background area.
The image processing method according to claim 5, wherein the fusing the processed area image with the RGB image comprises:

Performing Gaussian smoothing on the processed region image to obtain smooth pixel values of each pixel of the region image;

Calculating a fused pixel value of the fused image according to the smoothed pixel value and a pixel value of each pixel of the RGB image;

A fused image is output according to the fused pixel value.
An image processing apparatus, comprising:

An original image acquisition module, configured to acquire a depth image and an RGB image in the same scene;

An area image processing module, configured to divide the RGB image according to a depth value in the depth image to obtain at least two area images, and process at least one of the area images;

An image fusion module is configured to fuse the processed area image with the RGB image.
The image processing apparatus according to claim 7, wherein the area image processing module comprises:

a depth value determining submodule, configured to determine, according to the first depth value of each pixel of the depth image, Determining a second depth value corresponding to each pixel point of the RGB image;

And an area image generating submodule, configured to divide the RGB image according to the second depth value and a preset rule to obtain at least two area images;

The area image processing sub-module is configured to process at least one of the area images corresponding to the preset processing model.
The image processing device according to claim 8, wherein the depth value determining submodule comprises:

a mapping matrix calculation unit, configured to calculate a mapping matrix of a first coordinate of each pixel of the depth image and a second coordinate of each pixel of the RGB image;

a second depth value calculation unit, configured to determine a second depth value corresponding to each pixel point of the RGB image according to the second coordinate, the mapping matrix, and the first depth value of each pixel point of the depth image.
The image processing apparatus according to claim 8, wherein the area image generation sub-module is specifically configured to:

Determining whether the second depth value is greater than a preset depth threshold,

If yes, the position of the pixel corresponding to the second depth value is used as the foreground area.

If not, the position of the pixel corresponding to the second depth value is used as the background area;

Dividing the RGB image according to the foreground region and the background region results in two region images.