WO2019109805A1

WO2019109805A1 - Method and device for processing image

Info

Publication number: WO2019109805A1
Application number: PCT/CN2018/116752
Authority: WO
Inventors: 谭国辉; 姜小刚
Original assignee: Oppo广东移动通信有限公司
Priority date: 2017-12-06
Filing date: 2018-11-21
Publication date: 2019-06-13
Also published as: CN108111749A; CN108111749B

Abstract

The present application proposes a method and a device for processing an image, the method comprising: controlling a main camera to take multiple sets of main images, and meanwhile, controlling a secondary camera to take multiple sets of secondary images; obtaining a reference main image from the multiple sets of main images, and obtaining a reference secondary image taken in the same group as the reference main image from the multiple sets of secondary images; generating a target main image by performing a synthetic noise reduction process on the multiple sets of main images by a first thread and, at the same time, (N-1) multi-threads respectively obtaining, according to the reference main image and the reference secondary image, depth information of (N-1) target images corresponding one-to-one to the (N-1) multi-threads; and, according to preset image processing parameters corresponding to the (N-1) target images, performing corresponding image processing on (N-1) target areas. Thus, different image processing is performed on different areas at the same time, improving image processing efficiency, and satisfying various user processing requirements.

Description

Image processing method and device

Cross-reference to related applications

This application claims the priority of the Chinese Patent Application No. "201711276709.4" filed on December 06, 2017 by the Guangdong Appo Mobile Communications Co., Ltd., entitled "Image Processing Method and Apparatus".

Technical field

The present application relates to the field of image processing, and in particular, to an image processing method and apparatus.

Background technique

At present, in order to meet the needs of users in production and life, the functions of terminal devices are more and more diversified, and image processing functions that meet various needs of users are provided, for example, the beauty of photos in the terminal devices that satisfy the user's photo beauty requirements. The function of the color, for example, the filter adding function that satisfies the user's photo beautification function in the terminal device.

In the related art, after receiving the image processing function selected by the user, the terminal device performs unified processing on the image according to the image processing function selected by the user. However, in practical applications, the user has different image processing on different regions on the image. The above image processing method is difficult to meet the user's personalized image processing requirements.

Summary of the invention

The present application provides an image processing method and apparatus to solve the technical problem in the prior art that it is difficult to perform corresponding image processing on different areas in an image.

An embodiment of the present application provides an image processing method, including: controlling a main camera to capture a plurality of sets of main images, and controlling a sub-camera to take a plurality of sets of sub-images; acquiring a reference main image from the plurality of sets of main images, and Obtaining a reference sub-image captured in the same group as the reference main image in the group sub-image; performing synthesizing noise reduction processing on the plurality of groups of main images by the first thread to generate a target main image, and simultaneously passing (N-1) multi-threads Obtaining depth information of (N-1) target images corresponding to the (N-1) multi-threads one by one according to the reference main image and the reference sub-image, respectively, wherein the (N-1) (N-1) target images corresponding to a plurality of threads are not repeated, N is an integer greater than 2; and corresponding (N) is obtained in the target main image according to depth information of the (N-1) target images -1) target areas, corresponding image processing is performed on the (N-1) target areas according to preset image processing parameters corresponding to the (N-1) target images.

Another embodiment of the present invention provides an image processing apparatus, including: a photographing module, configured to control a main camera to capture a plurality of sets of main images, and simultaneously control a sub-camera to take a plurality of sets of sub-images; and a first acquiring module, configured to Obtaining a reference main image in the group main image, and acquiring a reference sub-image captured in the same group as the reference main image from the plurality of sets of sub-images; the first processing module is configured to pair the plurality of groups by the first thread Performing a synthetic noise reduction process on the image to generate a target main image, and acquiring (N-1) multi-threads according to the reference main image and the reference sub-image respectively by (N-1) multi-threads (N-1) depth information of the target image, wherein the (N-1) target images corresponding to the (N-1) multithreads are not repeated, and N is an integer greater than 2; the second processing module uses Obtaining corresponding (N-1) target regions in the target main image according to the depth information of the (N-1) target images, corresponding to the (N-1) target images according to a preset Image processing parameters perform corresponding image processing on the (N-1) target regions

A further embodiment of the present application provides a computer device including a memory and a processor, wherein the memory stores computer readable instructions, and when the instructions are executed by the processor, the processor performs the above implementation of the present application. The image processing method described in the example.

A further embodiment of the present application provides a non-transitory computer readable storage medium having stored thereon a computer program that, when executed by a processor, implements an image processing method as described in the above embodiments of the present application.

The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

Controlling the main camera to take multiple sets of main images, and controlling the sub-camera to take multiple sets of sub-images, acquiring reference main images from the plurality of sets of main images, and acquiring reference sub-images taken in the same group as the reference main images from the plurality of sets of sub-images, The target main image is generated by synthesizing the noise reduction processing of the plurality of sets of main images by the first thread, and acquiring (N-1) multi-threads according to the reference main image and the reference sub-image respectively by (N-1) multi-threads The depth of field information of the corresponding (N-1) target images, and further, the corresponding (N-1) target regions are acquired in the target main image according to the depth information of the (N-1) target images, according to the preset (N-1) image processing parameters corresponding to the target image perform corresponding image processing on (N-1) target regions. Thereby, corresponding image processing for different regions is realized, and image processing efficiency is improved.

The aspects and advantages of the present invention will be set forth in part in the description which follows.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the embodiments will be briefly described below. Obviously, the drawings in the following description are some embodiments of the present application. Those skilled in the art can also obtain other drawings based on these drawings without paying any creative work.

1 is a flow chart of an image processing method according to an embodiment of the present application;

2 is a schematic diagram of the principle of triangulation according to an embodiment of the present application;

FIG. 3 is a schematic diagram of acquiring depth information of a dual camera according to an embodiment of the present application; FIG.

4 is a schematic diagram of an image processing scenario according to an embodiment of the present application;

FIG. 5 is a schematic diagram of a scene of a tourist photograph according to an embodiment of the present application; FIG.

FIG. 6 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application; FIG.

FIG. 7 is a schematic structural diagram of an image processing apparatus according to another embodiment of the present application; FIG.

FIG. 8 is a schematic structural diagram of an image processing apparatus according to still another embodiment of the present application;

9 is a schematic diagram of an image processing circuit in accordance with an embodiment of the present application.

Detailed ways

The embodiments of the present application are described in detail below, and the examples of the embodiments are illustrated in the drawings, wherein the same or similar reference numerals are used to refer to the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are intended to be illustrative, and are not to be construed as limiting.

The image processing method and apparatus of the embodiment of the present application will be described below with reference to the drawings.

The execution body of the image processing method and apparatus of the present application may be a terminal device, where the terminal device may be a hardware device with a dual camera, such as a mobile phone, a tablet computer, a personal digital assistant, a wearable device, or the like. The wearable device can be a smart bracelet, a smart watch, smart glasses, and the like.

FIG. 1 is a flowchart of an image processing method according to an embodiment of the present application. As shown in FIG. 1, the method includes:

In step 101, the main camera is controlled to take a plurality of sets of main images, and the sub-camera is controlled to take a plurality of sets of sub-images.

Specifically, the terminal device performs photographing through a dual camera system, and the dual camera system calculates depth information by using a main image captured by the main camera and a sub image captured by the sub camera, wherein the dual camera system includes a main camera that acquires a main image of the photographing subject, and A sub-camera that assists the main image to acquire depth information, wherein the main camera and the sub-camera may be arranged in a horizontal direction, or may be arranged in a vertical direction, etc., in order to more clearly describe the dual camera. To obtain the depth of field information, the following describes the principle of obtaining the depth of field information by the dual camera with reference to the accompanying drawings:

In practical applications, the human eye discriminates the depth of field information mainly by relying on binocular vision to distinguish depth of field information, which is the same as the principle of dual camera resolution of depth information, mainly based on the principle of triangular ranging as shown in Fig. 2, based on the figure. 2, in the actual space, draw the imaging object, and the position of the two cameras O _R and O _T , and the focal plane of the two cameras, the distance between the focal plane and the plane of the two cameras is f, in the focal plane Two cameras are positioned to image, resulting in two captured images.

Among them, P and P' are the positions of the same object in different captured images, respectively. The distance from the P point to the left boundary of the captured image is X _R , and the distance from the P′ point to the left boundary of the captured image is X _T . O _R and O _T are two cameras respectively, and the two cameras are on the same plane with a distance B.

Based on the principle of triangulation, the distance Z between the object in Figure 2 and the plane of the two cameras has the following relationship:

Based on this, you can push

Where d is the difference in distance between the positions of the same object in different captured images. Since B and f are constant values, the distance Z of the object can be determined according to d.

Of course, in addition to the triangulation method, other methods can be used to calculate the depth information of the main image. For example, when the main camera and the sub camera are photographed for the same scene, the distance of the object in the scene from the camera is related to the main camera and the sub camera. The displacement difference, the posture difference, and the like of the imaging are proportional, and therefore, in one embodiment of the present application, the above-described distance Z can be obtained according to such a proportional relationship.

For example, as shown in FIG. 3, a map of different point differences is calculated by the main image acquired by the main camera and the sub-image acquired by the sub-camera, which is represented by a disparity map, which is the same on the two images. The difference in displacement of the points, but since the difference in displacement in the triangulation is proportional to Z, the disparity map is often used directly as the depth of field map.

Specifically, in the embodiment of the present application, the depth image information for the same object in the main image and the sub image is calculated by the main image captured by the main camera and the sub image captured by the sub camera, and the main image is used as the final image of the final image. The base image, and in order to avoid the depth information of the main image and the sub image, the depth of field information calculation is inaccurate due to the difference between the main image and the sub image, or the main image is unclear, resulting in the final imaging effect. Well, control the main camera to shoot multiple sets of main images, and control the sub-camera to shoot multiple sets of sub-images, so as to make optimal selection among multiple sets of main images and multiple sets of sub-images, improve the accuracy of the depth information calculation and the final imaging effect. .

Of course, since the imaging effect of the camera in the dark light is poor, when shooting in a well-lit environment, both the main camera and the sub-camera have better imaging effects. Therefore, in this application scenario, in order to improve the shooting efficiency. Further improving the image processing efficiency, in one embodiment of the present application, when the ambient light brightness is sufficient, the number of groups of the main image and the sub image to be photographed can be reduced, for example, only one set of main image and sub image are taken, and on the other hand, light is consumed. Sufficient, a set of main images and a set of sub-images can meet the requirements of the accuracy of the depth of field information and the resolution of the image. On the other hand, only taking a set of main images and sub-images improves the image processing efficiency.

In another embodiment of the present application, detecting the brightness of the shooting environment, for example, detecting the brightness of the current shooting environment by using a light sensor, if the detected brightness is less than a preset threshold, indicating that the current ambient brightness may affect the imaging effect of the terminal device, thereby , controlling the main camera and the sub camera to simultaneously shoot multiple sets of main images and multiple sets of sub images.

The preset threshold may be a reference brightness value used to determine whether the ambient brightness affects the imaging effect according to data of a large number of experiments, and the preset threshold may also be related to the imaging hardware of the terminal device, and the better the sensitivity of the imaging hardware is. The lower the preset threshold.

Step 102: Acquire a reference main image from the plurality of sets of main images, and acquire a reference sub-image taken in the same group as the reference main image from the plurality of sets of sub-images.

Step 103: Perform a composite noise reduction process on the plurality of sets of main images by the first thread to generate a target main image, and acquire (N-1) by (N-1) multi-threads according to the reference main image and the reference sub-image respectively. The depth of field information of the (N-1) target images corresponding to the multi-thread one-to-one, wherein (N-1) target images corresponding to (N-1) multi-threads are not repeated, and N is an integer greater than 2.

Based on the above analysis, when the dual camera acquires the depth information, it is necessary to acquire the position of the same object in different captured images. Therefore, if the two images of the dual cameras that acquire the depth information are relatively close, the efficiency and accuracy of the depth information acquisition can be improved. rate.

It can be understood that, in the embodiment of the present application, since the main camera and the sub camera simultaneously capture multiple sets of main images and multiple sets of sub images, the image information of the main image and the sub image belonging to the same group taken at the same time point is relatively Close to, and according to the source main image and the sub-image to calculate the depth of field information, it can ensure that the acquired depth information is more accurate.

Specifically, the reference main image is selected from the plurality of sets of main images, and the reference sub-images taken in the same group as the reference main image are selected from the plurality of sets of sub-images, and it is emphasized that, during actual shooting, the main image and the sub-picture The image captures a plurality of sets of images at the same frequency, wherein the main image and the sub-image taken at the same time belong to the same group of images, for example, in chronological order, the plurality of sets of main images taken by the main camera include the main image 11 and the main image. The image 12..., the plurality of sets of main images captured by the sub-camera include the sub-image 21, the sub-image 22, ..., the main image 11 and the sub-image 21 are the same group of images, and the main image 12 and the sub-image 22 are the same group of images... Further improving the efficiency and accuracy of the depth information acquisition, and selecting a reference main image with higher definition from the plurality of sets of main images, of course, when the number of image frames in the acquired image group is large, in order to improve the selection efficiency, It is also possible to initially select several frame main images and corresponding several sub-images according to image sharpness and the like, and select a reference main image from a plurality of high-definition main images and corresponding several sub-images. And the corresponding reference sub-image.

Further, since the depth of field information calculation takes a long time, the first main thread performs synthetic noise reduction processing on the plurality of sets of main images to generate a target main image, and simultaneously (N-1) multi-threads are respectively based on the reference main image and the reference. The sub-image acquires depth-of-field information of the target image corresponding to the own thread, wherein (N-1) target images corresponding to (N-1) multi-threads are not repeated, N is an integer greater than 2, and N is per target. The calculation of the depth of field information of the image is set, and the calculation amount of the depth information of each thread is made as close as possible.

Therefore, on the one hand, while calculating the depth of field information, synthesizing noise reduction processing on the plurality of sets of main images, and acquiring the target main image, so that after acquiring the depth information, the corresponding depth information and the target main image can be directly performed. Image processing, compared with the first method of obtaining depth of field information, and then performing noise reduction processing on the main image, the image processing efficiency is improved. On the other hand, the noise reduction synthesis according to the plurality of sets of main images makes the target main image detail clear, and the image is clear. The quality is higher, the processed image is better. In another aspect, the calculation of the depth of field information is further subdivided into (N-1) parallel thread runs, and each thread obtains the depth of field information of the corresponding target image, not only The depth of field information of the corresponding target object image can be acquired in a targeted manner to facilitate further image processing for the target image, and the depth of field information of the plurality of target objects is simultaneously calculated by using multiple parallel threads, thereby further shortening the depth of field calculation. The time difference between time and multi-frame noise reduction improves image processing efficiency.

In order to facilitate a clear understanding of the multi-frame synthesis noise reduction process, the multi-frame synthesis noise reduction of the main image is described below in a scene with poor lighting conditions.

When the ambient light is insufficient, imaging devices such as terminal devices generally adopt a method of automatically increasing the sensitivity. However, this way of increasing the sensitivity results in more noise in the image. Multi-frame synthesis noise reduction is to reduce the noise point in the image and improve the image quality of the image taken under high sensitivity. The principle is that noise is a priori knowledge of disordered arrangement. Specifically, after multiple sets of captured images are taken, the noise at the same position may be red noise, green noise, white noise, or even There is no noise, so there is a condition for comparison screening, which can be based on the value of each pixel corresponding to the same position in the plurality of sets of captured images (the value of the pixel includes the number of pixels included in the pixel, and the pixel included More, the higher the value of the pixel, the clearer the corresponding image. The pixels that belong to the noise (ie, the noise) are filtered out.

Further, after filtering out the noise, the noise can be guessed and pixel replaced according to the algorithm of the further method to achieve the effect of removing noise. Through such a process, it is possible to achieve a noise reduction effect with extremely low image quality loss.

For example, as a relatively simple multi-frame synthetic noise reduction method, after acquiring a plurality of sets of captured images, the values of the pixels corresponding to the same position in the plurality of sets of captured images may be read, and a weighted average is calculated by calculating the pixels. The value that produces the value of the pixel at that location in the composite image. In this way, a clear image can be obtained.

In addition, a target image that needs to acquire depth of field information for further image processing according to depth of field information may be set in advance, and the target image may be a face image, a specific gesture motion image (such as a scissors hand, a fueling gesture, etc.), and may include a famous building. Images (such as the Great Wall, Huangshan, etc.), or, may include images of objects of certain shapes (such as circular objects, triangular objects, etc.).

When the object corresponding to the target image is a specific object in the above example, the location of the target image may be determined by techniques such as image recognition and contour recognition, thereby acquiring corresponding depth of field information, and the object corresponding to the target image is only dividing the main image. The image corresponding to the plurality of sub-regions may acquire depth-of-field information corresponding to the image corresponding to the plurality of sub-regions according to the divided pixel positions.

Step 104: Acquire corresponding (N-1) target regions in the target main image according to the depth information of the (N-1) target images, according to preset image processing parameters corresponding to the (N-1) target images. Corresponding image processing is performed on (N-1) target areas.

Specifically, the corresponding (N-1) target regions are acquired in the target main image according to the depth information of the (N-1) target images, wherein the target region includes the target object, and further, according to the preset N-1) image processing parameters corresponding to the target image perform corresponding image processing on (N-1) target regions, whereby each target region is subjected to corresponding image processing, wherein (N-1) The image processing parameters corresponding to the target image may include the beauty processing parameters, the blurring processing parameters, the filter adding processing parameters, and the like, and the image processing parameters corresponding to the (N-1) target images may be the same or different, and the flexibility is satisfied. A variety of image processing needs for users. In an embodiment of the present application, the image processing parameters corresponding to the (N-1) target images may be generated by the system according to the current photographing scene, or may be set by the user, that is, the user is set and each Personalized image processing parameters corresponding to the target image.

Further, in an embodiment of the present application, in order to further improve the image processing effect, the background area of each target area may be blurred according to the depth information of the target image, that is, according to (N-1) target images. The depth of field information is blurred for (N-1) background regions corresponding to (N-1) target regions, respectively.

Specifically, the manner in which the (N-1) background regions corresponding to the (N-1) target regions are blurred according to the depth information of the (N-1) target images includes, but is not limited to, the following manners:

As a possible implementation:

Determining the blurring intensity of (N-1) background regions corresponding to (N-1) target regions according to depth information of (N-1) target images, and further, according to (N-1) background regions The intensity of the corresponding background area is blurred, so that different degrees of blurring are performed according to different depth of field information, so that the blurred image effect is more natural and layered.

It should be noted that, according to different application scenarios, different implementation manners may be used to determine (N-1) backgrounds corresponding to (N-1) target regions according to depth information of (N-1) target images. The ambiguity of the region, as a possible implementation, the more accurate the depth of field information of the target image, the more clear the contour of the target object is. At this time, even if the background region in the target region is blurred, the less It is easy to cause a blurring of the target image. At this time, the blurring intensity corresponding to the background region can be larger. Therefore, the correspondence between the calculation accuracy of the depth information of the target image and the blurring intensity corresponding to the background region can be established in advance. Obtaining a blurring intensity corresponding to the background area according to the correspondence.

The corresponding background area may be blurred according to the ambiguity of the (N-1) background regions by different implementation manners:

Example 1:

Obtaining a blurring coefficient of each pixel according to the blurring intensity of the (N-1) background regions and the depth information of each pixel in the background region corresponding to the target region, wherein the blurring coefficient is related to the blurring intensity of the background region, The larger the imaginary coefficient is, the higher the ambiguity is. For example, the imaginary coefficient of each pixel can be obtained by calculating the imaginary intensity of the background region and the depth information of each pixel in the background region corresponding to the target region. The background area of the target area is blurred according to the ambiguity coefficient of each pixel.

Example two:

Since the difference between the depth information of the background area and the depth information of the target area is larger, the farther the distance between the corresponding background area and the target area is, the less relevant, and the corresponding blurring intensity is larger. In this example, Corresponding relationship between the difference between the depth information of the target area and the background area and the blurring intensity is stored in advance. In the corresponding relationship, the difference between the depth information of the target area and the background area is larger, and the blurring intensity corresponding to the background area is larger. And obtaining a difference between the target area and the background area, and querying the corresponding relationship according to the difference to obtain the ambiguous intensity of the corresponding (N-1) background regions, according to the ambiguity of the (N-1) background regions The corresponding background area is blurred.

Therefore, the image processing method of the embodiment of the present application not only divides the calculation of the depth information into multiple threads for parallel calculation, but also processes the depth of field calculation in parallel with the acquisition of the target main image, thereby improving the efficiency of image processing, and The captured scene includes multiple target objects, especially for different image processing operations including multiple different target objects and multiple target images, which is of great significance and flexibly meets the user's personalized image processing requirements. In order to more clearly illustrate the effect of the image processing in the embodiment of the present application, the following is an example of a specific application scenario:

The first scenario:

In this scenario, the target image corresponds to a face image, and the current photo scene is a multi-person photo scene, including the face images of the three users A, B, and C, wherein the image processing parameter corresponding operation is beautiful. In the example, in the example, the depth of field information of the face images of the three users A, B, and C are respectively acquired by three multi-threads.

As shown in FIG. 4, when the photographing instruction is acquired, the main camera and the sub-camera are controlled to simultaneously capture, and the main image of 4 frames and the sub-image of 4 frames are acquired, wherein the numbers of the main images of the four frames in the shooting order are 11, 12, respectively. The 13 and 14 frames of the sub-pictures are numbered 21, 22, 23, and 24, respectively.

Wherein, the reference main image is selected from the plurality of sets of main images to be 12, and the sub-image 22 captured in the same group as the reference main image is selected from the plurality of sets of sub-images, and further, the composite noise reduction of the plurality of sets of main images is performed by the first thread. The target image is generated by the processing, and the depth information corresponding to the face images of the users A, B, and C are respectively acquired according to the reference main image and the reference sub image by the three multithreads.

Further, the corresponding three target areas including the face image are acquired in the target main image according to the depth information of the three face images, and the three face images are beautifully performed according to the preset beauty processing parameters corresponding to the three face images. Yan treatment, in which user A is a female young user, then face A, whitening, acne and eye-catching beauty treatment for user A, user B is a male young user, then whitening and acne beauty treatment for user B, If the user C is a male child user, the user C is not subjected to the beauty treatment, or the user A is added with the beauty treatment of the pig nose, the user B is added with the beard to add the beauty treatment, and the user C is added with the beauty treatment for the frog eye. Thereby, not only the users A, B, and C are each obtained with a suitable beauty treatment, but also the image processing efficiency is improved.

The second scenario: in the scenario, the target image is a building and a portrait, and the current scene is a travel scene including the building 1 and the character image 2, wherein the operation corresponding to the image processing parameter is a blurring operation, and then In this example, the depth of field information of the image corresponding to the user A and the building 1 is obtained by dividing the two threads into two.

As shown in FIG. 5, when the photographing instruction is acquired, the main camera and the sub-camera are controlled to simultaneously capture, and four main frames and four sub-images are acquired, wherein the main images of the four frames in the order of shooting are respectively 11, 12, The 13 and 14 frames of the sub-pictures are numbered 21, 22, 23, and 24, respectively.

Wherein, the reference main image is selected from the plurality of sets of main images to be 12, and the sub-image 22 captured in the same group as the reference main image is selected from the plurality of sets of sub-images, and further, the composite noise reduction of the plurality of sets of main images is performed by the first thread. The generated target main image is processed, and the depth information of the image corresponding to the user A and the building 1 is respectively acquired according to the reference main image and the reference sub image by the two multi-threads.

Further, the corresponding target area is acquired in the target main image according to the depth information of the image corresponding to the user A and the building 1. Further, in order to highlight the current photographing subject, the target area of the user A is not blurred, and the building 1 is performed. The lower-intensity blurring process performs a strong blurring operation on the background area corresponding to the target area where the user A is located and the background area corresponding to the target area in which the building 1 is located, which not only highlights the photographing subject, but also satisfies the user and the building. The image processing efficiency is improved, and the image processing time caused by simultaneously calculating the depth information of the image corresponding to the user A and the building 1 is avoided, and the user A and the building 1 are properly blurred. Processing improves the visual effect of image processing.

In summary, the image processing method in the embodiment of the present application controls the main camera to capture multiple sets of main images, and simultaneously controls the sub-camera to take multiple sets of sub-images, obtains reference main images from multiple sets of main images, and obtains multiple sets of sub-images. Obtaining a reference sub-image taken in the same group as the reference main image, synthesizing and denoising the plurality of sets of main images by the first thread to generate a target main image, and simultaneously (N-1) multi-threading according to the reference main image and the reference respectively The sub-image acquires depth-of-field information of (N-1) target images corresponding to (N-1) multi-threads one by one, and further acquires corresponding images in the target main image according to depth information of (N-1) target images (N-1) target areas, corresponding image processing is performed on (N-1) target areas according to preset image processing parameters corresponding to (N-1) target images. Thereby, corresponding image processing for different regions is realized, and image processing efficiency is improved.

In order to implement the above embodiments, the present application also provides an image processing apparatus. FIG. 6 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application. As shown in FIG. 6, the image processing apparatus includes: a photographing module 100, The first obtaining module 200, the first processing module 300, and the second processing module 400. The photographing module 100 is configured to control the main camera to capture a plurality of sets of main images, and simultaneously control the sub-camera to take a plurality of sets of sub-images.

In an embodiment of the present application, as shown in FIG. 7, the photographing module 100 includes a detecting unit 110 and a photographing unit 120, wherein the detecting unit 110 is configured to detect the brightness of the photographing environment.

The photographing unit 120 is configured to control the main camera to capture the plurality of sets of main images when detecting that the learned brightness is less than the preset threshold, and simultaneously control the sub-camera to take the plurality of sets of sub-images.

The first obtaining module 200 is configured to acquire a reference main image from the plurality of sets of main images, and acquire a reference sub-image taken in the same group as the reference main image from the plurality of sets of sub-images.

The first processing module 300 is configured to perform a composite noise reduction process on the plurality of sets of main images by the first thread to generate a target main image, and obtain (N-1) multi-threads according to the reference main image and the reference sub-image respectively. -1) depth-of-depth information of (N-1) target images corresponding to one thread, wherein (N-1) multi-threaded corresponding (N-1) target images are not repeated, and N is greater than 2. Integer.

The second processing module 400 is configured to acquire corresponding (N-1) target regions in the target main image according to the depth information of the (N-1) target images, according to the preset (N-1) target images. Corresponding image processing parameters perform corresponding image processing on (N-1) target areas.

In an embodiment of the present application, FIG. 8 is a schematic structural diagram of an image processing apparatus according to another embodiment of the present application. As shown in FIG. 8, the apparatus further includes a second obtaining module 500, where the second obtaining module 500 is used. The personalized image processing parameters corresponding to each target image set by the user are obtained.

It should be noted that the foregoing description of the method embodiment is also applicable to the device in the embodiment of the present application, and the implementation principle is similar, and details are not described herein again.

The division of each module in the above image processing apparatus is for illustrative purposes only. In other embodiments, the image processing apparatus may be divided into different modules as needed to complete all or part of the functions of the image processing apparatus.

In summary, the image processing apparatus of the embodiment of the present application controls the main camera to capture a plurality of sets of main images, and simultaneously controls the sub-camera to take a plurality of sets of sub-images, obtains a reference main image from the plurality of sets of main images, and obtains a plurality of sets of sub-images. Obtaining a reference sub-image taken in the same group as the reference main image, synthesizing and denoising the plurality of sets of main images by the first thread to generate a target main image, and simultaneously (N-1) multi-threading according to the reference main image and the reference respectively The sub-picture acquires depth-of-field information of (N-1) target images corresponding to (N-1) multi-threads one by one, and further acquires corresponding information in the target main image according to depth information of (N-1) target images (N-1) target areas, corresponding image processing is performed on (N-1) target areas according to preset image processing parameters corresponding to (N-1) target images. Thereby, corresponding image processing for different regions is realized, and image processing efficiency is improved.

In order to implement the above embodiments, the present application further provides a computer device, wherein the computer device is any device including a memory including a storage computer program and a processor running the computer program, for example, a smart phone, a personal computer, or the like. The computer device further includes an image processing circuit, and the image processing circuit may be implemented by using hardware and/or software components, and may include various processing units defining an ISP (Image Signal Processing) pipeline. Figure 9 is a schematic illustration of an image processing circuit in one embodiment. As shown in FIG. 9, for convenience of explanation, only various aspects of the image processing technique related to the embodiment of the present application are shown.

As shown in FIG. 9, the image processing circuit includes an ISP processor 1040 and a control logic 1050. The image data captured by imaging device 1010 is first processed by ISP processor 1040, which analyzes the image data to capture image statistical information that may be used to determine and/or control one or more control parameters of imaging device 1010. The imaging device 1010 (camera) may include a camera having one or more lenses 1012 and an image sensor 1014, wherein the imaging device 1010 includes two sets of cameras in order to implement the background blurring method of the present application, wherein, with continued reference to FIG. The imaging device 1010 can simultaneously capture a scene image based on the main camera and the sub camera. Image sensor 1014 may include a color filter array (such as a Bayer filter) that may acquire light intensity and wavelength information captured with each imaging pixel of image sensor 1014 and provide a set of primitives that may be processed by ISP processor 1040 Image data, wherein the ISP processor 1040 can calculate depth information and the like based on the original image data acquired by the image sensor 1014 in the main camera provided by the sensor 1020 and the original image data acquired by the image sensor 1014 in the sub camera. Sensor 1020 can provide raw image data to ISP processor 1040 based on sensor 1020 interface type. The sensor 1020 interface may utilize a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above.

The ISP processor 1040 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel can have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 1040 can perform one or more image processing operations on the raw image data, collecting statistical information about the image data. Among them, image processing operations can be performed with the same or different bit depth precision.

ISP processor 1040 can also receive pixel data from image memory 1030. For example, raw pixel data is sent from the sensor 1020 interface to the image memory 1030, and the raw pixel data in the image memory 1030 is then provided to the ISP processor 1040 for processing. Image memory 1030 can be part of a memory device, a storage device, or a separate dedicated memory within an electronic device, and can include DMA (Direct Memory Access) features.

When receiving raw image data from sensor 1020 interface or from image memory 1030, ISP processor 1040 can perform one or more image processing operations, such as time domain filtering. The processed image data can be sent to image memory 1030 for additional processing prior to being displayed. The ISP processor 1040 receives the processed data from the image memory 1030 and performs image data processing in the original domain and in the RGB and YCbCr color spaces. The processed image data may be output to display 1070 for viewing by a user and/or further processed by a graphics engine or a GPU (Graphics Processing Unit). Additionally, the output of ISP processor 1040 can also be sent to image memory 1030, and display 1070 can read image data from image memory 1030. In one embodiment, image memory 1030 can be configured to implement one or more frame buffers. Additionally, the output of ISP processor 1040 can be sent to encoder/decoder 1060 to encode/decode image data. The encoded image data can be saved and decompressed before being displayed on the display 1070 device. Encoder/decoder 1060 can be implemented by a CPU or GPU or coprocessor.

The statistics determined by the ISP processor 1040 can be sent to the control logic 1050 unit. For example, the statistical data may include image sensor 1014 statistical information such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, lens 1012 shading correction, and the like. Control logic 1050 can include a processor and/or a microcontroller that executes one or more routines, such as firmware, and one or more routines can determine control parameters and control of imaging device 1010 based on received statistical data. parameter. For example, the control parameters may include sensor 1020 control parameters (eg, gain, integration time for exposure control), camera flash control parameters, lens 1012 control parameters (eg, focus or zoom focal length), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (eg, during RGB processing), as well as lens 1012 shading correction parameters.

The following are the steps for implementing the image processing method using the image processing technique of FIG. 9:

Controlling the main camera to take multiple sets of main images, and controlling the sub-camera to take multiple sets of sub-images;

Acquiring a reference main image from the plurality of sets of main images, and acquiring a reference sub-image taken in the same group as the reference main image from the plurality of sets of sub-images;

Performing a composite noise reduction process on the plurality of sets of main images by a first thread to generate a target main image, and acquiring (N-1) according to the reference main image and the reference sub image by (N-1) multithreads respectively a depth-of-depth information of (N-1) target images corresponding to the multi-thread one-to-one, wherein (N-1) target images corresponding to the (N-1) multi-threads are not repeated, and N is greater than 2. Integer

Obtaining corresponding (N-1) target regions in the target main image according to the depth information of the (N-1) target images, according to preset presets corresponding to the (N-1) target images The image processing parameters perform corresponding image processing on the (N-1) target areas.

In order to implement the above embodiments, the present application also proposes a non-transitory computer readable storage medium that enables execution of an image processing method as in the above embodiment when instructions in the storage medium are executed by a processor.

In the description of the present specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means a specific feature described in connection with the embodiment or example. A structure, material or feature is included in at least one embodiment or example of the application. In the present specification, the schematic representation of the above terms is not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, various embodiments or examples described in the specification, as well as features of various embodiments or examples, may be combined and combined.

Moreover, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, features defining "first" or "second" may include at least one of the features, either explicitly or implicitly. In the description of the present application, the meaning of "a plurality" is at least two, such as two, three, etc., unless specifically defined otherwise.

Any process or method description in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logic function or process. And the scope of the preferred embodiments of the present application includes additional implementations, in which the functions may be performed in a substantially simultaneous manner or in the reverse order depending on the functions involved, in accordance with the illustrated or discussed order. It will be understood by those skilled in the art to which the embodiments of the present application pertain.

The logic and/or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer readable medium, Used in conjunction with, or in conjunction with, an instruction execution system, apparatus, or device (eg, a computer-based system, a system including a processor, or other system that can fetch instructions and execute instructions from an instruction execution system, apparatus, or device) Or use with equipment. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport a program for use in an instruction execution system, apparatus, or device, or in conjunction with the instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections (electronic devices) having one or more wires, portable computer disk cartridges (magnetic devices), random access memory (RAM), Read only memory (ROM), erasable editable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the computer readable medium may even be a paper or other suitable medium on which the program can be printed, as it may be optically scanned, for example by paper or other medium, followed by editing, interpretation or, if appropriate, other suitable The method is processed to obtain the program electronically and then stored in computer memory.

It should be understood that portions of the application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware and in another embodiment, it can be implemented by any one or combination of the following techniques well known in the art: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

One of ordinary skill in the art can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. The integrated modules, if implemented in the form of software functional modules and sold or used as stand-alone products, may also be stored in a computer readable storage medium.

The above mentioned storage medium may be a read only memory, a magnetic disk or an optical disk or the like. While the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are illustrative and are not to be construed as limiting the scope of the present application. The embodiments are subject to variations, modifications, substitutions and variations.

Claims

An image processing method, comprising:

Controlling the main camera to take multiple sets of main images, and controlling the sub-camera to take multiple sets of sub-images;

Acquiring a reference main image from the plurality of sets of main images, and acquiring a reference sub-image taken in the same group as the reference main image from the plurality of sets of sub-images;

Performing a composite noise reduction process on the plurality of sets of main images by a first thread to generate a target main image, and acquiring (N-1) multi-threads according to the reference main image and the reference sub-image, respectively N-1) depth-of-depth information of (N-1) target images corresponding to a multi-thread one-to-one correspondence, wherein (N-1) target images corresponding to the (N-1) multi-threads are not repeated, and N is An integer greater than 2;

Obtaining corresponding (N-1) target regions in the target main image according to the depth information of the (N-1) target images, according to preset presets corresponding to the (N-1) target images The image processing parameters perform corresponding image processing on the (N-1) target areas.
The method according to claim 1, wherein said (N-1) target regions are corresponding to said image processing parameters corresponding to said (N-1) target images according to a preset Before image processing, it also includes:

The image processing parameters corresponding to each target image set by the user are acquired.
The method according to claim 1 or 2, wherein the controlling the main camera to capture the plurality of sets of main images while controlling the sub-camera to take the plurality of sets of sub-images comprises:

Detecting the brightness of the shooting environment;

If the detection is that the brightness is less than a preset threshold, the main camera is controlled to take a plurality of sets of main images, and the sub-camera is controlled to take a plurality of sets of sub-images.
The method of any of claims 1-3, further comprising:

The (N-1) background regions corresponding to the (N-1) target regions are respectively subjected to blurring processing based on the depth information of the (N-1) target images.
The method according to claim 4, wherein said (N-1) corresponding to said (N-1) target regions are respectively according to depth information of said (N-1) target images The background area is blurred, including:

Determining the blurring intensity of the (N-1) background regions corresponding to the (N-1) target regions according to the depth information of the (N-1) target images;

The corresponding background area is blurred according to the blurring intensity of the (N-1) background regions.
A method according to any of claims 1-5, wherein

The main camera and the sub camera are horizontally arranged, or

The main camera and the sub camera are vertically disposed.
A multi-person co-processing device, comprising:

a shooting module for controlling the main camera to take multiple sets of main images, and controlling the sub camera to take multiple sets of sub images;

a first acquiring module, configured to acquire a reference main image from the plurality of sets of main images, and acquire a reference sub-image captured in the same group as the reference main image from the plurality of sets of sub-images;

a first processing module, configured to perform a composite noise reduction process on the plurality of sets of main images by a first thread to generate a target main image, and simultaneously pass (N-1) multi-threads according to the reference main image and the reference sub- And acquiring, by the image, depth-of-depth information of (N-1) target images corresponding to the (N-1) multi-threads, wherein (N-1) multi-threads correspond to (N-1) The target image is not repeated, and N is an integer greater than 2;

a second processing module, configured to acquire corresponding (N-1) target regions in the target main image according to the depth information of the (N-1) target images, according to the preset and the (N- 1) The image processing parameters corresponding to the target images perform corresponding image processing on the (N-1) target regions.
The device of claim 7 further comprising:

And a second acquiring module, configured to acquire the image processing parameter corresponding to each target image set by the user.
The apparatus according to claim 7 or 8, wherein the photographing module comprises:

a detecting unit, configured to detect brightness of a shooting environment;

The photographing unit is configured to control the main camera to capture the plurality of sets of main images while detecting that the brightness is less than the preset threshold, and simultaneously control the sub-camera to take the plurality of sets of sub-images.
The device according to any one of claims 7-9, further comprising:

The blurring processing module is configured to perform blurring processing on the (N-1) background regions corresponding to the (N-1) target regions according to the depth information of the (N-1) target images.
The device according to claim 10, wherein the blurring processing module is specifically configured to:

Determining the blurring intensity of the (N-1) background regions corresponding to the (N-1) target regions according to the depth information of the (N-1) target images;

The corresponding background area is blurred according to the blurring intensity of the (N-1) background regions.
A device according to any of claims 7-11, wherein

The main camera and the sub camera are horizontally arranged, or

The main camera and the sub camera are vertically disposed.
A computer device, comprising: a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein when the processor executes the program, implementing any one of claims 1-6 The image processing method described.
A computer readable storage medium having stored thereon a computer program, characterized in that the program is executed by a processor to implement the image processing method according to any of claims 1-6.