CN110458860A - Image processing method and image processing device - Google Patents

Abstract

The present invention proposes a kind of image processing method and image processing apparatus.Described image processing method includes: the multiple depth informations for obtaining multiple images by processor and calculating correspondence image；By processor according to the multiple colouring informations, multiple luminance informations and depth information of image by each image segmentation at foreground area and background area；The foreground area of the first image in image is obtained by processor, wherein the brightness of the foreground area of the first image is greater than the first threshold value；The background area that the second image in image is obtained by processor, wherein the brightness of the background area of the second image is less than the second threshold value；And the foreground area of the first image and the background area of the second image are overlapped to generate high dynamic range images by processor.

Description

Image processing method and image processing device

technical field

The present invention relates to an image processing method and an image processing device, in particular to an image processing method and an image processing device that can generate clear images by using less system resources.

Background technique

In the traditional single-image acquisition technology, if the brightness of the scene presents high contrast, the brightness of the subject (eg, a portrait) will become darker due to the limitation of the dynamic range of the sensor. If a longer exposure time is selected in order to improve the image brightness in the dark area, other bright areas in the image will be overexposed due to too long exposure time. 1 shown. If a shorter exposure time is selected to avoid overexposure of the bright area, the main area in the image will be too dark due to underexposure, as shown in Figure 2. Therefore, no matter how the exposure time is set, it is impossible to capture a clear and bright image of both the subject and the background.

In order to solve the above problems, high dynamic image processing technology is proposed. High dynamic image processing technology is to stack multiple images into a single high dynamic range image, and then adjust the brightness of each pixel one by one through tone reproduction technology. The high dynamic image processing technology can accurately restore the dynamic range of the scene, avoid the image with overexposed areas or dark areas, and can display the details of bright areas and dark areas. However, high dynamic image processing technology usually needs to obtain multiple images with different exposure times, so it must spend more computing time and memory space to complete the image overlay. In order to reduce the image overlay time, the method of regional tone reproduction is proposed. However, regional tone reproduction is prone to halo problems, such as abnormal white or black borders at black and white boundaries in the image, such as halo 300 in FIG. 3 . Therefore, how to use less system resources to generate clear images is the goal that those skilled in the art should strive for.

SUMMARY OF THE INVENTION

In view of this, the present invention provides an image processing method and an image processing apparatus, which can utilize less system resources to generate clear images.

The present invention provides an image processing method, comprising: obtaining a plurality of images by a processor and calculating a plurality of depth information corresponding to the above images; Each image is divided into a foreground area and a background area; the foreground area of the first image in the above image is obtained by the processor, wherein the brightness of the foreground area of the first image is greater than the first threshold value; the first image in the above image is obtained by the processor. background areas of two images, wherein the brightness of the background area of the second image is less than the second threshold value; and the processor superimposes the foreground area of the first image and the background area of the second image to generate a high dynamic range image.

In an embodiment of the present invention, the above-mentioned image processing method further includes: performing an image filtering operation on the background area of the second image by the processor, and the image filtering operation includes a background blur operation and an object removal operation.

In an embodiment of the present invention, the image processing method further includes: obtaining, by a processor, an offset of a corresponding object from a foreground area or a background area of the image, and transforming these images according to the offset. )operate.

In an embodiment of the present invention, the above image processing method further includes: using a processor to tone the edges of the foreground area and the background area of the high dynamic range image according to the brightness of the foreground area of the first image and the background area of the second image. Repeat operation.

In an embodiment of the present invention, the above-mentioned depth information is obtained according to a multi-lens image or structured light information.

The present invention provides an image processing device, comprising: a processor; an image sensor, coupled to the processor and acquiring a plurality of images; and a memory, coupled to the processor. The above-mentioned processor obtains a plurality of images and calculates a plurality of depth information corresponding to the above-mentioned images; divides each image into a foreground area and a background area according to a plurality of color information, a plurality of brightness information and the above-mentioned depth information of the above-mentioned images; obtains the above-mentioned images The foreground area of the first image in the first image, wherein the brightness of the foreground area of the first image is greater than the first threshold value; obtain the background area of the second image in the above image, wherein the brightness of the background area of the second image is less than the second threshold value ; and superimposing the foreground area of the first image and the background area of the second image to generate a high dynamic range image.

In an embodiment of the present invention, the processor performs an image filtering operation on the background region of the second image, and the image filtering operation includes a background blur operation and an object removal operation.

In an embodiment of the present invention, the processor obtains the offset of the corresponding object from the foreground area or the background area of the image, and performs a deformation transformation operation on these images according to the offset.

In an embodiment of the present invention, the processor performs a tone reproduction operation on the edges of the foreground area and the background area of the high dynamic range image according to the brightness of the foreground area of the first image and the background area of the second image.

In an embodiment of the present invention, the above-mentioned depth information is obtained according to a multi-lens image or structured light information.

Based on the above, the image processing method and the image processing device of the present invention will calculate the depth information of multiple images, and divide each image into a foreground area and a background area according to the color information, brightness information and depth information of the images, and then divide each image into a foreground area and a background area. In the image, a foreground area with a brightness greater than the first threshold value and a background area with a brightness less than the second threshold value are obtained. Finally, the aforementioned foreground and background regions are superimposed to generate a high dynamic range image. By applying image segmentation technology to high dynamic range images and superimposing images based on regions, it is possible not to adjust pixel brightness one by one to save computing time and obtain high-quality high dynamic range images.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, the following embodiments are given and described in detail with the accompanying drawings as follows.

Description of drawings

FIG. 1 is a schematic diagram of a background overexposed image in the prior art of the present invention.

FIG. 2 is a schematic diagram of an image in which the subject is too dark in the prior art of the present invention.

3 is a schematic diagram of a halo phenomenon caused by regional tone reproduction in the prior art of the present invention.

FIG. 4 is a block diagram of an image processing apparatus according to an embodiment of the present invention.

FIG. 5 is a flowchart of an image processing method according to an embodiment of the present invention.

FIG. 6 is a schematic diagram of a high dynamic range image according to an embodiment of the present invention.

FIG. 7 is a flowchart of an image processing method according to another embodiment of the present invention.

Description of reference numerals

300: halo;

400: image processing device;

410: processor;

420: lens module;

430: memory;

501(1)～501(n): Image;

502(1)～502(n): depth information;

S510, S520, S530, S540, S550: steps of the image processing method;

701(1)～701(n): Image;

702(1)～702(n): depth information;

S710, S720, S730, S740, S750, S760, S770, S780: steps of the image processing method.

Detailed ways

FIG. 4 is a block diagram of an image processing apparatus according to an embodiment of the present invention.

Referring to FIG. 4 , the image processing apparatus 400 of the present invention includes a processor 410 , a lens module 420 and a memory 430 . The lens module 420 can acquire one or more images and transmit the images to the memory 430 for storage or temporary storage. The processor 410 may perform operations on one or more images in the memory 430 to obtain a high dynamic range image.

The processor 410 may be a central processing unit (Central Processing Unit, CPU), or other programmable general-purpose or special-purpose microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP), programmable Controller, Application Specific Integrated Circuit (ASIC) or other similar components or a combination of the above components. The lens module 420 may include a lens (not shown in the figure) and an image sensor (not shown in the figure). The image sensor may be a Charge Coupled Device (CCD) image sensor, a Complementary Metal-Oxide-Semiconductor (CMOS) or other types of image sensors. The memory 430 can be any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (flash memory), hard disk drive (Hard Disk Drive) , HDD), solid state drive (Solid State Drive, SSD) or similar components or a combination of the above components.

FIG. 5 is a flowchart of an image processing method according to an embodiment of the present invention.

Referring to FIG. 5, in step S510, the processor 410 obtains a plurality of images and calculates depth information corresponding to each image. For example, the processor 410 obtains the images 501(1)-501(n), and respectively calculates the depth information 502(1)-502(n) corresponding to the images 501(1)-501(n). Images 501(1) to 501(n) are images with different exposure times. In one embodiment, the depth information of each image can be obtained by computing images captured by dual or triple cameras. In another embodiment, the depth information of each image can be obtained by collecting depth data with a three-dimensional structured light depth sensor. The present invention does not limit the method for obtaining the depth information of the image.

In step S520, the processor 410 divides each image into a foreground area and a background area. Specifically, the processor 410 can distinguish the foreground area and the background area in the image according to the color information, brightness information of the image pixels and the depth information calculated in step S510.

In step S530, the processor 410 performs image filtering on the image. Specifically, after obtaining background pixels and foreground pixels, the processor 410 may perform object removal processing on foreground pixels whose brightness is greater than a threshold value (eg, a first threshold value) to remove a non-subject and moving person. The pixel brightness greater than the first threshold value means that the brightness of the foreground pixel will not be too dark. The processor 410 may also perform a blurring process on background pixels whose brightness is less than a threshold value (eg, a second threshold value) to obtain a blurred background image. A pixel brightness less than the second threshold value means that the brightness of the background pixel will not be overexposed.

In step S540, the processor 410 performs image fusion. Specifically, the processor 410 selects foreground pixels with normal brightness (ie, not too dark) after the image is cut, and superimposes the selected foreground pixels into the background image without overexposure. As a result, an image with a blurred background and retained high dynamic range image detail is obtained, as shown in Figure 6.

In step S550, the processor 410 outputs a high dynamic range image.

This embodiment proposes a region-based high dynamic range image calculation method, which directly divides the image into a foreground region and a background region, and then superimposes the eligible foreground and background regions into a high dynamic range image. Since the brightness of each pixel is not adjusted to perform image stacking in this embodiment, the computation time for generating a high dynamic range image can be greatly reduced.

FIG. 7 is a flowchart of an image processing method according to another embodiment of the present invention.

Referring to FIG. 7, in step S710, the processor 410 obtains multiple images and calculates depth information corresponding to each image. For example, the processor 410 obtains the images 701(1)-701(n), and respectively calculates the depth information 702(1)-702(n) corresponding to the images 701(1)-701(n). Images 701(1) to 701(n) are images with different exposure times.

In step S720, the processor 410 divides each image into a foreground area and a background area. Specifically, the processor 410 can distinguish the foreground area and the background area in the image according to the color information, brightness information of the image pixels and the depth information calculated in step S710.

In step S730, the processor 410 performs motion estimation. In step S740, the processor 410 performs an image transform. Specifically, the processor 410 can estimate the movement amount of the foreground object or the background object, and perform image deformation transformation according to the movement amount to reduce the distortion of the superimposed image. This prevents images from shifting due to jitter or other movement when shooting multiple images.

In step S750, the processor 410 performs image filtering on the image. Specifically, after obtaining background pixels and foreground pixels, the processor 410 may perform object removal processing on foreground pixels whose brightness is greater than a threshold value (eg, a first threshold value) to remove a non-subject and moving person. The pixel brightness greater than the first threshold value means that the brightness of the foreground pixel will not be too dark. The processor 410 may also perform a blurring process on background pixels whose brightness is less than a threshold value (eg, a second threshold value) to obtain a blurred background image. A pixel brightness less than the second threshold value means that the brightness of the background pixel will not be overexposed.

In step S760, the processor 410 performs image stacking. Specifically, the processor 410 selects foreground pixels with normal brightness (ie, not too dark) after the image is cut, and superimposes the selected foreground pixels into the background image without overexposure.

In step S770, the processor 410 performs tone adjustment on the superimposed images. Specifically, the processor 410 may adjust the overall contrast effect of the superimposed image to obtain an image with higher contrast. In addition, the processor 410 can also perform tone reproduction operation on the edges of the foreground area and the background area of the high dynamic range image according to the brightness of the selected foreground area and background area, so as to solve the problem of halo caused by the traditional regional tone reproduction operation .

In step S780, the processor 410 outputs a high dynamic range image.

In this embodiment, methods such as motion estimation and deformation conversion are used to reduce the problem of incorrect superimposed images caused by hand shake or other movements when shooting multiple images. In this embodiment, the overall contrast effect of the superimposed images can also be adjusted to generate a high dynamic range image with better quality.

To sum up, the image processing method and the image processing device of the present invention will calculate the depth information of multiple images, and divide each image into a foreground area and a background area according to the color information, brightness information and depth information of the images, and then divide each image into a foreground area and a background area. A foreground area with a brightness greater than the first threshold value and a background area with a brightness less than the second threshold value are obtained from the plurality of images. Finally, the aforementioned foreground and background regions are superimposed to generate a high dynamic range image. By applying image segmentation technology to high dynamic range images and superimposing images based on regions, it is possible not to adjust pixel brightness one by one to save computing time and obtain high-quality high dynamic range images.

Although the present invention has been disclosed above with examples, it is not intended to limit the present invention. Any person skilled in the art can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the present invention The scope of protection shall be subject to those defined in the claims.

Claims (12)

1. a kind of image processing method characterized by comprising

Multiple images are obtained by processor and calculate multiple depth informations of corresponding described multiple images；

Believed by the processor according to the multiple colouring informations, multiple luminance informations and the multiple depth of described multiple images Each described multiple images are divided into foreground area and background area by breath；

The foreground area of the first image in described multiple images is obtained by the processor, wherein the first image The foreground area brightness be greater than the first threshold value；

The background area that the second image in described multiple images is obtained by the processor, wherein second image The background area brightness less than the second threshold value；And

The foreground area of the first image and the background area of second image are overlapped by the processor To generate a high dynamic range images.

2. image processing method according to claim 1, further includes: by the processor to the institute of second image It states background area and carries out image filtering operation, described image filter operation includes that background blurring operation and object remove operation.

3. image processing method according to claim 1, further includes: by the processor from the institute of described multiple images State the offset that foreground area or the background area obtain corresponding object, and according to the offset to described multiple images into Row deformation conversion operation.

4. image processing method according to claim 1, further includes: by the processor according to the first image The prospect of the brightness of the background area of the foreground area and second image to the high dynamic range images The edge of region and the background area carries out tone rendering operation.

5. image processing method according to claim 1, wherein the multiple depth information is according to more lens images or knot Structure optical information obtains.

6. a kind of image processing apparatus characterized by comprising

Processor；

Imaging sensor is couple to the processor and obtains multiple images；And

Memory is couple to the processor, wherein the processor

Obtain multiple depth informations of described multiple images and corresponding described multiple images；

It will be each described more according to the multiple colouring informations, multiple luminance informations and the multiple depth information of described multiple images A image segmentation is at foreground area and background area；

The foreground area of the first image in described multiple images is obtained, wherein the foreground area of the first image Brightness be greater than the first threshold value；

The background area for obtaining the second image in described multiple images, wherein the background area of second image Brightness less than the second threshold value；And

The foreground area of the first image and the background area of second image are overlapped to generate high dynamic range Enclose image.

7. image processing apparatus according to claim 6, wherein the background of the processor to second image Region carries out image filtering operation, and described image filter operation includes that background blurring operation and object remove operation.

8. image processing apparatus according to claim 6, wherein the prospect of the processor from described multiple images Region or the background area obtain the offset of corresponding object, and are deformed according to the offset to described multiple images Conversion operation.

9. image processing apparatus according to claim 6, wherein the processor according to the first image it is described before The brightness of the background area of scene area and second image to the foreground areas of the high dynamic range images and The edge of the background area carries out tone rendering operation.

10. image processing apparatus according to claim 6, wherein the multiple depth information is according to more lens images or knot Structure optical information obtains.

11. a kind of image processing method characterized by comprising

Obtain multiple depth informations of multiple images and corresponding described multiple images；

It will be each described more according to the multiple colouring informations, multiple luminance informations and the multiple depth information of described multiple images A image segmentation is at foreground area and background area；And

According to the back of one of the foreground area of one of described multiple images and described multiple images Scene area generates high dynamic range images.

12. a kind of image processing apparatus characterized by comprising

Processor；

Imaging sensor is couple to the processor and obtains multiple images；And

Memory is couple to the processor, wherein the processor

Obtain multiple depth informations of multiple images and corresponding described multiple images；

It will be each described more according to the multiple colouring informations, multiple luminance informations and the multiple depth information of described multiple images A image segmentation is at foreground area and background area；And

According to the back of one of the foreground area of one of described multiple images and described multiple images Scene area generates high dynamic range images.