WO2018201809A1 - Double cameras-based image processing device and method - Google Patents

Abstract

The invention relates to a double cameras-based image processing device, comprising first and second cameras and a control module. The first camera is used for capturing an overall image. The control module is used for sending a shooting instruction to the second camera; and the second camera captures a local image according to the received shooting instruction. The device is characterized in that the shooting instruction includes shooting information of the first camera and image information of the overall image. With the present invention, the imaging effect of a local object can be improved, and representation of details of the whole image can be enhanced.

Description

Image processing device and method based on dual camera Technical field

The present invention relates to the field of digital image processing technologies, and in particular, to an image processing apparatus and method based on a dual camera.

Background technique

In today's world, digital photography technology is developing rapidly. More and more electronic devices have camera functions, and people's work and life have become more pleasant and convenient. At the same time, everyone's requirements for shooting functions have also increased.

The traditional camera device mostly uses a single camera to perform photographing operations. Since the single camera is limited by the focal length, the aperture size, the shutter speed, and the metering mode, the single shooting mode has a single shooting mode, and the imaging effect is limited, which cannot meet the application requirements of the high-definition image.

In order to overcome the defects of the conventional photographing method, in the prior art, a photographing device designed by a dual camera of a main camera and a sub camera is used. The camera system utilizes the difference between the two cameras in terms of focal length and aperture, and selects one of the appropriate cameras for shooting to improve the imaging quality of the camera. However, the above-mentioned devices are still insufficient in image fusion, and it is difficult to ensure the global definition of images for images with complicated pictures and large depth of field.

Summary of the invention

The present invention provides a dual camera-based image processing apparatus including a first camera, a second camera, and a control module, wherein the first camera is used to capture an overall image; the control module is configured to send a shooting instruction to the a second camera; the second camera is configured to capture a partial image according to the received photographing instruction, and wherein the photographing instruction includes photographing information of the first camera and image information of the whole image.

Preferably, the image information of the overall image includes location information of a specified target in the overall image, wherein the partial image is associated with the specified target.

Preferably, the image information of the overall image is information of a region graphic containing a specified target in the overall image, wherein the partial image is associated with the specified target.

Preferably, the image processing apparatus further includes a processing module for identifying information of the area graphic.

Preferably, the information of the area graphic is central position information of the area graphic detected by the processing module using a depth neural network based image recognition algorithm.

Preferably, the processing module is further configured to perform image fusion on the entire image and the partial image.

Preferably, the partial image may be one or more.

According to another aspect of the present invention, a dual camera based image processing method is provided, comprising the steps of:

Taking an overall image by using the first camera, and recording shooting information of the first camera;

Detecting the overall image to obtain image information of the overall image;

A partial image is captured by the second camera based on the photographing information and the image information.

Wherein detecting the overall image comprises:

A target detection algorithm is employed to detect the center position coordinates of the area graphic containing the specified target in the overall image.

Preferably, the image processing method further includes: fusing the entire image and the partial image into a complete image.

Compared with the prior art, the present invention achieves the following beneficial technical effects: the dual camera-based image processing apparatus and method provided by the present invention detects a specified target in an original image captured by the camera 1 by using a target detection algorithm, and according to The detection result and the shooting position of the camera 1 are used to perform secondary shooting on the specified target using the camera 2, and the original image and the secondary captured image are processed by the image fusion algorithm to compensate for the image due to partial underexposure or focus difference in the original image. Problems such as poor imaging results improve the imaging performance of the specified target and enhance the detailed performance of the entire image.

DRAWINGS

The following drawings are only illustrative of the invention and are not intended to limit the scope of the invention,

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a block diagram showing an embodiment of an image processing apparatus according to the present invention.

2 is a flow chart of a method of performing image processing using the image processing apparatus shown in FIG. 1.

detailed description

In order to make the objects, technical solutions, and advantages of the present invention more comprehensible, the dual camera-based image processing apparatus and method provided in the embodiments according to the present invention will be further described in detail below with reference to the accompanying drawings.

According to the length of the focal length, the lens is generally divided into a wide-angle lens and a telephoto lens. In general, the wide-angle lens has a wide viewing angle and a wider range of viewing than the human eye can, so it is especially suitable for a wide range of overall shooting. In contrast, a telephoto lens is similar to the telescope principle and is suitable for shooting inaccessible objects. However, its range of view is far smaller than the range that people can see.

In order to solve the problem that both wide angle and telephoto can not be combined, the inventors have found through research that a wide-angle lens can be used to capture the whole object, a telephoto lens is used to capture a local target, and then an image processing method is used to merge the images captured by the two to Get an overall wide and partially clear image.

1 is a schematic structural view of an embodiment of the present invention. As shown in FIG. 1, according to an embodiment of the present invention, an image processing apparatus based on a dual camera is provided, which includes a camera 1, a camera 2, and a control module. , storage modules and processing modules. The camera 1 is a short focal length wide-angle lens as a first camera for capturing a wide image of a wide viewing angle; the camera 2 is a telephoto lens as a second camera for capturing a partial clear image of a specified target; and the control module is configured to command according to a user Control and schedule each of the other modules; the storage module is used to store image data collected by the camera 1 and the camera 2 and related software programs; the processing module is used for data operations and image processing.

2 is a flowchart of a method for performing image processing by using the image processing apparatus shown in FIG. 1. As shown in FIG. 2, according to an embodiment of the present invention, an image processing method based on a dual camera is provided, and the method specifically includes the following step:

S10. Take an overall image 1.

According to the user's setting instruction, the control module controls the motor to adjust the shooting position of the camera 1 according to the instruction, sends the photographing instruction 1 to the camera 1, and controls the camera 1 to take a picture. The photographing instruction 1 includes shooting parameters such as a camera position, a focus point, and an aperture size, and the shooting parameters may be user-defined settings or default automatic settings.

The camera 1 receives the photographing instruction 1 from the control module and enters the shooting mode. According to the received shooting parameters, the camera 1 performs the photographing operation after adjusting the focus and the aperture, and obtains the overall image 1 and can send an instruction to the control module to complete the photographing. In order to ensure that the parameters such as the amount of light entering and the sharpness of the overall image 1 captured by the camera 1 reach an index, an image processing algorithm can be used to set the index parameter. For example, the amount of light entering can be evaluated by parameters such as image brightness and exposure; sharpness can be evaluated by parameters such as edge width function, entropy function, and gradient function for image edge width, edge peak, and grayscale change rate.

After the camera 1 is completed, the control module can store the overall image 1 in a binary form in the storage module, and record the shooting information of the camera 1 when the overall image 1 is captured (for example, lens position, focus position, aperture size, exposure time). , image brightness, exposure, etc.).

S20, the specified target is selected.

After the photographing of the camera 1 is completed, the user can select the designated target D in the obtained overall image 1. There are several ways for the user to specify the target. For example, the user can select a fixed area, set the image in the area to the specified target D, or specify according to the characteristics of the target, and use the image recognition algorithm to set the targets with the same type of features. Specify the target D, such as a face.

According to the target specified by the user, the processing module can detect the position information of the specified target D (for example, the center position coordinate of the target) from the overall image 1 captured by the camera 1, and transmit the position information to the control module. The processing module can perform detection by using a target detection algorithm. For example, an image algorithm based on pattern recognition, or an image recognition algorithm based on deep neural networks. An image recognition algorithm based on a deep neural network will be described below as an example.

S201. Divide the whole image 1 including the specified target D into n regions, calculate the similarity between each region and its adjacent region by using the average value of the gray levels in each region, and obtain a two-dimensional array composed of similarity values;

S202. Find the maximum value of the similarity and the corresponding two regions from the two-dimensional array obtained in step 201, and merge the two regions, and record the merged region as m ₀ ; calculate the remaining n-1 regions. Within the similarity, and merge the two regions with the highest similarity, repeat the above steps until they are merged into the same region, and get n-1 regions (m ₀ , m ₁ ... m _n-1 ) .

S203. Adjust the n-1 regions obtained in step S202 to the region pattern A of the pixel n*n. If the region is an irregular pattern, the pixel may be filled into a size of n*n, where the pixel value is filled. Can be the grayscale average of the region;

S204. The depth pattern is used to identify whether the area pattern A obtained in step S203 includes the feature of the specified target D. For example, it can be set that if the area coincidence degree IOU of the area figure A and the designated object D is greater than 60%, it is determined that the area figure A includes the specified recognition target and the center point coordinates of the area figure A are calculated. The above coincidence degree IOU can be defined as:

IOU=S _coincidence / (S _{area A} + S _{area D -} S _coincides )

Wherein, S _{coincides to} represent the area of the coincident portion, S _{area A} represents the area of the area figure A, and S _{area D} represents the area of the designated target D.

S30. Take a partial image 2.

The control module according to the photographing information (for example, lens position, focus position, aperture size, exposure time, image brightness, exposure, etc.) when the entire image 1 is captured by the camera 1 obtained in step S10, and the specified target D obtained in step 20 The position information (for example, the central position coordinates) generates a photographing instruction 2 and transmits it to the camera 2, wherein the photographing instruction 2, for example, may include photographing parameters such as a camera position, a focus point, and an aperture size. In one embodiment, some of these shooting parameters may be user-defined settings, while others may be default automatic settings by the control module based on the shooting information and the location information.

The camera 2 receives the photographing instruction 2 from the control module and enters the shooting mode. According to the received shooting parameters, the camera 2 performs the photographing operation after focusing and aperture adjustment, obtains the partial image 2 containing the specified target D′, and can send an instruction to the control module to complete the shooting, wherein the specified target D in the partial image 2 'The specified object D in the entire image 1 obtained in step S10 is the same object, but the performance in the entire image 1 and the partial image 2 is different. In order to ensure that the parameters such as the amount of light entering and the sharpness of the partial image 2 captured by the camera 2 reach an index, an image processing algorithm can be used to set the index parameter. For example, the amount of light entering can be evaluated by parameters such as image brightness and exposure; sharpness can be evaluated by parameters such as edge width function, entropy function, and gradient function for image edge width, edge peak, and grayscale change rate.

After the camera 2 is photographed, the control module stores the partial image 2 in a binary form in the storage module, and can transmit an image processing instruction to the processing module.

S40. Perform image processing on the entire image 1 and the partial image 2.

After receiving the image processing command, the processing module processes the entire image 1 captured by the camera 1 and the partial image 2 captured by the camera 2 by means of image processing. The image processing method here can provide various options according to user requirements. The following is an example of pixel-level image fusion.

The pixel-level image fusion refers to replacing the specified target D in the overall image 1 with the specified target D′ in the image 2, that is, the specified target D is extracted from the overall image 1 and the designation in the partial image 2 is performed. The target D' is filled into the overall image 1, thereby obtaining a complete image containing a clear locally specified target and a wide viewing angle.

Since the image parameters of the specified target D in the overall image 1 and the specified target D′ in the partial image 2 are different, when performing image fusion, it is necessary to use the image fusion algorithm to remove the partial pixel points from the specified target D′ to match the overall image 1 For example, a simple weighted fusion algorithm, a Laplacian pyramid fusion algorithm, a contrast fusion algorithm, a gradient fusion algorithm, or a wavelet fusion algorithm; in addition, when performing image filling, it is also necessary to optimize the image mosaic edges to improve details. which performed. In the prior art, similar image processing methods are numerous and will not be repeated here.

In another embodiment of the present invention, the camera 1 and the camera 2 may be a photosensitive sensor using a charge coupled device (CCD) type, or a metal oxide semiconductor material may be used as the photosensitive sensor.

In another embodiment of the present invention, the camera 1 and the camera 2 are located on the same plane, and the focusing operation is performed in a motor-driven manner.

In another embodiment of the present invention, the control unit may be a central processing unit, a micro control unit, a programmable gate array, or the like.

In another embodiment of the present invention, the processing unit may be a digital signal processing unit, or a dedicated graphics processing circuit, or a deep learning based neural network processor.

In another embodiment of the present invention, the storage unit may be a storage medium such as a memory, an external hard disk, and a flash memory card.

In another embodiment of the present invention, the user can specify a plurality of targets in the overall image 1 taken by the camera 1 as needed.

According to the above step S20, when the user specifies a plurality of targets, the processing module sequentially detects the specified target D1 to be photographed from the entire image 1 including the plurality of specified targets captured by the camera 1 in the above step S10 by using the target detection algorithm. Specify target D2, specify target D3, ..., and determine the location information of all the specified targets above.

According to the above step S30, the camera 2 receives a plurality of shooting commands transmitted from the control module, and performs shooting on the designated target D1, the designated target D2, the designated target D3, ..., respectively, to obtain a partial image 21 including the specified target D1', including the specified target. A partial image 22 of D2', a partial image 23 containing a specified target D3'.

According to the above step S40, taking the image fusion processing as an example, the processing module replaces the designated target D1, the designated target D2, the designated target D3, ... in the overall image 1 captured by the camera 1 with the image fusion algorithm, respectively, in the partial image 21. The target D1', the designated target D2' in the partial image 22, the specified target D3' in the partial image 23, ... are obtained, thereby obtaining a complete image including a plurality of clear locally specified targets and having a wide viewing angle.

In another embodiment of the present invention, in the above step S40, the image processing instruction is sent to the processing module as a user selectable operation, that is, when the camera 1 and the camera 2 respectively complete the shooting and the overall image 1 and one or more parts are captured. After the image is stored in the storage module, the user can perform the image processing of step S40 as needed, directly output the image, or select a plurality of execution steps S40 from the plurality of partial images.

In another embodiment of the present invention, in the above step S20, the user may also select one or more specified targets D that need to be partially photographed during the camera framing before the camera 1 captures the overall image 1 (for example, in the framing frame). One or more designated targets D) are manually selected; thus, steps S10-S30 or steps S10-S40 are automatically performed by the apparatus provided by the present invention, and the user does not even feel the difference in shooting time and the delay time of image processing.

Although in the above embodiment, the camera 1 is a short-focus lens and the camera 2 is a telephoto lens, it will be understood by those skilled in the art that the focal length here is a relative evaluation, that is, the camera 2 is telephoto compared to the camera 1. The lens 1 is a short-focus lens compared to the camera 2. In addition, although the processing of the whole image and the partial image is performed by taking the pixel-level image fusion as an example, those skilled in the art should understand that other image processing methods may be adopted in other embodiments to implement the dual camera based on the present invention. Image processing apparatus and methods, such as feature level image fusion or decision level image fusion.

Compared with the prior art, the dual camera-based image processing apparatus and method provided in the embodiments of the present invention combines the advantages of the wide-angle camera and the telephoto camera, and analyzes the captured image information by using an image processing algorithm. Extraction and fusion, so that multiple images can complement each other, thereby improving the overall performance of the image.

While the present invention has been described in its preferred embodiments, the invention is not limited to the embodiments described herein, and various changes and modifications may be made without departing from the scope of the invention.

Claims (10)

1. A dual camera-based image processing apparatus includes a first camera, a second camera, and a control module, wherein the first camera is used to capture an overall image; and the control module is configured to send a shooting instruction to the second camera The second camera is configured to capture a partial image according to the received photographing instruction, and wherein the photographing instruction includes photographing information of the first camera and image information of the whole image.
2. The image processing device according to claim 1, wherein the image information of the overall image includes position information of a specified target in the overall image, wherein the partial image is associated with the specified target.
3. The image processing device according to claim 1, wherein the image information of the entire image is information of a region graphic containing a specified target in the entire image, wherein the partial image is associated with the specified target .
4. The image processing device according to claim 3, wherein said image processing device further comprises a processing module for identifying information of said region graphic.
5. The image processing apparatus according to claim 4, wherein the information of the area pattern is center position information of the area pattern detected by the processing module using a depth neural network based image recognition algorithm.
6. The image processing apparatus according to claim 4, wherein the processing module is further configured to perform image fusion on the entire image and the partial image.
7. The image processing device according to any one of claims 1 to 6, wherein the partial image may be one or more.
8. A dual camera based image processing method comprising the following steps:

   Taking an overall image by using the first camera, and recording shooting information of the first camera;

   Detecting the overall image to obtain image information of the overall image;

   A partial image is captured by the second camera based on the photographing information and the image information.
9. The image processing method according to claim 8, wherein detecting the entire image comprises:

   A target detection algorithm is employed to detect the center position coordinates of the area graphic containing the specified target in the overall image.
10. The image processing method according to claim 8, further comprising:

    The entire image and the partial image are fused into a complete image.

Images