CN110929615B - Image processing method, image processing apparatus, storage medium, and terminal device - Google Patents
Image processing method, image processing apparatus, storage medium, and terminal device Download PDFInfo
- Publication number
- CN110929615B CN110929615B CN201911112965.9A CN201911112965A CN110929615B CN 110929615 B CN110929615 B CN 110929615B CN 201911112965 A CN201911112965 A CN 201911112965A CN 110929615 B CN110929615 B CN 110929615B
- Authority
- CN
- China
- Prior art keywords
- image
- camera
- auxiliary
- main
- scene
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 35
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 238000000034 method Methods 0.000 claims abstract description 43
- 238000013507 mapping Methods 0.000 claims description 21
- 230000006978 adaptation Effects 0.000 claims description 18
- 230000004927 fusion Effects 0.000 claims description 14
- 238000013527 convolutional neural network Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 2
- 230000008901 benefit Effects 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000007547 defect Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 239000013598 vector Substances 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 101100248200 Arabidopsis thaliana RGGB gene Proteins 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000009432 framing Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
Abstract
The disclosure provides an image processing method, an image processing device, a storage medium and a terminal device, and relates to the technical field of image processing. The method is applied to terminal equipment, the terminal equipment comprises a main camera and at least one auxiliary camera, and the number of pixels of the main camera is higher than that of the auxiliary camera; the method comprises the following steps: acquiring a main image acquired by the main camera and at least one auxiliary image acquired by the auxiliary camera; identifying a scene in the primary image or the auxiliary image; determining weights of the main image and the auxiliary image according to a scene recognition result; and fusing the main image and the auxiliary image based on the weight of the main image and the auxiliary image to obtain a target image. The method and the device can realize the cooperative work of the main camera and the auxiliary camera, give play to the advantages of each camera and output high-quality images.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, an image processing apparatus, a computer-readable storage medium, and a terminal device.
Background
At present, terminal devices such as smart phones and the like are generally provided with two or more cameras to support users to select different photographing modes and take pictures with different characteristics by using the characteristics of the different cameras. For example, a main camera generally has high pixels and can capture a high-definition image, and a wide-angle camera can capture an image having a wide viewing range.
However, each camera has its own limitations, including the performance limitations of the image sensor, limitations of physical parameters such as aperture, focal length, etc., resulting in defects in the image captured by a single camera. For example, although the main camera has a high pixel, the higher the requirement for the illumination condition at the time of photographing, and the crosstalk is likely to be received in the case of non-strong illumination, resulting in a large amount of noise in a photographed image.
Therefore, how to overcome the limitation of a single camera and shoot high-quality images is an urgent problem to be solved in the prior art.
It is noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure and therefore may include information that does not constitute prior art that is already known to a person of ordinary skill in the art.
Disclosure of Invention
The present disclosure provides an image processing method, an image processing apparatus, a computer-readable storage medium, and a terminal device, thereby improving, at least to some extent, the problem that existing image capturing is limited by the limitation of a single camera.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to a first aspect of the present disclosure, an image processing method is provided, which is applied to a terminal device, where the terminal device includes a main camera and at least one auxiliary camera, and the number of pixels of the main camera is higher than that of the auxiliary camera; the method comprises the following steps: acquiring a main image acquired by the main camera and at least one auxiliary image acquired by the auxiliary camera; performing scene recognition on at least one of the main image and the auxiliary image; determining weights of the main image and the auxiliary image according to a scene recognition result; and fusing the main image and the auxiliary image based on the weight to obtain a target image.
According to a second aspect of the present disclosure, there is provided an image processing apparatus configured to a terminal device including a main camera and at least one auxiliary camera, the main camera having a higher number of pixels than the auxiliary camera; the device comprises: an image acquisition module for acquiring a main image acquired by the main camera and at least one auxiliary image acquired by the auxiliary camera; a scene recognition module for recognizing a scene in the main image or the auxiliary image; a weight determination module for determining the weight of the main image and the auxiliary image according to the scene recognition result; and the image fusion module is used for fusing the main image and the auxiliary image based on the weight of the main image and the auxiliary image to obtain a target image.
According to a third aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described image processing method.
According to a fourth aspect of the present disclosure, there is provided a terminal device comprising: a processor; a memory for storing executable instructions of the processor; a main camera; and at least one auxiliary camera; wherein the processor is configured to perform the above-described image processing method via execution of the executable instructions.
The technical scheme of the disclosure has the following beneficial effects:
according to the image processing method, the image processing device, the storage medium and the terminal equipment, when a user takes a picture, the main camera collects the main image, the auxiliary camera collects the auxiliary image, scene recognition is carried out on the image to determine the weight of the main image and the auxiliary image, and finally image fusion is carried out based on the weight to obtain the target image. On one hand, the finally output target image is formed by fusing the main image and the auxiliary image, the characteristic of high pixel resolution of the main camera and the advantage of cooperative work of the auxiliary camera can be exerted, the defect that the main camera is serious in noise in a non-strong-light environment and the defect of the auxiliary camera in image details are made up, and the high-quality image is output. On the other hand, the weight of the image fusion is determined according to the scene recognition result, so that the adaptation conditions of different cameras to different scenes are considered, the advantages and the disadvantages of the characteristics of each camera can be further exploited, and the intelligent image fusion is realized. On the other hand, the processing of the image belongs to a software algorithm process, can be realized by utilizing the camera configuration of the existing terminal equipment, and does not need to change hardware, thereby saving the cost and having higher practicability.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. It is apparent that the drawings in the following description are only some embodiments of the present disclosure, and that other drawings can be obtained from those drawings without inventive effort for a person skilled in the art.
Fig. 1 shows a flowchart of an image processing method in the present exemplary embodiment;
FIG. 2 illustrates a sub-flowchart of an image processing method in the present exemplary embodiment;
fig. 3 shows a schematic diagram of a color filter array in the present exemplary embodiment;
fig. 4 is a schematic diagram showing the acquisition of a main image in the present exemplary embodiment;
fig. 5 shows a schematic flowchart of image processing in the present exemplary embodiment;
fig. 6 shows a block diagram of the structure of an image processing apparatus in the present exemplary embodiment;
FIG. 7 illustrates a computer-readable storage medium for implementing the above-described method in the present exemplary embodiment;
fig. 8 shows a terminal device for implementing the above method in the present exemplary embodiment.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the embodiments of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
Exemplary embodiments of the present disclosure provide an image processing method, which may be applied to a terminal device such as a mobile phone, a tablet computer, and a digital camera. The terminal equipment is provided with at least two cameras, one of the cameras is a main camera, and the rest cameras are auxiliary cameras. The number of pixels of the main camera is higher than that of the auxiliary camera, so that the view finding preview can be provided when the picture is taken, and the image with higher definition can be acquired. The auxiliary camera is used for taking pictures in different modes, and can comprise any one or more of the following components: depth of field camera, wide angle camera, long focus camera, black and white camera.
Fig. 1 shows a flow of the method, which may include the following steps S110 to S140:
step S110, a main image captured by the main camera and at least one auxiliary image captured by the auxiliary camera are acquired.
The main image and the auxiliary image are images which are acquired aiming at the same scene or the same target at the same time. When the user is shooing, can present through main camera and find a view the preview, when the user pressed the shutter key, main camera and supplementary camera can gather the image simultaneously. Generally, the main content in the main image and the subsidiary image is the same, but the framing ranges of the main camera and the subsidiary camera may be different, resulting in different background ranges of the main image and the subsidiary image. For example, when the auxiliary camera is a wide-angle camera, the view range is large, and a large area of background image around the target can be shot.
It should be noted that, if the terminal device is equipped with only one auxiliary camera, an auxiliary image is acquired through the auxiliary camera; if the terminal equipment is provided with a plurality of auxiliary cameras, any one or more of the auxiliary cameras can be adopted, so that each auxiliary camera respectively collects an auxiliary image. Which auxiliary camera or cameras are specifically enabled may be determined according to the photographing mode selected by the user or the current photographing environment, for example: when a user selects a wide-angle mode, the main camera can be started to collect the main image, and the wide-angle camera is started to collect the wide-angle auxiliary image; when the face image is shot at present, the system identifies the face shooting environment, the main camera can be started to collect the main image, and the long-focus camera is started to collect the close-range auxiliary image.
Step S120, performing scene recognition on at least one of the main image and the auxiliary image.
In the present exemplary embodiment, scene recognition may be performed using any one of the main image and the auxiliary image, for example: in a general mode, because the definition of the main image is higher, the scene in the main image can be identified; in the wide-angle mode, the scene in the wide-angle auxiliary image can be identified because the wide-angle auxiliary image has more image content. Of course, scene recognition may be performed in combination with any of the main image and the auxiliary image. The scene recognition result may be a determination of what kind of scene the image is, or may be a matching probability of the image with each standard scene. The standard scene refers to a pre-selected series of typical scenes that can be used as a reference, such as sky, seaside, street, indoor, night scene, etc. In view of the complexity of the image content, the image content usually contains a plurality of scene elements, which can be regarded as the synthesis of different scenes, so in scene recognition, the features of the different scene elements in the image can be extracted, and the matching probability of the image and each standard scene is determined according to the proportion of the features in the image.
In an alternative embodiment, referring to fig. 2, step S120 may be specifically implemented by the following step S210:
step S210, at least one image in the main image and the auxiliary image is processed by using a pre-trained convolutional neural network, and the matching probability of the image and each standard scene is obtained.
It should be noted that, the convolutional neural network in step S210 does not classify the image, but outputs the matching probability corresponding to different standard scenes, and the probability should be close to the actual situation. Therefore, the convolutional neural network may not set a Softmax layer (a normalization index layer, which increases the difference between values after Softmax processing and causes distortion of the result), and use the last fully-connected layer as an output layer, where each dimension of the output layer corresponds to the matching probability of the input image and each standard scene respectively. For example, 50 standard scenes are set, and 50 dimensions are set for the output layer. If the input layer of the convolutional neural network can be set to be a single channel, then when step S210 is executed, the main image and each auxiliary image can be input into the convolutional neural network, and multiple sets of scene recognition results are obtained correspondingly; the input layer of the convolutional neural network can also be set to be double channels or three channels and the like, a plurality of images can be input at one time, and a group of scene recognition results can be obtained after comprehensive processing of the network.
The following is an exemplary description of the training process of a convolutional neural network:
firstly, an initial network structure is set according to actual requirements, for example: the terminal equipment is provided with a plurality of auxiliary cameras, and each time at least one auxiliary camera shoots an auxiliary image, the network can be set to be dual-channel input; the number of the convolution layers and the pooling layers, and the size of each layer are set according to the number of pixels of the main camera and the auxiliary camera. And (4) not setting a Softmax layer in the network, and outputting a result after the last intermediate layer is completely connected.
Then, a data set for training is prepared, and a data set for open-source scene recognition can be selected, but most of the open-source scene recognition in the data set is a label for scene classification, and cannot be directly used in the scheme. The method is improved on the basis of the source data set, the proportion of a main scene and the proportion of a secondary scene in each image of the data set can be set manually, or object elements are detected from each image of the data set through target detection to add the proportion of the secondary scene, for example, in an image with a scene classification label of seaside, if a portrait is detected, a portrait area is standardized, the matching probability of the image and the scene of the portrait is added according to the proportion of the portrait in the whole image, and the matching probability of the image and the scene of the seaside is updated. Thus, a scene matching probability array (or vector) corresponding to each image is obtained, and the label is obtained.
And then training the convolutional neural network by using a data set, dividing the data set into a training set and a verification set (for example, 8.
It should be added that if multiple sets of scene recognition results are obtained by performing scene recognition on multiple images in the main image and the auxiliary image in step S120, the multiple sets of scene recognition results may also be synthesized, for example, the matching probabilities of different images and the standard scene are averaged, and the average value is used as the final matching probability.
In step S130, weights of the main image and the auxiliary image are determined according to the scene recognition result.
The main camera and each auxiliary camera have different physical characteristics, and are suitable for different scenes, for example: in the night scene, the black-and-white camera can acquire details in a dark place, and noise is low, so that the method is suitable for the night scene. Therefore, the weight of the main image and the auxiliary image can be determined according to the scene recognition result, the weight can be understood as the degree of engagement between different cameras and the current image scene, and the higher the weight is, the more the corresponding camera is engaged with the current image scene, and the higher the weight is, the higher the weight is to be taken in the subsequent image fusion.
In an alternative embodiment, based on the determined matching probability between the image and each standard scene, as shown in fig. 2, step S130 may be specifically implemented by the following step S220:
and S220, calculating the weights of the main image and the auxiliary image according to the preset adaptation degree of the main camera and the auxiliary camera to each standard scene and the matching probability of at least one image and each standard scene.
The adaptation degree refers to the degree that each camera is suitable for shooting each standard scene, for example, a wide-angle camera is suitable for shooting a scene of combined photograph, and the adaptation degree of the wide-angle camera and the scene of combined photograph is higher; the long-focus camera is suitable for shooting a human face scene, and the adaptability between the long-focus camera and the human face scene is high. For the convenience of subsequent calculation, the adaptation degree can be in the range of 0 to 1.
The following is an exemplary description of the method of calculating the weights: assuming that n standard scenes S1 to Sn are preset, the degree of adaptation of the main camera C1 to each standard scene is recorded as a vector P1:
P1=[(C1,S1),(C1,S2)…(C1,Sn)];
(C1, S1) represents the degree of adaptation of C1 and S1. The adaptation degrees of the auxiliary cameras C2 and C3 to the standard scenes are respectively recorded as vectors P2 and P3.
A main image is collected by the C1 to be K1, and auxiliary images are collected by the C2 and the C3 to be K2 and K3 respectively. Determining the matching probability of at least one image in K1, K2 and K3 and each standard scene through step S210, and recording the matching probability as a vector Q (K):
Q(K)=[(K,S1),(K,S2)…(K,Sn)];
k represents a combination of one or more of K1, K2 and K3.
The weights of K1, K2, K3 may be calculated by:
further, normalization processing may be performed on the calculation results W (K1), W (K2), and W (K3).
And step S140, fusing the main image and the auxiliary image based on the weight to obtain a target image.
The weights are used to determine the specific weight of each image in image fusion, so that the main image and each auxiliary image can be weighted to be fused into one image, i.e., a target image, and the target image is finally output.
When fusing images, the corresponding pixels may be subjected to weighting calculations. However, the number of pixels in the main image and the number of pixels in the auxiliary image are generally different, and a plurality of pixels in the main image and one pixel in the auxiliary image may be merged. Therefore, when the number of pixels of the main camera is integral multiple of that of the auxiliary camera, the fusion method is easier to realize. For example, if the main camera is 6400 ten thousand pixels and the auxiliary camera is 1600 ten thousand pixels, then 2 × 2 pixels in the main image correspond to one pixel in the auxiliary image, and each pixel in 2 × 2 and the pixel in the auxiliary image may be weighted to obtain a new 2 × 2 pixel value.
In an alternative embodiment, the primary camera may be a Quad Bayer (Quad Bayer) color filter array based camera. Referring to fig. 3, the left diagram shows a standard bayer color filter array, the cell array of the filter is arranged as GRBG (or BGGR, GBRG, RGGB), and most cameras (or image sensors) adopt the standard bayer color filter array; the right diagram in fig. 3 shows a four-bayer color filter array, in which adjacent four cells in the cell array of the filter are the same color, and a part of high-pixel cameras (or image sensors) currently adopt the four-bayer color filter array. Based on this, acquiring the main image acquired by the main camera may specifically include:
collecting a raw Bayer image based on a four-Bayer color filter array through a main camera;
and performing demosaicing processing and demosaicing processing on the original Bayer image to obtain a main image.
The bayer image is an image in RAW format, and is image data obtained by converting an acquired optical signal into a digital signal by an image sensor, and each pixel point in the bayer image has only one color of RGB. In the present exemplary embodiment, after the image is captured by the main camera, the obtained raw image data is the raw bayer image, the color arrangement of the pixels in the image is as shown in the right diagram in fig. 3, and the adjacent four pixels have the same color.
Demosaic processing (Remosaic) refers to fusing a raw bayer image based on a quad bayer color filter array to a bayer image based on a standard bayer color filter array; demosaicing (Demosaic) refers to the fusion of bayer images into complete RGB images. As shown in fig. 4, the raw bayer image E may be demosaiced to obtain a bayer image F based on a standard bayer color filter array; and demosaicing the Bayer image F based on the standard Bayer color filter array to obtain a main image K in an RGB format. Demosaicing and demosaicing can be realized by different interpolation algorithms, and can also be realized by other related algorithms such as a neural network, and the like, which is not limited by the disclosure. An ISP (Image Signal Processing) unit is usually provided in the terminal device in cooperation with the camera to perform the above-described demosaicing and demosaicing processes. Each pixel of the main image K has pixel values of three channels of RGB, denoted by C. In addition, the processing procedures of demosaicing and demosaicing may also be combined into a primary interpolation procedure, that is, each pixel point is directly interpolated based on the pixel data in the raw bayer image to obtain the pixel value of the missing color channel, for example, the pixel value may be implemented by using algorithms such as linear interpolation and mean interpolation, so as to obtain the main image.
In an alternative embodiment, after the main image and the auxiliary image are acquired in step S110, the mapping relationship between the main image and the auxiliary image may be determined, and an image mapping table may be generated. The mapping relationship mainly refers to mapping of pixel positions, for example, in the main image acquired by the main camera with 6400 ten thousand pixels and the auxiliary image acquired by the auxiliary camera with 1600 ten thousand pixels, a proportional relationship of 4. The image mapping table is used for recording the information of the mapping relation. When the number of the auxiliary images is two (or more), the image mapping table may further record the mapping relationship between three (or more) images.
It should be added that the viewing ranges of the main image and the auxiliary image may be different, for example, the main image corresponds to a part of the area in the middle of the auxiliary image, and then a part of pixels in the auxiliary image (pixels in the peripheral area) and the main image have no mapping relationship.
Correspondingly, step S140 may specifically be: and based on the weight of the main image and the auxiliary image, fusing the main image and the auxiliary image by adopting the image mapping table to obtain a target image. Through the image mapping table, the auxiliary image pixels corresponding to each pixel in the main image can be found, so that the corresponding pixels are fused as a group, for example, weighting or maximum value is taken, and then the pixels are arranged according to the positions of the pixels in the main image to obtain the target image.
In an alternative embodiment, in order to perform one-to-one fusion on the pixels, after step S110, super-resolution reconstruction may be performed on the auxiliary image according to the resolution of the main image to obtain an auxiliary image with the same number of pixels as the main image, and then the pixels between the main image and the auxiliary image may form a one-to-one correspondence relationship, which facilitates the fusion calculation.
Fig. 5 shows a schematic flow of the present exemplary embodiment. Taking a smart phone as an example, acquiring a main image and an auxiliary image through a main camera and an auxiliary camera respectively, and generating an image mapping table according to the mapping relation of the main image and the auxiliary image; identifying a scene in the image to determine weights of the main image and the auxiliary image; and then, based on the weight of the image and the image mapping table, the main image and the auxiliary image are subjected to fusion processing, and finally, a target image is output.
In summary, in the exemplary embodiment, on one hand, the finally output target image is formed by fusing the main image and the auxiliary image, the characteristic of high pixel resolution of the main camera and the advantage of cooperative work of the auxiliary camera can be brought into play, the defect that the main camera is serious in noise in a non-strong-light environment and the defect in the aspect of image details of the auxiliary camera are made up, and a high-quality image is output. On the other hand, the weight of the image fusion is determined according to the scene recognition result, so that the adaptation situation of different cameras to different scenes is considered, the advantages and the disadvantages of the characteristics of each camera can be further exploited, and the intelligent image fusion is realized. On the other hand, the processing of the image belongs to a software algorithm process, can be realized by utilizing the camera configuration of the existing terminal equipment, and does not need to change hardware, thereby saving the cost and having higher practicability.
Exemplary embodiments of the present disclosure also provide an image processing apparatus that may be configured in a terminal device including a main camera and at least one auxiliary camera, the main camera having a higher number of pixels than the auxiliary camera. As shown in fig. 6, the image processing apparatus 600 may include: an image acquisition module 610 for acquiring a main image acquired by a main camera and at least one auxiliary image acquired by an auxiliary camera; a scene recognition module 620, configured to perform scene recognition on at least one of the main image and the auxiliary image; a weight determining module 630 for determining weights of the main image and the auxiliary image according to the scene recognition result; and an image fusion module 640, configured to fuse the main image and the auxiliary image based on the above weights to obtain a target image.
In an optional implementation, the image processing apparatus 600 may further include: the image mapping module is used for determining the mapping relation between the main image and the auxiliary image and generating an image mapping table; the image fusion module 640 may be configured to fuse the main image and the auxiliary image by using the image mapping table based on the weight to obtain the target image.
In an alternative embodiment, the scene recognition module 620 may be configured to process at least one of the main image and the auxiliary image by using a pre-trained convolutional neural network, so as to obtain a matching probability between the at least one image and each standard scene.
In an alternative embodiment, the weight determining module 630 may be configured to calculate the weights of the main image and the auxiliary image according to the pre-configured degree of adaptation of the main camera and the auxiliary camera to each standard scene, and the matching probability of the at least one image and each standard scene.
In an alternative embodiment, the primary camera may be a quad bayer color filter array based camera; the image acquisition module 610 may be configured to acquire, by the main camera, a raw bayer image based on a quad bayer color filter array, and perform demosaicing and demosaicing to obtain a main image.
In an alternative embodiment, the auxiliary camera may include any one or more of: depth of field camera, wide angle camera, long focus camera, black and white camera.
In an alternative embodiment, the number of pixels of the main camera may be an integer multiple of the number of the auxiliary cameras.
The specific details of each module in the above apparatus have been described in detail in the method section, and details that are not disclosed may refer to the contents of the method section, and thus are not described again.
As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the present disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
Exemplary embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the disclosure as described in the above-mentioned "exemplary methods" section of this specification, when the program product is run on the terminal device.
Referring to fig. 7, a program product 700 for implementing the above method according to an exemplary embodiment of the present disclosure is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In situations involving remote computing devices, the remote computing devices may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to external computing devices (e.g., through the internet using an internet service provider).
The exemplary embodiment of the present disclosure also provides a terminal device capable of implementing the above method. A terminal apparatus 800 according to this exemplary embodiment of the present disclosure is described below with reference to fig. 8. The terminal device 800 shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 8, terminal device 800 may take the form of a general purpose computing device. The components of terminal device 800 may include, but are not limited to: at least one processing unit 810, at least one memory unit 820, a bus 830 connecting the different system components (including memory unit 820 and processing unit 810), a display unit 840 and a camera unit 870, the camera unit 870 including a primary camera and at least one secondary camera, may be used to capture images.
The storage unit 820 stores program code that may be executed by the processing unit 810 to cause the processing unit 810 to perform steps according to various exemplary embodiments of the present disclosure described in the "exemplary methods" section above in this specification. For example, processing unit 810 may perform the method steps shown in fig. 1 or fig. 2.
The storage unit 820 may include readable media in the form of volatile storage units, such as a random access storage unit (RAM) 821 and/or a cache storage unit 822, and may further include a read only storage unit (ROM) 823.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, and may also be implemented by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary embodiments of the present disclosure.
Furthermore, the above-described figures are merely schematic illustrations of processes included in methods according to exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed, for example, synchronously or asynchronously in multiple modules.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functions of two or more modules or units described above may be embodied in one module or unit, according to exemplary embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the terms of the appended claims.
Claims (10)
1. An image processing method is applied to terminal equipment, and is characterized in that the terminal equipment comprises a main camera and at least one auxiliary camera, wherein the number of pixels of the main camera is higher than that of the auxiliary camera; the method comprises the following steps:
acquiring a main image acquired by the main camera and at least one auxiliary image acquired by the auxiliary camera;
performing scene recognition on at least one of the main image and the auxiliary image; the scene recognition result comprises the matching probability of the at least one image and each standard scene;
determining weights of the main image and the auxiliary image according to the scene recognition result;
fusing the main image and the auxiliary image based on the weight to obtain a target image;
wherein the determining weights of the main image and the auxiliary image according to the scene recognition result comprises:
and calculating the weights of the main image and the auxiliary image according to the pre-configured adaptation degree of the main camera, the auxiliary camera and each standard scene and the matching probability of the at least one image and each standard scene.
2. The method according to claim 1, wherein after acquiring the primary image and the auxiliary image, the method further comprises:
determining the mapping relation between the main image and the auxiliary image, and generating an image mapping table;
the fusing the main image and the auxiliary image based on the weight to obtain a target image comprises:
and fusing the main image and the auxiliary image by adopting the image mapping table based on the weight to obtain a target image.
3. The method according to claim 1, wherein the scene recognition of at least one of the primary image and the auxiliary image comprises:
and processing at least one image in the main image and the auxiliary image by utilizing a pre-trained convolutional neural network to obtain the matching probability of the at least one image and each standard scene.
4. The method according to claim 1, characterized in that the terminal device comprises a main camera C1, an auxiliary camera C2 and an auxiliary camera C3 for respectively acquiring a main image K1, an auxiliary image K2 and an auxiliary image K3; the number of the standard scenes is n, and the standard scenes are S1, S2, \ 8230and Sn respectively; the calculating the weight of the main image and the auxiliary image according to the pre-configured degree of adaptation of the main camera, the auxiliary camera and each standard scene, and the matching probability of the at least one image and each standard scene includes:
respectively acquiring the adaptation degrees P1 of the main camera C1 and each standard scene, the adaptation degrees P2 of the auxiliary camera C2 and each standard scene, and the adaptation degrees P3 of the auxiliary camera C3 and each standard scene:
P1=[(C1,S1),(C1,S2)…(C1,Sn)];
P2=[(C2,S1),(C2,S2)…(C2,Sn)];
P3=[(C3,S1),(C3,S2)…(C3,Sn)];
(C1, S1) represents the adaptation degree of the main camera C1 and the standard scene S1;
the weights of the main image K1, the auxiliary image K2, the auxiliary image K3 are calculated by:
Q(K)=[(K,S1),(K,S2)…(K,Sn)];
where K denotes a combination of any one or more of the main image K1, the auxiliary image K2, and the auxiliary image K3, Q (K) denotes a matching probability of K with each of the standard scenes, W (K1) denotes a weight of the main image K1, W (K2) denotes a weight of the auxiliary image K2, and W (K3) denotes a weight of the auxiliary image K3.
5. The method of claim 1, wherein the primary camera is a quad bayer color filter array based camera;
the acquiring a main image captured by the main camera includes:
acquiring a raw Bayer image based on a four-Bayer color filter array through the main camera;
and performing demosaicing processing and demosaicing processing on the original Bayer image to obtain the main image.
6. The method of claim 1, wherein the auxiliary camera comprises any one or more of: depth of field camera, wide-angle camera, long focus camera, black and white camera.
7. The method of any of claims 1 to 6, wherein the number of pixels of the primary camera is an integer multiple of the number of pixels of the secondary camera.
8. An image processing device is configured on a terminal device, and is characterized in that the terminal device comprises a main camera and at least one auxiliary camera, and the number of pixels of the main camera is higher than that of the auxiliary camera; the device comprises:
the image acquisition module is used for acquiring a main image acquired by the main camera and at least one auxiliary image acquired by the auxiliary camera;
a scene recognition module for performing scene recognition on at least one of the main image and the auxiliary image; the scene recognition result comprises the matching probability of the at least one image and each standard scene;
a weight determination module for determining weights of the main image and the auxiliary image according to the scene recognition result;
the image fusion module is used for fusing the main image and the auxiliary image based on the weight to obtain a target image;
wherein the determining weights of the main image and the auxiliary image according to the scene recognition result comprises:
and calculating the weights of the main image and the auxiliary image according to the pre-configured adaptation degree of the main camera, the auxiliary camera and each standard scene and the matching probability of the at least one image and each standard scene.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1 to 7.
10. A terminal device, comprising:
a processor;
a memory for storing executable instructions of the processor;
a main camera; and
at least one auxiliary camera;
wherein the processor is configured to perform the method of any of claims 1 to 7 via execution of the executable instructions.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911112965.9A CN110929615B (en) | 2019-11-14 | 2019-11-14 | Image processing method, image processing apparatus, storage medium, and terminal device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911112965.9A CN110929615B (en) | 2019-11-14 | 2019-11-14 | Image processing method, image processing apparatus, storage medium, and terminal device |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110929615A CN110929615A (en) | 2020-03-27 |
CN110929615B true CN110929615B (en) | 2022-10-18 |
Family
ID=69853844
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911112965.9A Active CN110929615B (en) | 2019-11-14 | 2019-11-14 | Image processing method, image processing apparatus, storage medium, and terminal device |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110929615B (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113572980B (en) * | 2020-04-28 | 2022-10-11 | 华为技术有限公司 | Photographing method and device, terminal equipment and storage medium |
CN113141488B (en) * | 2021-03-23 | 2023-03-21 | 南京威派视半导体技术有限公司 | Color restoration method of small-size pixel image sensor |
CN116017128A (en) * | 2022-12-02 | 2023-04-25 | 北京拙河科技有限公司 | Edge camera auxiliary image construction method and device |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107820020A (en) * | 2017-12-06 | 2018-03-20 | 广东欧珀移动通信有限公司 | Method of adjustment, device, storage medium and the mobile terminal of acquisition parameters |
CN108712608A (en) * | 2018-05-16 | 2018-10-26 | Oppo广东移动通信有限公司 | Terminal device image pickup method and device |
CN109218619A (en) * | 2018-10-12 | 2019-01-15 | 北京旷视科技有限公司 | Image acquiring method, device and system |
CN110430359A (en) * | 2019-07-31 | 2019-11-08 | 北京迈格威科技有限公司 | Shoot householder method, device, computer equipment and storage medium |
CN110445989A (en) * | 2019-08-05 | 2019-11-12 | Oppo广东移动通信有限公司 | Image processing method, device, storage medium and electronic equipment |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103268485A (en) * | 2013-06-09 | 2013-08-28 | 上海交通大学 | Sparse-regularization-based face recognition method capable of realizing multiband face image information fusion |
CN105227837A (en) * | 2015-09-24 | 2016-01-06 | 努比亚技术有限公司 | A kind of image combining method and device |
CN107343140A (en) * | 2017-06-14 | 2017-11-10 | 努比亚技术有限公司 | A kind of image processing method and mobile terminal |
CN107958219A (en) * | 2017-12-06 | 2018-04-24 | 电子科技大学 | Image scene classification method based on multi-model and Analysis On Multi-scale Features |
CN108364275B (en) * | 2018-03-02 | 2022-04-12 | 成都西纬科技有限公司 | Image fusion method and device, electronic equipment and medium |
CN108833785B (en) * | 2018-07-03 | 2020-07-03 | 清华-伯克利深圳学院筹备办公室 | Fusion method and device of multi-view images, computer equipment and storage medium |
CN109167931B (en) * | 2018-10-23 | 2021-04-13 | Oppo广东移动通信有限公司 | Image processing method, device, storage medium and mobile terminal |
CN109194881A (en) * | 2018-11-29 | 2019-01-11 | 珠海格力电器股份有限公司 | Image processing method, system and terminal |
-
2019
- 2019-11-14 CN CN201911112965.9A patent/CN110929615B/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107820020A (en) * | 2017-12-06 | 2018-03-20 | 广东欧珀移动通信有限公司 | Method of adjustment, device, storage medium and the mobile terminal of acquisition parameters |
CN108712608A (en) * | 2018-05-16 | 2018-10-26 | Oppo广东移动通信有限公司 | Terminal device image pickup method and device |
CN109218619A (en) * | 2018-10-12 | 2019-01-15 | 北京旷视科技有限公司 | Image acquiring method, device and system |
CN110430359A (en) * | 2019-07-31 | 2019-11-08 | 北京迈格威科技有限公司 | Shoot householder method, device, computer equipment and storage medium |
CN110445989A (en) * | 2019-08-05 | 2019-11-12 | Oppo广东移动通信有限公司 | Image processing method, device, storage medium and electronic equipment |
Also Published As
Publication number | Publication date |
---|---|
CN110929615A (en) | 2020-03-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN108419023B (en) | Method for generating high dynamic range image and related equipment | |
CN107925751B (en) | System and method for multiple views noise reduction and high dynamic range | |
CN110929615B (en) | Image processing method, image processing apparatus, storage medium, and terminal device | |
WO2021047345A1 (en) | Image noise reduction method and apparatus, and storage medium and electronic device | |
CN108712608B (en) | Terminal equipment shooting method and device | |
CN109671106B (en) | Image processing method, device and equipment | |
US8947501B2 (en) | Scene enhancements in off-center peripheral regions for nonlinear lens geometries | |
CN109844804B (en) | Image detection method, device and terminal | |
WO2021115179A1 (en) | Image processing method, image processing apparatus, storage medium, and terminal device | |
CN110930301B (en) | Image processing method, device, storage medium and electronic equipment | |
CN110809101B (en) | Image zooming processing method and device, electronic equipment and storage medium | |
CN102075683A (en) | Digital image processing apparatus and photographing method of digital image processing apparatus | |
CN113518210A (en) | Method and device for automatic white balance of image | |
EP4053784A1 (en) | Image processing method and apparatus, electronic device, and storage medium | |
CN115550570B (en) | Image processing method and electronic equipment | |
CN113850367A (en) | Network model training method, image processing method and related equipment thereof | |
CN115314617A (en) | Image processing system and method, computer readable medium, and electronic device | |
CN110855957A (en) | Image processing method and device, storage medium and electronic equipment | |
WO2023089510A1 (en) | Method and apparatus for image processing, electronic device, storage medium, program and product | |
CN115835034B (en) | White balance processing method and electronic equipment | |
CN108370415A (en) | Image processing apparatus and image processing method | |
WO2024051697A1 (en) | Image fusion method and apparatus, electronic device, and storage medium | |
CN116437198B (en) | Image processing method and electronic equipment | |
CN112241935A (en) | Image processing method, device and equipment and storage medium | |
CN115278191B (en) | Image white balance method and device, computer readable medium and electronic equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |