CN110929615A

CN110929615A - Image processing method, image processing apparatus, storage medium, and terminal device

Info

Publication number: CN110929615A
Application number: CN201911112965.9A
Authority: CN
Inventors: 姚坤
Original assignee: Realme Chongqing Mobile Communications Co Ltd
Current assignee: Realme Chongqing Mobile Communications Co Ltd
Priority date: 2019-11-14
Filing date: 2019-11-14
Publication date: 2020-03-27
Anticipated expiration: 2039-11-14
Also published as: CN110929615B

Abstract

The disclosure provides an image processing method, an image processing device, a storage medium and a terminal device, and relates to the technical field of image processing. The method is applied to terminal equipment, the terminal equipment comprises a main camera and at least one auxiliary camera, and the number of pixels of the main camera is higher than that of the auxiliary camera; the method comprises the following steps: acquiring a main image acquired by the main camera and at least one auxiliary image acquired by the auxiliary camera; identifying a scene in the primary image or the auxiliary image; determining weights of the main image and the auxiliary image according to a scene recognition result; and fusing the main image and the auxiliary image based on the weight of the main image and the auxiliary image to obtain a target image. The method and the device can realize the cooperative work of the main camera and the auxiliary camera, exert the advantages of each camera and output high-quality images.

Description

Image processing method, image processing apparatus, storage medium, and terminal device

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, an image processing apparatus, a computer-readable storage medium, and a terminal device.

Background

At present, terminal devices such as smart phones and the like are generally provided with two or more cameras to support users to select different photographing modes and take pictures with different characteristics by using the characteristics of the different cameras. For example, a main camera generally has high pixels and can capture a high-definition image, and a wide-angle camera can capture an image having a large viewing range.

However, each camera has its own limitations, including performance limitations of the image sensor, limitations of physical parameters such as aperture, focal length, etc., resulting in defects in the image captured by a single camera. For example, although the main camera has a high pixel, the higher the requirement for the illumination condition at the time of photographing, and the crosstalk is likely to be received in the case of non-strong illumination, resulting in a large amount of noise in a photographed image.

Therefore, how to overcome the limitation of a single camera and shoot high-quality images is a problem to be solved urgently in the prior art.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.

Disclosure of Invention

The present disclosure provides an image processing method, an image processing apparatus, a computer-readable storage medium, and a terminal device, thereby improving, at least to some extent, the problem that existing image capturing is limited by the limitation of a single camera.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.

According to a first aspect of the present disclosure, an image processing method is provided, which is applied to a terminal device, the terminal device includes a main camera and at least one auxiliary camera, and the number of pixels of the main camera is higher than that of the auxiliary camera; the method comprises the following steps: acquiring a main image acquired by the main camera and at least one auxiliary image acquired by the auxiliary camera; performing scene recognition on at least one of the main image and the auxiliary image; determining weights of the main image and the auxiliary image according to a scene recognition result; and fusing the main image and the auxiliary image based on the weight to obtain a target image.

According to a second aspect of the present disclosure, there is provided an image processing apparatus configured to a terminal device including a main camera and at least one auxiliary camera, the main camera having a higher number of pixels than the auxiliary camera; the device comprises: an image acquisition module for acquiring a main image acquired by the main camera and at least one auxiliary image acquired by the auxiliary camera; a scene recognition module for recognizing a scene in the main image or the auxiliary image; a weight determination module for determining the weight of the main image and the auxiliary image according to the scene recognition result; and the image fusion module is used for fusing the main image and the auxiliary image based on the weight of the main image and the auxiliary image to obtain a target image.

According to a third aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described image processing method.

According to a fourth aspect of the present disclosure, there is provided a terminal device comprising: a processor; a memory for storing executable instructions of the processor; a main camera; and at least one auxiliary camera; wherein the processor is configured to perform the above-described image processing method via execution of the executable instructions.

The technical scheme of the disclosure has the following beneficial effects:

according to the image fusion method, the image fusion device, the storage medium and the terminal equipment, when a user takes a picture, the main camera collects the main image, the auxiliary camera collects the auxiliary image, scene recognition is carried out on the image to determine the weight of the main image and the auxiliary image, and finally image fusion is carried out based on the weight to obtain the target image. On one hand, the finally output target image is formed by fusing the main image and the auxiliary image, the characteristic of high pixel resolution of the main camera and the advantage of cooperative work of the auxiliary camera can be exerted, the defect that the main camera is serious in noise in a non-strong-light environment and the defect of the auxiliary camera in image details are made up, and the high-quality image is output. On the other hand, the weight of the image fusion is determined according to the scene recognition result, so that the adaptation situation of different cameras to different scenes is considered, the advantages and the disadvantages of the characteristics of each camera can be further exploited, and the intelligent image fusion is realized. On the other hand, the processing of the image belongs to a software algorithm process, can be realized by utilizing the camera configuration of the existing terminal equipment, and does not need to change hardware, thereby saving the cost and having higher practicability.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is apparent that the drawings in the following description are only some embodiments of the present disclosure, and that other drawings can be obtained from those drawings without inventive effort for a person skilled in the art.

Fig. 1 shows a flowchart of an image processing method in the present exemplary embodiment;

FIG. 2 illustrates a sub-flowchart of one method of image processing in the present exemplary embodiment;

fig. 3 shows a schematic diagram of a color filter array in the present exemplary embodiment;

fig. 4 is a schematic diagram showing the acquisition of a main image in the present exemplary embodiment;

fig. 5 shows a schematic flowchart of image processing in the present exemplary embodiment;

fig. 6 shows a block diagram of the structure of an image processing apparatus in the present exemplary embodiment;

FIG. 7 illustrates a computer-readable storage medium for implementing the above-described method in the present exemplary embodiment;

fig. 8 shows a terminal device for implementing the above method in the present exemplary embodiment.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

Exemplary embodiments of the present disclosure provide an image processing method, which may be applied to a terminal device such as a mobile phone, a tablet computer, and a digital camera. The terminal equipment is provided with at least two cameras, one of which is a main camera, and the rest are auxiliary cameras. The number of pixels of the main camera is higher than that of the auxiliary camera, so that the view finding preview can be provided when the picture is taken, and the image with higher definition can be acquired. The auxiliary camera is used for taking pictures in different modes, and can comprise any one or more of the following components: depth of field camera, wide-angle camera, long focus camera, black and white camera.

Fig. 1 shows a flow of the method, which may include the following steps S110 to S140:

step S110, a main image captured by the main camera and at least one auxiliary image captured by the auxiliary camera are acquired.

The main image and the auxiliary image are images which are acquired aiming at the same scene or the same target at the same time. When the user shoots, the user can present the preview of finding a view through the main camera, and when the user pressed the shutter key, the main camera and the auxiliary camera can gather the image simultaneously. Generally, the main content in the main image and the subsidiary image is the same, but the viewing ranges of the main camera and the subsidiary camera may be different, resulting in a difference in the background ranges of the main image and the subsidiary image. For example, when the auxiliary camera is a wide-angle camera, the view range is large, and a large area of background image around the target can be shot.

It should be noted that, if the terminal device is equipped with only one auxiliary camera, an auxiliary image is acquired through the auxiliary camera; if the terminal equipment is provided with a plurality of auxiliary cameras, any one or more of the auxiliary cameras can be adopted, so that each auxiliary camera respectively collects an auxiliary image. Which auxiliary camera or cameras are specifically enabled may be determined according to a photographing mode selected by a user or a current photographing environment, for example: when a user selects a wide-angle mode, the main camera can be started to collect the main image, and the wide-angle camera is started to collect the wide-angle auxiliary image; when the face image is shot at present, the system identifies the face shooting environment, the main camera can be started to collect the main image, and the long-focus camera is started to collect the close-range auxiliary image.

Step S120, performing scene recognition on at least one of the main image and the auxiliary image.

In the present exemplary embodiment, scene recognition may be performed using any one of the main image and the auxiliary image, for example: in a general mode, because the definition of the main image is higher, the scene in the main image can be identified; in the wide-angle mode, the scene in the wide-angle auxiliary image can be identified because the wide-angle auxiliary image has more image content. Of course, scene recognition may be performed in combination with any of the main image and the auxiliary image. The scene recognition result may be a determination of what kind of scene the image is, or may be a matching probability of the image with each standard scene. The standard scene refers to a pre-selected series of typical scenes that can be used as a reference, such as sky, seaside, street, indoor, night scene, etc. In view of the complexity of the image content, the image content usually contains a plurality of scene elements, which can be regarded as the synthesis of different scenes, so in scene recognition, the features of the different scene elements in the image can be extracted, and the matching probability of the image and each standard scene is determined according to the proportion of the features in the image.

In an alternative embodiment, referring to fig. 2, step S120 may be specifically implemented by the following step S210:

step S210, at least one image in the main image and the auxiliary image is processed by utilizing a pre-trained convolutional neural network, and the matching probability of the image and each standard scene is obtained.

It should be noted that, the convolutional neural network in step S210 does not classify the image, but outputs the matching probability corresponding to different standard scenes, and the probability should be close to the actual situation. Therefore, the convolutional neural network may not set a Softmax layer (a normalization index layer, which increases the difference between values after Softmax processing and causes distortion of the result), and use the last fully-connected layer as an output layer, where each dimension of the output layer corresponds to the matching probability of the input image and each standard scene respectively. For example, 50 standard scenes are set, and 50 dimensions are set for the output layer. If the input layer of the convolutional neural network can be set to be a single channel, then when step S210 is executed, the main image and each auxiliary image can be respectively input into the convolutional neural network, and a plurality of sets of scene recognition results are correspondingly obtained; the input layer of the convolutional neural network can also be set to be double channels or three channels and the like, a plurality of images can be input at one time, and a group of scene recognition results are obtained after comprehensive processing of the network.

The following is an exemplary description of the training process of a convolutional neural network:

firstly, an initial network structure is set according to actual requirements, for example: the terminal equipment is provided with a plurality of auxiliary cameras, and each time at least one auxiliary camera shoots an auxiliary image, the network can be set to be dual-channel input; the number of the convolution and pooling layers, and the size of each layer are set according to the number of pixels of the main camera and the auxiliary camera. And (4) not setting a Softmax layer in the network, and outputting a result after the last intermediate layer is completely connected.

And then preparing a data set for training, wherein the data set for the open-source scene recognition can be selected, but the scene recognition in the open-source data set is mostly a label of scene classification, and cannot be directly used in the scheme. The method is improved on the basis of the source data set, the proportion of a main scene and the proportion of a secondary scene in each image of the data set can be set manually, or object elements are detected from each image of the data set through target detection to add the proportion of the secondary scene, for example, in an image with a scene classification label of seaside, if a portrait is detected, a portrait area is standardized, the matching probability of the image and the scene of the portrait is added according to the proportion of the portrait in the whole image, and the matching probability of the image and the scene of the seaside is updated. Thus, a scene matching probability array (or vector) corresponding to each image is obtained, i.e. a label is obtained.

And then training the convolutional neural network by using the data set, dividing the data set into a training set and a verification set (for example, 8:2 division is adopted), adjusting network parameters through the training set, and finishing training when the verification on the verification set reaches a preset accuracy rate to obtain an available convolutional neural network.

It should be added that if multiple sets of scene recognition results are obtained by performing scene recognition on multiple images in the main image and the auxiliary image in step S120, the multiple sets of scene recognition results may also be synthesized, for example, the matching probabilities of different images and the standard scene are averaged, and the average value is used as the final matching probability.

In step S130, weights of the main image and the auxiliary image are determined according to the scene recognition result.

The main camera and each auxiliary camera have different physical characteristics, and are suitable for different scenes, for example: in a night scene, the black-and-white camera can acquire details in a dark place, and noise is low, so that the method is suitable for the night scene. Therefore, the weight of the main image and the auxiliary image can be determined according to the scene recognition result, the weight can be understood as the degree of engagement between different cameras and the current image scene, and the higher the weight is, the more the corresponding camera is engaged with the current image scene is, and the higher the weight is in the subsequent image fusion.

In an alternative embodiment, based on the determined matching probability between the image and each standard scene, as shown in fig. 2, step S120 may be specifically implemented by the following step S220:

and step S220, calculating the weight of the main image and the auxiliary image according to the preset adaptation degree of the main camera and the auxiliary camera to each standard scene and the matching probability of at least one image and each standard scene.

The degree of adaptation refers to the degree that each camera is suitable for shooting each standard scene, for example, a wide-angle camera is suitable for shooting a scene of a photo, and the degree of adaptation of the wide-angle camera to the scene of the photo is high; the long-focus camera is suitable for shooting a human face scene, and the adaptability of the long-focus camera to the human face scene is high. For the convenience of subsequent calculation, the adaptation degree can be in a range of 0-1.

The following is an exemplary description of the method of calculating the weights: assuming that n standard scenes S1 to Sn are preset, the degree of adaptation of the main camera C1 to each standard scene is expressed as a vector P1:

P1＝[(C1,S1),(C1,S2)…(C1,Sn)]；

(C1, S1) indicates the degree of adaptation of C1 and S1. The adaptation degrees of the auxiliary cameras C2 and C3 to the standard scenes are respectively recorded as vectors P2 and P3.

A main image is acquired by C1 as K1, and auxiliary images are acquired by C2 and C3 as K2 and K3 respectively. Determining the matching probability of at least one image in K1, K2 and K3 and each standard scene through step S210, and recording the matching probability as a vector Q (K):

Q(K)＝[(K,S1),(K,S2)…(K,Sn)]；

k represents a combination of any one or more of K1, K2 and K3.

The weights of K1, K2, K3 may be calculated by:

further, normalization processing may be performed on the calculation results W (K1), W (K2), and W (K3).

Step S140, fusing the main image and the auxiliary image based on the above weight to obtain a target image.

The weights are used to determine the specific weight of each image in image fusion, so that the main image and each auxiliary image can be weighted to be fused into one image, i.e., a target image, and the target image is finally output.

When fusing images, the corresponding pixels may be weighted. However, the number of pixels in the main image and the number of pixels in the auxiliary image are generally different, and a plurality of pixels in the main image and one pixel in the auxiliary image may be merged. Therefore, when the number of pixels of the main camera is integral multiple of that of the auxiliary camera, the fusion method is easier to realize. For example, if the main camera is 6400 ten thousand pixels and the auxiliary camera is 1600 ten thousand pixels, then 2 × 2 pixels in the main image correspond to one pixel in the auxiliary image, and each pixel in 2 × 2 and the pixel in the auxiliary image may be weighted to obtain a new 2 × 2 pixel value.

In an alternative embodiment, the primary camera may be a Quad Bayer (Quad Bayer) color filter array based camera. Referring to fig. 3, the left diagram shows a standard bayer color filter array, the cell array of the filter is arranged as GRBG (or BGGR, GBRG, RGGB), and most cameras (or image sensors) adopt the standard bayer color filter array; the right diagram in fig. 3 shows a four-bayer color filter array, in which adjacent four cells in the cell array of the filter are the same color, and a part of high-pixel cameras (or image sensors) currently adopt the four-bayer color filter array. Based on this, acquiring the main image captured by the main camera may specifically include:

collecting a raw Bayer image based on a four-Bayer color filter array through a main camera;

and performing demosaicing processing and demosaicing processing on the original Bayer image to obtain a main image.

The bayer image is an image in RAW format, and is image data obtained by converting an acquired optical signal into a digital signal by an image sensor, and each pixel point in the bayer image has only one color of RGB. In the present exemplary embodiment, after the image is captured by the main camera, the obtained raw image data is the raw bayer image, the color arrangement of the pixels in the image is as shown in the right diagram in fig. 3, and the adjacent four pixels have the same color.

Demosaic processing (Remosaic) refers to fusing a raw bayer image based on a quad bayer color filter array to a bayer image based on a standard bayer color filter array; demosaicing (Demosaic) refers to the fusion of bayer images into complete RGB images. As shown in fig. 4, the raw bayer image E may be demosaiced to obtain a bayer image F based on a standard bayer color filter array; and demosaicing the Bayer image F based on the standard Bayer color filter array to obtain a main image K in an RGB format. Demosaicing and demosaicing can be realized by different interpolation algorithms, and can also be realized by other related algorithms such as a neural network, and the like, which is not limited by the disclosure. An ISP (image signal Processing) unit is usually configured in the terminal device to cooperate with the camera to perform the above-mentioned demosaicing and demosaicing processes. Each pixel of the main image K has pixel values of three channels of RGB, denoted by C. In addition, the processing procedures of demosaicing and demosaicing may also be combined into a primary interpolation procedure, that is, each pixel point is directly interpolated based on the pixel data in the raw bayer image to obtain the pixel value of the missing color channel, for example, the pixel value may be implemented by using algorithms such as linear interpolation and mean interpolation, so as to obtain the main image.

In an alternative embodiment, after the main image and the auxiliary image are acquired in step S110, the mapping relationship between the main image and the auxiliary image may be determined, and an image mapping table may be generated. The mapping relationship mainly refers to mapping of pixel positions, for example, in the main image acquired by the main camera with 6400 ten thousand pixels and the auxiliary image acquired by the auxiliary camera with 1600 ten thousand pixels, the proportional relationship is 4:1, the mapping relationship of positions can be calculated, and the pixel positions corresponding to the pixels in the main image in the auxiliary image can be determined through the mapping relationship. The image mapping table is used for recording the information of the mapping relation. When the number of the auxiliary images is two (or more), the image mapping table may further record the mapping relationship between three (or more) images.

It should be added that the viewing ranges of the main image and the auxiliary image may be different, for example, the main image corresponds to a part of the area in the middle of the auxiliary image, and there is no mapping relationship between a part of pixels in the auxiliary image (pixels in the surrounding area) and the main image.

Correspondingly, step S140 may specifically be: and fusing the main image and the auxiliary image by adopting the image mapping table based on the weight of the main image and the auxiliary image to obtain the target image. Through the image mapping table, the auxiliary image pixels corresponding to each pixel in the main image can be found, so that the corresponding pixels are fused as a group, for example, weighting or maximum value is taken, and then the pixels are arranged according to the positions of the pixels in the main image to obtain the target image.

In an alternative embodiment, in order to perform one-to-one fusion on the pixels, after step S110, super-resolution reconstruction may be performed on the auxiliary image according to the resolution of the main image to obtain an auxiliary image with the same number of pixels as the main image, and then the pixels between the main image and the auxiliary image may form a one-to-one correspondence relationship, which facilitates the fusion calculation.

Fig. 5 shows a schematic flow of the present exemplary embodiment. Taking a smart phone as an example, acquiring a main image and an auxiliary image through a main camera and an auxiliary camera respectively, and generating an image mapping table according to the mapping relation of the main image and the auxiliary image; identifying a scene in the image to determine weights of the main image and the auxiliary image; and then, based on the weight of the image and the image mapping table, the main image and the auxiliary image are subjected to fusion processing, and finally, a target image is output.

In summary, in the exemplary embodiment, on one hand, the finally output target image is formed by fusing the main image and the auxiliary image, the characteristic of high pixel resolution of the main camera and the advantage of cooperative work of the auxiliary camera can be brought into play, the defect that the main camera is serious in noise in a non-strong-light environment and the defect in the aspect of image details of the auxiliary camera are made up, and a high-quality image is output. On the other hand, the weight of the image fusion is determined according to the scene recognition result, so that the adaptation situation of different cameras to different scenes is considered, the advantages and the disadvantages of the characteristics of each camera can be further exploited, and the intelligent image fusion is realized. On the other hand, the processing of the image belongs to a software algorithm process, can be realized by utilizing the camera configuration of the existing terminal equipment, and does not need to change hardware, thereby saving the cost and having higher practicability.

Exemplary embodiments of the present disclosure also provide an image processing apparatus that may be configured in a terminal device including a main camera and at least one auxiliary camera, the main camera having a higher number of pixels than the auxiliary camera. As shown in fig. 6, the image processing apparatus 600 may include: an image acquisition module 610 for acquiring a main image captured by a main camera and at least one auxiliary image captured by an auxiliary camera; a scene recognition module 620, configured to perform scene recognition on at least one of the main image and the auxiliary image; a weight determining module 630 for determining weights of the main image and the auxiliary image according to the scene recognition result; and an image fusion module 640, configured to fuse the main image and the auxiliary image based on the above weights to obtain a target image.

In an optional implementation, the image processing apparatus 600 may further include: the image mapping module is used for determining the mapping relation between the main image and the auxiliary image and generating an image mapping table; the image fusion module 640 may be configured to fuse the main image and the auxiliary image by using the image mapping table based on the above weight to obtain the target image.

In an alternative embodiment, the scene recognition module 620 may be configured to process at least one of the main image and the auxiliary image by using a pre-trained convolutional neural network, so as to obtain a matching probability between the at least one image and each standard scene.

In an alternative embodiment, the weight determining module 630 may be configured to calculate the weights of the main image and the auxiliary image according to the pre-configured degree of adaptation of the main camera and the auxiliary camera to each standard scene, and the matching probability of the at least one image and each standard scene.

In an alternative embodiment, the primary camera may be a quad bayer color filter array based camera; the image acquisition module 610 may be configured to acquire, by the main camera, a raw bayer image based on a quad bayer color filter array, and perform demosaicing and demosaicing to obtain a main image.

In an alternative embodiment, the auxiliary camera may include any one or more of: depth of field camera, wide-angle camera, long focus camera, black and white camera.

In an alternative embodiment, the number of pixels of the main camera may be an integer multiple of the number of pixels of the auxiliary camera.

The specific details of each module in the above apparatus have been described in detail in the method section, and details that are not disclosed may refer to the method section, and thus are not described again.

As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the present disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

Exemplary embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the disclosure described in the above-mentioned "exemplary methods" section of this specification, when the program product is run on the terminal device.

Referring to fig. 7, a program product 700 for implementing the above method according to an exemplary embodiment of the present disclosure is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The exemplary embodiment of the present disclosure also provides a terminal device capable of implementing the above method. A terminal apparatus 800 according to this exemplary embodiment of the present disclosure is described below with reference to fig. 8. The terminal device 800 shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 8, terminal device 800 may take the form of a general purpose computing device. The components of terminal device 800 may include, but are not limited to: at least one processing unit 810, at least one memory unit 820, a bus 830 connecting different system components (including memory unit 820 and processing unit 810), a display unit 840, and a camera unit 870, the camera unit 870 including a main camera and at least one auxiliary camera, may be used to capture images.

The storage unit 820 stores program code that may be executed by the processing unit 810 to cause the processing unit 810 to perform steps according to various exemplary embodiments of the present disclosure described in the "exemplary methods" section above in this specification. For example, processing unit 810 may perform the method steps shown in fig. 1 or fig. 2.

The storage unit 820 may include readable media in the form of volatile storage units, such as a random access storage unit (RAM)821 and/or a cache storage unit 822, and may further include a read only storage unit (ROM) 823.

Storage unit 820 may also include a program/utility 824 having a set (at least one) of program modules 825, such program modules 825 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.

Bus 830 may be any of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

Terminal device 800 can also communicate with one or more external devices 900 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with terminal device 800, and/or with any devices (e.g., router, modem, etc.) that enable terminal device 800 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 850. Also, the terminal device 800 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the internet) via the network adapter 860. As shown, the network adapter 860 communicates with the other modules of the terminal device 800 via a bus 830. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the terminal device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary embodiments of the present disclosure.

Furthermore, the above-described figures are merely schematic illustrations of processes included in methods according to exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functions of two or more modules or units described above may be embodied in one module or unit, according to exemplary embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the terms of the appended claims.

Claims

1. An image processing method is applied to terminal equipment, and is characterized in that the terminal equipment comprises a main camera and at least one auxiliary camera, wherein the number of pixels of the main camera is higher than that of the auxiliary camera; the method comprises the following steps:

acquiring a main image acquired by the main camera and at least one auxiliary image acquired by the auxiliary camera;

performing scene recognition on at least one of the main image and the auxiliary image;

determining weights of the main image and the auxiliary image according to a scene recognition result;

and fusing the main image and the auxiliary image based on the weight to obtain a target image.

2. The method according to claim 1, wherein after acquiring the primary image and the auxiliary image, the method further comprises:

determining the mapping relation between the main image and the auxiliary image, and generating an image mapping table;

the fusing the main image and the auxiliary image based on the weight to obtain a target image comprises:

and fusing the main image and the auxiliary image by adopting the image mapping table based on the weight to obtain a target image.

3. The method according to claim 1, wherein the scene recognition of at least one of the primary image and the auxiliary image comprises:

and processing at least one image in the main image and the auxiliary image by utilizing a pre-trained convolutional neural network to obtain the matching probability of the at least one image and each standard scene.

4. The method according to claim 3, wherein the determining the weight of the main image and the auxiliary image according to the scene recognition result comprises:

and calculating the weights of the main image and the auxiliary image according to the pre-configured adaptation degree of the main camera, the auxiliary camera and each standard scene and the matching probability of the at least one image and each standard scene.

5. The method of claim 1, wherein the primary camera is a quad bayer color filter array based camera;

the acquiring a main image captured by the main camera includes:

collecting, by the primary camera, a raw bayer image based on a quad bayer color filter array;

and performing demosaicing processing and demosaicing processing on the original Bayer image to obtain the main image.

6. The method of claim 1, wherein the auxiliary camera comprises any one or more of: depth of field camera, wide-angle camera, long focus camera, black and white camera.

7. The method of any of claims 1 to 6, wherein the number of pixels of the primary camera is an integer multiple of the number of pixels of the secondary camera.

8. An image processing device is configured on a terminal device, and is characterized in that the terminal device comprises a main camera and at least one auxiliary camera, wherein the number of pixels of the main camera is higher than that of the auxiliary camera; the device comprises:

an image acquisition module for acquiring a main image acquired by the main camera and at least one auxiliary image acquired by the auxiliary camera;

a scene recognition module for performing scene recognition on at least one of the main image and the auxiliary image;

a weight determination module for determining the weight of the main image and the auxiliary image according to the scene recognition result;

and the image fusion module is used for fusing the main image and the auxiliary image based on the weight to obtain a target image.

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1 to 7.

10. A terminal device, comprising:

a processor;

a memory for storing executable instructions of the processor;

a main camera; and

at least one auxiliary camera;

wherein the processor is configured to perform the method of any of claims 1 to 7 via execution of the executable instructions.