WO2023015981A1 - Procédé de traitement d'images et son dispositif associé - Google Patents

Procédé de traitement d'images et son dispositif associé Download PDF

Info

Publication number
WO2023015981A1
WO2023015981A1 PCT/CN2022/091225 CN2022091225W WO2023015981A1 WO 2023015981 A1 WO2023015981 A1 WO 2023015981A1 CN 2022091225 W CN2022091225 W CN 2022091225W WO 2023015981 A1 WO2023015981 A1 WO 2023015981A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
block
camera
fused
processing method
Prior art date
Application number
PCT/CN2022/091225
Other languages
English (en)
Chinese (zh)
Inventor
肖斌
乔晓磊
朱聪超
王宇
Original Assignee
荣耀终端有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 荣耀终端有限公司 filed Critical 荣耀终端有限公司
Publication of WO2023015981A1 publication Critical patent/WO2023015981A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present application relates to the field of image processing, in particular to an image processing method and related equipment.
  • the present application provides an image processing method and related equipment, which can perform image restoration processing on low-resolution areas in an image to restore details, thereby improving user experience.
  • an image processing method which is applied to an electronic device including a first camera and a second camera, and the method includes:
  • the electronic device starts the camera; displays a preview interface, and the preview interface includes a first control; detects a first operation on the first control; in response to the first operation, the first camera captures a first image and the second camera captures a second image, the second
  • the definition of an image is lower than that of the second image, the first image includes a first region, and the first region is a region in the first image whose resolution is less than a preset threshold; a mask block is obtained according to the first image, and the mask The block corresponds to the first area; the first image and the second image are fused to obtain the first fused image; according to the mask block, the first image block in the first image and the second image block in the second image are determined , the first image block corresponds to the mask block, and the second image block corresponds to the mask block; the first image block and the second image block are fused to obtain a fused image block; the first fused image is fused with the fused image block , to get the third image.
  • the first control may be the camera key 11 .
  • the embodiment of the present application provides an image processing method, by determining the mask block corresponding to the first region missing details from the first low-resolution image, and then obtaining the mask block corresponding to the first image from the first image An image block, and obtaining a second image block corresponding to the mask block from a second image with high definition and rich details, and fusing the first image block and the second image block to obtain a clear fused image block; Then, the first fused image obtained by merging the first image and the second image is further fused with the fused image block to restore missing details and obtain a high-definition third image.
  • obtaining the mask block according to the first image includes: inputting the first image into a segmentation model for segmentation, and generating a mask block; A region is segmented, and a mask block corresponding to the first region is generated.
  • the segmentation model may be: a fully convolutional neural network.
  • the first image can be finely segmented through the segmentation model to obtain multiple segmented image regions, which facilitates the subsequent independent repair of regions with severe local missing details in the first image without affecting the second image.
  • An image of the surrounding area An image of the surrounding area.
  • merging the first image and the second image to obtain the first fused image includes: merging the first image and the second image using the first fusion model to obtain the first Blend images.
  • the second image since the second image has a higher resolution than the first image, after the fusion of the first image and the second image, the resolution of the overall image can be improved, and a higher-definition first fusion can be obtained. image.
  • merging the first image block and the second image block to obtain the fused image block includes: merging the first image block and the second image block using a second fusion model, Get the fused image block.
  • the resolution of the first image is lower than that of the second image
  • the resolution of the first image block is also lower than that of the second image block, and even there is no Any details. Therefore, by fusing the first image block that is unclear and lacks details with the second image block that is clear and rich in details, a higher-definition fused image block can be obtained.
  • merging the first fused image and the fused image block to obtain the third image includes: merging the first fused image and the fused image block using a third fusion model to obtain the third image Three images.
  • the first fused image has improved overall definition compared with the first image
  • the fused image block has locally improved definition compared with the first image block in the first image
  • the second A fused image is fused with the fused image block, and a part in the first fused image can be further repaired to obtain a third image with higher definition.
  • the method further includes: when the mask block is not obtained according to the first image, fusing the first image and the second image using the first fusion model to obtain the first fusion image .
  • the method further includes: registering the first image and the second image.
  • registration can improve the accuracy of fusing the first image and the second image.
  • the method further includes: registering the first image block and the second image block.
  • registration can improve the accuracy of fusing the first image block and the second image block.
  • registration includes: global registration and/or local registration, global registration is used to register all content in multiple images, and local registration is used to represent Register local content in multiple images.
  • global registration is used to register all content in multiple images
  • local registration is used to represent Register local content in multiple images.
  • the alignment accuracy of all content in multiple images can be improved through global registration
  • the alignment accuracy of local content in multiple images can be improved through local registration.
  • the method further includes: using the training image set and adding random highlight noise to train the first fusion model to obtain the second fusion model, wherein the training image set includes the original image , the original image is annotated with a mask block.
  • the training image set includes the original image
  • the original image is annotated with a mask block.
  • the third fusion model is a Laplace fusion model.
  • the Laplacian fusion model when using the Laplacian fusion model for fusion, can first decompose the first fused image and the fused image block into different spatial frequency bands, and then in each spatial frequency band layer Fusion is performed separately, so that through the frequency division processing, the fusion of the first fusion image and the fusion image blocks can be made more natural, the connection is more delicate, and the obtained third image is of higher quality.
  • an image processing apparatus in a second aspect, includes a unit for performing each step in the above first aspect or any possible implementation manner of the first aspect.
  • an electronic device including a camera module, a processor, and a memory; the camera module is used to collect a first image and a second image, and the definition of the first image is lower than that of the second image , the first image includes a first area, and the first area is an area whose resolution in the first image is less than a preset threshold; the memory is used to store a computer program that can run on the processor; the processor is used to execute the first A step of processing in the image processing method provided in any possible implementation manner of the aspect or the first aspect.
  • the camera module includes a wide-angle camera, a main camera, and a telephoto camera; the wide-angle camera is used to obtain the first image after the processor obtains a camera instruction; The second image is obtained after the processor obtains the photographing instruction, or; the main camera is used to obtain the first image after the processor obtains the photographing instruction; the telephoto camera is used to obtain the first image after the processor obtains the photographing instruction second image.
  • a chip including: a processor, configured to call and run a computer program from a memory, so that a device installed with the chip executes the chip as provided in the first aspect or any possible implementation manner of the first aspect. The steps of processing in the image processing method.
  • a computer-readable storage medium stores a computer program.
  • the computer program includes program instructions. Steps of performing processing in the image processing method provided in any possible implementation manner of the aspect.
  • a computer program product in a sixth aspect, includes a computer-readable storage medium storing a computer program, and the computer program enables the computer to execute the image provided in the first aspect or any possible implementation manner of the first aspect The step in the processing method that performs the processing.
  • Fig. 1 is a schematic diagram of an image obtained by using related technologies
  • FIG. 2 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of an image processing method provided in an embodiment of the present application.
  • Fig. 4 is a schematic flow chart of another image processing method provided by the embodiment of the present application.
  • FIG. 5 is a schematic diagram of image processing by a segmentation model provided in an embodiment of the present application.
  • FIG. 6 is a schematic diagram of image processing when obtaining a mask block provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a display interface for zooming when taking pictures and previewing provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of the process of multi-camera zooming during photo preview provided by the embodiment of the present application.
  • Fig. 9 is a schematic diagram of a hardware system applicable to the device of the present application.
  • Fig. 10 is a schematic diagram of a software system applicable to the device of the present application.
  • FIG. 11 is a schematic structural diagram of an image processing device provided by an embodiment of the present application.
  • FIG. 12 is a schematic structural diagram of a chip provided in the embodiment of the application.
  • a relationship means that there may be three kinds of relationships, for example, A and/or B means: A exists alone, A and B exist simultaneously, and B exists alone.
  • plural refers to two or more than two.
  • first and second are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as “first” and “second” may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, “plurality” means two or more.
  • RGB (red, green, blue) color space refers to a color model related to the structure of the human visual system. According to the structure of the human eye, all colors are seen as different combinations of red, green and blue.
  • a pixel value refers to a set of color components corresponding to each pixel in a color image located in the RGB color space.
  • each pixel corresponds to a group of three primary color components, wherein the three primary color components are red component R, green component G and blue component B respectively.
  • Image registration refers to the matching of geographic coordinates of different images obtained by different imaging methods in the same area. Among them, it includes the processing of three aspects: geometric correction, projection transformation and unified scale.
  • FOV Field of view
  • the camera can be divided into a main camera, a wide-angle camera, and a telephoto camera due to different field of view angles.
  • the field of view of the wide-angle camera is larger than that of the main camera, and the focal length is smaller, which is suitable for close-up shooting; while the field of view of the telephoto camera is smaller than that of the main camera, and the focal length is longer. Suitable for remote shooting.
  • Backlighting is a situation where the subject is just between the light source and the camera. In this state, it is easy to cause insufficient exposure of the subject. Therefore, in general, users should try to avoid shooting objects under backlight conditions.
  • Fig. 1 is an image captured by using related technologies.
  • FIG 1 there are three people in the scene to be shot waiting for the user to take a photo in the sun. Since the sun shines on the face area, and the sunlight is very strong, the face area produces high light reflection, and the face area is high Brightness area.
  • the captured image loses the details of the face area, resulting in poor image quality and the content of the face area cannot be seen clearly, which affects the user experience.
  • the embodiment of the present application provides an image processing method, by collecting the first image and the second image with different resolutions, using the content corresponding to the high-brightness area in the clearer second image, and the low-definition first image
  • the high-brightness area in the first image is fused, so that the missing details in the high-brightness area in the first image can be recovered, and then a higher-quality captured image can be obtained through multiple fusions to improve user experience.
  • Fig. 2 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • the image processing method provided in this application can be applied to restore the details of the high-brightness area in the image.
  • GUI graphical user interface
  • the preview interface may include a viewfinder window 21 .
  • the preview image can be displayed in the viewfinder window 21 in real time.
  • the preview interface may also include a variety of shooting mode options and a first control, that is, the shooting key 11 .
  • the multiple shooting mode options include, for example: a shooting mode, a video recording mode, etc., and the shooting key 11 is used to indicate that the current shooting mode is a shooting mode, a video recording mode or other modes. Among them, when the camera application is opened, it is generally in the camera mode by default.
  • the electronic device when the electronic device starts the camera application, the electronic device runs the program corresponding to the image processing method, and acquires and stores the captured image in response to the user's click operation on the shooting key 11 .
  • the image processing method of the present application can detect the highlighted face area, and then restore the details of the face area to obtain a high-quality captured image.
  • FIG. 3 is a schematic flowchart of an image processing method provided by an embodiment of the present application. As shown in FIG. 3 , the image processing method includes the following S10-S60.
  • the electronic device starts the camera, and displays a preview interface as shown in (b) in FIG.
  • the first image and the second image are images captured for the same scene to be captured.
  • the definition of the first image is lower than that of the second image, and the first image includes a first area, where the first area is an area in the first image whose resolution is less than a preset threshold.
  • the preset threshold can be set and modified according to needs, which is not limited in this embodiment of the present application.
  • both the first image and the second image are Bayer format images, and may also be referred to as images in the RAW domain.
  • the first area is used to represent an area in the first image that is unclear and lacks details.
  • the first area may refer to a high-brightness area where details are missing due to strong illumination when the first image is acquired, or may refer to a key area where details are missing when the first image is acquired, for example, a human face, Human body, facial features, etc.
  • the mask block refers to a mask image corresponding to the first region in the first image.
  • the processing of the first region in the first image where details need to be restored is controlled by replacing or merging the first region in the first image where details are missing.
  • the definition of the second image is higher than that of the first image, the definition of the overall image can be improved after the fusion of the first image and the second image, and a first fused image with higher definition can be obtained.
  • the first area may refer to the face areas where the three colleagues are illuminated by strong light so that facial features cannot be seen clearly, and the generated mask block corresponds to the first area , used to represent the face area.
  • the first image block is the human face area determined from the first image
  • the second image block is the human face area determined from the second image.
  • the definition of the first image is lower than that of the second image
  • the definition of the first image block is also lower than that of the second image block, and even there is no detail in the first image block,
  • the first image block that is not clear and lacks details is fused with the second image block that is clear and rich in details, and a fused image block with higher definition can be obtained.
  • the overall definition of the first fused image is improved relative to the first image, the fused image block is partially improved relative to the first image block in the first image, and the first fused image By merging with the fused image block, the part in the first fused image can be further repaired to obtain a third image with higher definition.
  • An embodiment of the present application provides an image processing method, by determining the mask block corresponding to the first area missing details from the low-resolution first image, and then obtaining the first mask block corresponding to the mask block from the first image.
  • the first fused image obtained by fusing the first image and the second image is further fused with the fused image block to restore missing details and obtain a high-definition third image.
  • FIG. 4 is a schematic flowchart of another image processing method provided by an embodiment of the present application.
  • the image processing method 10 includes: S110 to S190.
  • the first image and the second image are images captured for the same scene to be captured.
  • the resolution of the first image is lower than that of the second image.
  • first image and the second image are images captured by the electronic device through a camera, or the first image and the second image may also be images obtained from inside the electronic device, for example, images stored in the electronic device, or , the image obtained by the electronic device from the cloud. Wherein, both the first image and the second image are Bayer format images.
  • the corresponding low-resolution image of the two images is called the first image; and the corresponding high-resolution image is called the second image. Since the definition is relative, the first image and the second image are also relative.
  • image processing method provided by the embodiment of the present application is used to perform image processing on image a and image b
  • image a is the first image
  • image b That is the second image
  • image b is the first image
  • image c is second image
  • the first image is an image collected by a wide-angle camera
  • the second image is an image collected by a telephoto camera
  • the wide-angle camera and the telephoto camera collect images at the same time
  • the first image is an image collected by a wide-angle camera
  • the second image is an image collected by the ultra-wide-angle camera
  • the wide-angle camera and the ultra-wide-angle camera collect images at the same time.
  • the first image may be an image with a region of missing details, and the missing details in the first image may be restored through the image processing method of the embodiment of the present application.
  • the segmentation model is used to segment the first region in the first image, and generate a mask block corresponding to the first region.
  • the first area is used to represent an area in the first image whose sharpness is less than a preset threshold, that is, an area lacking certain details.
  • the segmentation model may be: a fully convolutional neural network (fully convolutional networks, FCN) and the like.
  • the segmentation model may segment the first image to obtain a plurality of segmented image regions, where the plurality of image regions include some regions containing details, and may also include some regions lacking details.
  • the segmentation model can segment the one or more regions with missing details, and generate corresponding one or more mask blocks.
  • the segmentation model will not segment out areas with missing details, let alone generate corresponding mask blocks.
  • the first area may refer to a high-brightness area where details are missing due to strong illumination when the first image is acquired, for example, in an HDR scene, or may also refer to a key area where details are missing when the first image is acquired, For example, human face, human body, facial features, etc.
  • the number of first regions is the same as the number of mask blocks, and the range of viewing angles of the mask blocks corresponding to each first region is the same as the range of viewing angles corresponding to the first region.
  • FIG. 5 shows a schematic diagram of processing an image by a segmentation model provided in an embodiment of the present application.
  • the first image is input into the segmentation model. Since the first image contains three face areas that are illuminated by strong light and lack details, the segmentation model can segment three first areas and generate corresponding The 3 mask blocks. For example, the three mask blocks corresponding to the face area in Fig. 1 .
  • the pixel value of each pixel corresponding to the mask block is 0.
  • the registration may be global registration.
  • Global registration is used to register all the content in multiple images, that is to say, here you can register all the content in the first image and the second image, so that the first image and the second image are in In the subsequent fusion, it can correspond more accurately.
  • the registration may include global registration and local registration.
  • Local registration is used to refer to the registration of local content in multiple images.
  • the first region is not segmented from the first image by using the segmentation model, some other regions can be segmented, for example, the human body region and the background region other than the human body region, thus, the first The body region in the image is locally registered with the body region in the second image without registering the background region in the first image with the background region in the second image.
  • the global registration can be performed first, and then the local registration can be performed, or the local registration can be performed first, and then the global registration can be performed, and the registration order can be set and adjusted as required. No restrictions are imposed.
  • the first fusion model can fuse images with different resolutions.
  • the first fusion model may be a VGG net model.
  • the field angle range corresponding to the second image in the first image can be improved The sharpness of the content in the image, so as to obtain the first fused image with higher definition.
  • the field angle range of the first fused image is the same as the field angle range of the first image.
  • first image and the second image may not be registered, and the acquired first image and the second image may be fused using the first fusion model to obtain the first fused image.
  • one first image block corresponding to the mask block can be determined from the first image according to the mask block.
  • the field angle ranges of the first image block and the mask block are the same.
  • a corresponding second image block can be determined from the second image.
  • the field angle range of the second image block is the same as that of the mask block.
  • each mask block in the plurality of mask blocks Corresponding to one first image block, that is to say, the same number of multiple first image blocks can be determined from the first image, and the first image block corresponds to the mask block one by one.
  • a second image block corresponding to each mask block in the plurality of mask blocks can also be determined from the second image, that is, it can be obtained from the second image
  • a plurality of second image blocks of the same number are determined, and the second image blocks are in one-to-one correspondence with the mask block and the first image block.
  • registering the first image block and the second image block refers to registering the first image block and the second image block in each group of image blocks.
  • the registration may be global registration.
  • the first image block and the second image block in each group of image blocks are globally registered. Here, it refers to registering all contents of the first image block and all contents of the second image block in each group of image blocks.
  • the registration may include global registration and local registration.
  • the first image block and the second image block in each group of image blocks are firstly registered globally, and then locally registered.
  • local registration refers to registering the local content of the first image block in each group of image blocks with the local content of the second image block.
  • both the first image block and the second image block include a human face
  • the corresponding regions of the eyes in the human face in the first image block and the second image block are respectively registered
  • the eyes in the human face are The mouth is registered in the regions corresponding to the first image block and the second image block respectively.
  • the present application extracts the first image block from the first image, extracts the second image block from the second image, and then extracts the first image block Global registration is performed with the second image block, so that the background area is isolated without affecting the surrounding background area.
  • local registration may continue to be performed on the first image block and the second image block, so as to improve the registration accuracy, and obtain the first image block and the second image block with higher registration accuracy.
  • first image block and the second image block after registration still have different resolutions
  • the second fusion model can fuse image blocks with different resolutions.
  • the definition of the second image is higher than that of the first image
  • the definition of the second image block is higher than that of the first image block, thus, the registered first image block and the second image After the blocks are fused, a fused image block with higher definition can be obtained.
  • the field angle range of the fused image block is the same as the field angle ranges of the first image block and the second image block.
  • the second fusion model is a pre-trained fusion model.
  • the training image set may include an original image and a manually marked mask block, the mask block is used to identify a first region of missing details in the original image.
  • original images refer to images in various HDR scenarios.
  • On each original image there are 1 or more mask blocks that indicate high-brightness regions (ie, the first region where details are missing) are manually marked.
  • the second fusion model is trained from the first fusion model.
  • the second fusion model is trained by adding random highlight noise to the first fusion model.
  • the second image block with higher definition is larger than that of the first image block.
  • the proportion of the weight is larger, so that the fused image block obtained by fusion can obtain more details from the second image block.
  • the third blending model may be a Laplacian blending model.
  • the Laplacian fusion model can first decompose the first fused image and the fused image block into different spatial frequency bands, and then perform fusion on each spatial frequency band layer, thus , through the frequency division processing, the fusion of the first fused image and the fused image block can be made more natural, the joint is more delicate, and the obtained third image has higher quality.
  • FIG. 6 shows a schematic diagram of processing an image when obtaining a mask block according to an embodiment of the present application.
  • the first image is input into the segmentation model. Since the first image contains three face areas that are illuminated by strong light and lack details, the segmentation model can segment three first areas and generate corresponding The 3 mask blocks.
  • the first image and the second image are registered, and the registered first image and the second image are fused using the first fusion model to obtain the first fusion image.
  • the corresponding 3 first image blocks in the first image are obtained, and the corresponding 3 second image blocks in the second image are obtained, and then, the first image corresponding to the same mask block
  • the block and the second image block are registered and fused by using the second fusion model to obtain a fused image block, thus, three fused image blocks can be obtained.
  • the first fused image and the 3 fused image blocks are fused using the third fused model to obtain the third image.
  • the mask block when the mask block is not obtained from the first image by using the segmentation model, only the first fusion model is used to register and fuse the first image and the second image, and the obtained first The fused images are taken as captured images.
  • the segmentation model when the segmentation model is used to obtain the mask block from the first image, it means that there are areas with missing details in the first image. At this time, the first image and the second image are first fused to obtain a large-scale definition improvement.
  • the first image block is obtained from the first image
  • the second image block is obtained from the second image
  • the first image block and the second image block are registered and Fusion, thereby obtaining a fused image block that effectively restores clarity and details, and then further fusing the first fused image with the fused image block to repair missing details and obtain a high-definition, high-quality captured image.
  • the image processing method of the embodiment of the present application has been described in detail above with reference to FIGS. 2 to 6.
  • the first image and the second image are captured by two cameras.
  • current electronic devices usually include 3 or more The camera, therefore, needs to trigger two different cameras at different focal lengths to acquire the first image and the second image.
  • the zoom factor range corresponding to the electronic device is set to [0.4, 100].
  • the zoom factor range is divided into three zoom factor ranges, and the three zoom factor ranges are respectively the first zoom factor range, the second zoom factor range, and the third zoom factor range, and the zoom factors included in the three zoom factor ranges are sequentially increase.
  • the first zoom factor range F1 is [0.4, 0.9)
  • the second zoom factor range F2 is [0.9, 3.5)
  • the third zoom factor range F3 is [3.5, 100]. It should be understood that, here, each number is only for illustration, which can be set and changed as required, and is not limited in this embodiment of the present application.
  • the applicable zoom factor range of the wide-angle camera itself is [0.4,1]
  • the applicable zoom factor range of the main camera itself is [0.6,3.5]
  • the applicable zoom factor range of the telephoto camera itself is [2.0,100] ].
  • the target camera corresponding to the first zoom range is set as the wide-angle camera
  • the target camera corresponding to the second zoom range is the main camera
  • the target camera corresponding to the third zoom range is set as the telephoto camera.
  • FIG. 7 is a schematic diagram of an interface for zooming during photo preview provided by an embodiment of the present application.
  • FIG. 8 shows a schematic diagram of a process of multi-camera zooming during photo preview provided by an embodiment of the present application.
  • the electronic device 100 displays a preview interface as shown in (a) in FIG. 7 .
  • the shooting key 11 indicates that the current shooting mode is the shooting mode.
  • the preview interface also includes a viewfinder window 21, and the viewfinder window 21 can be used to display a preview image before taking pictures in real time.
  • a zoom option 22 is also displayed on the preview screen. The user can select the zoom factor of the current photo taking in the zoom option 22, for example, 0.4 times, 2 times or 50 times.
  • the preview image in response to the user's zoom operation, can be enlarged or reduced according to the currently selected zoom factor, and as the zoom factor is enlarged or reduced, the preview image in the viewfinder window 21 also becomes larger or smaller. zoom out.
  • two different cameras are invoked to acquire captured images by using the image processing method provided by the embodiment of the present application.
  • the wide-angle camera corresponding to the first zoom multiple range is in the foreground sending display state, and the acquired image is sent to the display screen show.
  • the wide-angle camera When zooming to the first zoom switching point (for example, 0.6X), the wide-angle camera continues to be in the foreground display state, and the main camera corresponding to the second zoom range F2 starts to enter the background operation state.
  • the first zoom switching point for example, 0.6X
  • the wide-angle camera Since the wide-angle camera has a larger field of view and lower definition than the main camera, therefore, within the zoom range F11 of [0.6,0.9], in response to the user's operation on the shooting key 11, the wide-angle camera captures The image is used as the first image, and the image obtained by the main camera is used as the second image. Then, based on the first image obtained by the wide-angle camera and the second image obtained by the main camera, using the image processing method provided in the embodiment of the present application, a clear Capture images with high resolution and rich details.
  • the wide-angle camera When zooming to 0.9X, the wide-angle camera is turned off, and the main camera switches to the foreground sending display state, that is, the main camera sends the acquired image to the display screen for display.
  • the wide-angle camera continues to be in the foreground display state, and the telephoto camera corresponding to the third zoom multiple range F3 starts to enter the background operation state.
  • the image captured by the main camera has low resolution and a large field of view. Therefore, within the zoom range F21 of [2.0, 3.5], in response to the user's operation on the shooting key 11, the main camera The acquired image is used as the first image, and the image acquired by the telephoto camera is used as the second image. Then, based on the first image acquired by the main camera and the second image acquired by the telephoto camera, the image processing method provided by the embodiment of the present application is used. , to obtain images with high definition and rich details.
  • the main camera When zooming to 3.5X, the main camera is turned off, and the telephoto camera switches to the foreground sending display state, that is, the telephoto camera sends the acquired image to the display screen for display.
  • the image processing method provided in the embodiment of the present application may be applicable to various electronic devices, and correspondingly, the image processing apparatus provided in the embodiment of the present application may be electronic devices in various forms.
  • the electronic device may be various camera devices such as SLR cameras and card players, mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (augmented reality, AR)/virtual reality (virtual reality) reality, VR) equipment, notebook computer, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook, personal digital assistant (personal digital assistant, PDA), etc., or other equipment or devices capable of image processing,
  • camera devices such as SLR cameras and card players, mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (augmented reality, AR)/virtual reality (virtual reality) reality, VR) equipment, notebook computer, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook, personal digital assistant (personal digital assistant, PDA), etc., or other equipment or devices capable of image processing
  • the embodiment of the present application does not set any limitation on the specific type of the electronic device.
  • FIG. 9 shows a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.
  • the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like.
  • SIM subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.
  • the structure shown in FIG. 1 does not constitute a specific limitation on the electronic device 100 .
  • the electronic device 100 may include more or fewer components than those shown in FIG. 1 , or the electronic device 100 may include a combination of some of the components shown in FIG. 1 , or , the electronic device 100 may include subcomponents of some of the components shown in FIG. 1 .
  • the components shown in FIG. 1 can be realized in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • application processor application processor, AP
  • modem processor graphics processing unit
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit
  • the controller may be the nerve center and command center of the electronic device 100 .
  • the controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
  • a memory may also be provided in the processor 110 for storing instructions and data.
  • the memory in processor 110 is a cache memory.
  • the memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.
  • the processor 110 may run the software code of the image processing method provided in the embodiment of the present application to capture an image with higher definition.
  • processor 110 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input and output
  • subscriber identity module subscriber identity module
  • SIM subscriber identity module
  • USB universal serial bus
  • the MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 .
  • MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc.
  • the processor 110 communicates with the camera 193 through the CSI interface to realize the shooting function of the electronic device 100 .
  • the processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the electronic device 100 .
  • the GPIO interface can be configured by software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 110 with the camera 193 , the display screen 194 , the wireless communication module 160 , the audio module 170 , the sensor module 180 and so on.
  • the GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.
  • the USB interface 130 is an interface conforming to the USB standard specification, specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like.
  • the USB interface 130 can be used to connect a charger to charge the electronic device 100 , and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.
  • the interface connection relationship between the modules shown in the embodiment of the present application is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 .
  • the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.
  • the charging management module 140 is configured to receive a charging input from a charger.
  • the charger may be a wireless charger or a wired charger.
  • the charging management module 140 can receive the current of the wired charger through the USB interface 130 .
  • the charging management module 140 can receive electromagnetic waves through the wireless charging coil of the electronic device 100 (the current path is shown as a dotted line). While the charging management module 140 is charging the battery 142 , it can also supply power to the electronic device 100 through the power management module 141 .
  • the power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 .
  • the power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the display screen 194 , the camera 193 , and the wireless communication module 160 .
  • the wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas.
  • Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
  • the antenna may be used in conjunction with a tuning switch.
  • the mobile communication module 150 may provide a wireless communication solution applied to the electronic device 100, such as at least one of the following solutions: a second generation (2th generation, 2G) mobile communication solution, a third generation (3th generation, 3G) Mobile communication solutions, fourth generation (4th generation, 5G) mobile communication solutions, fifth generation (5th generation, 5G), sixth generation (6th generation, 6G) mobile communication solutions.
  • the mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like.
  • the mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation.
  • the mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves through the antenna 1 for radiation.
  • at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 .
  • at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.
  • the wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. applied on the electronic device 100.
  • System global navigation satellite system, GNSS
  • frequency modulation frequency modulation, FM
  • near field communication technology near field communication, NFC
  • infrared technology infrared, IR
  • the wireless communication module 160 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 .
  • the wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.
  • the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc.
  • GSM global system for mobile communications
  • GPRS general packet radio service
  • code division multiple access code division multiple access
  • CDMA broadband Code division multiple access
  • WCDMA wideband code division multiple access
  • time division code division multiple access time-division code division multiple access
  • TD-SCDMA time-division code division multiple access
  • the GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou navigation satellite system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • Beidou navigation satellite system beidou navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite based augmentation systems
  • the electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor.
  • the GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 194 is used to display images, videos and the like.
  • the display screen 194 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc.
  • the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.
  • Camera 193 is used to capture images or videos. It can be triggered by an application command to realize the camera function, such as capturing images of any scene.
  • a camera may include components such as an imaging lens, an optical filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor.
  • the image sensor is mainly used for converging and imaging the light emitted or reflected by all objects in the camera perspective (also called the scene to be shot, the target scene, or the scene image that the user expects to shoot); the filter is mainly used to It is used to filter out redundant light waves (such as light waves other than visible light, such as infrared) in the light; the image sensor is mainly used to perform photoelectric conversion on the received light signal, convert it into an electrical signal, and input it into the processor 130 for subsequent processing .
  • the camera 193 may be located at the front of the electronic device 100, or at the back of the electronic device 100, and the specific number and arrangement of the cameras may be set according to requirements, which are not limited in this application.
  • the electronic device 100 includes a front camera and a rear camera.
  • a front camera or a rear camera may include one or more cameras.
  • the camera is arranged on an external accessory of the electronic device 100, the external accessory is rotatably connected to the frame of the mobile phone, and the angle formed between the external accessory and the display screen 194 of the electronic device 100 is 0-360 degrees any angle between.
  • the external accessory drives the camera to rotate to a position facing the user.
  • the mobile phone has multiple cameras, only some of the cameras may be set on the external accessories, and the rest of the cameras may be set on the electronic device 100 body, which is not limited in this embodiment of the present application.
  • the internal memory 121 may be used to store computer-executable program codes including instructions.
  • the internal memory 121 may include an area for storing programs and an area for storing data.
  • the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like.
  • the storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like.
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
  • the processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.
  • the internal memory 121 can also store the software code of the image processing method provided by the embodiment of the present application.
  • the processor 110 runs the software code, it executes the process steps of the image processing method to obtain an image with higher definition.
  • the internal memory 121 can also store captured images.
  • the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving files such as music in an external memory card.
  • the software code of the image processing method provided in the embodiment of the present application can also be stored in an external memory, and the processor 110 can run the software code through the external memory interface 120 to execute the process steps of the image processing method to obtain a high-definition image.
  • Image Images captured by the electronic device 100 may also be stored in an external memory.
  • the user can designate whether to store the image in the internal memory 121 or the external memory.
  • the electronic device 100 when the electronic device 100 is currently connected to the external memory, if the electronic device 100 captures one frame of image, a prompt message may pop up to remind the user whether to store the image in the external memory or the internal memory; of course, there may be other specified ways , the embodiment of the present application does not impose any limitation on this; alternatively, when the electronic device 100 detects that the memory capacity of the internal memory 121 is less than a preset amount, it may automatically store the image in the external memory.
  • the electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.
  • the pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal.
  • pressure sensor 180A may be disposed on display screen 194 .
  • the gyro sensor 180B can be used to determine the motion posture of the electronic device 100 .
  • the angular velocity of the electronic device 100 around three axes ie, x, y and z axes
  • the gyro sensor 180B can be used for image stabilization.
  • the air pressure sensor 180C is used to measure air pressure.
  • the electronic device 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
  • the magnetic sensor 180D includes a Hall sensor.
  • the electronic device 100 may use the magnetic sensor 180D to detect the opening and closing of the flip leather case.
  • the electronic device 100 when the electronic device 100 is a clamshell machine, the electronic device 100 can detect opening and closing of the clamshell according to the magnetic sensor 180D.
  • features such as automatic unlocking of the flip cover are set.
  • the acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.
  • the distance sensor 180F is used to measure the distance.
  • the electronic device 100 may measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 may use the distance sensor 180F for distance measurement to achieve fast focusing.
  • Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes.
  • the light emitting diodes may be infrared light emitting diodes.
  • the electronic device 100 emits infrared light through the light emitting diode.
  • Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it may be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 .
  • the electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to make a call, so as to automatically turn off the screen to save power.
  • the proximity light sensor 180G can also be used in leather case mode, automatic unlock and lock screen in pocket mode.
  • the ambient light sensor 180L is used for sensing ambient light brightness.
  • the electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness.
  • the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.
  • the ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket, so as to prevent accidental touch.
  • the fingerprint sensor 180H is used to collect fingerprints.
  • the electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access to application locks, take pictures with fingerprints, answer incoming calls with fingerprints, and the like.
  • the temperature sensor 180J is used to detect temperature.
  • the electronic device 100 uses the temperature detected by the temperature sensor 180J to implement a temperature treatment strategy. For example, when the temperature reported by the temperature sensor 180J exceeds the threshold, the electronic device 100 may reduce the performance of the processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection.
  • the electronic device 100 when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by the low temperature.
  • the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
  • the touch sensor 180K is also called “touch device”.
  • the touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”.
  • the touch sensor 180K is used to detect a touch operation on or near it.
  • the touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.
  • Visual output related to the touch operation can be provided through the display screen 194 .
  • the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the position of the display screen 194 .
  • the bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180M can also contact the human pulse and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone, combined into a bone conduction earphone.
  • the audio module 170 can analyze the voice signal based on the vibration signal of the vibrating bone mass of the voice part acquired by the bone conduction sensor 180M, so as to realize the voice function.
  • the application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180M, so as to realize the heart rate detection function.
  • the keys 190 include a power key, a volume key and the like.
  • the key 190 may be a mechanical key. It can also be a touch button.
  • the electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 .
  • the motor 191 can generate a vibrating reminder.
  • the motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.
  • touch operations applied to different applications may correspond to different vibration feedback effects.
  • the indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.
  • the SIM card interface 195 is used for connecting a SIM card.
  • the SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 .
  • the hardware system of the electronic device 100 is described in detail above, and the software system of the electronic device 100 is introduced below.
  • the software system may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.
  • the embodiment of the present application uses a layered architecture as an example to exemplarily describe the software system of the electronic device 100 .
  • a software system adopting a layered architecture is divided into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces.
  • the software system can be divided into five layers, which are application layer 210 , application framework layer 220 , hardware abstraction layer 230 , driver layer 240 and hardware layer 250 from top to bottom.
  • the application layer 210 may include application programs such as camera and gallery, and may also include application programs such as calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.
  • the application framework layer 220 provides application program access interfaces and programming frameworks for the applications of the application layer 210 .
  • the application framework layer includes a camera access interface, and the camera access interface is used to provide camera shooting services through camera management and camera equipment.
  • Camera management in the application framework layer is used to manage cameras. Camera management can obtain camera parameters, such as judging the working status of the camera.
  • the camera device in the application framework layer is used to provide a data access interface between different camera devices and camera management.
  • the hardware abstraction layer 230 is used to abstract hardware.
  • the hardware abstraction layer can include the camera hardware abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can include camera device 1, camera device 2, etc.; the camera hardware abstraction layer can be connected with the camera algorithm library, and the camera hardware abstraction layer Algorithms in the camera algorithm library can be called.
  • the driver layer 240 is used to provide drivers for different hardware devices.
  • the driver layer may include camera drivers; digital signal processor drivers and graphics processor drivers.
  • the hardware layer 250 may include sensors, image signal processors, digital signal processors, graphics processors, and other hardware devices.
  • the sensor may include sensor 1, sensor 2, etc., and may also include a depth sensor (time of flight, TOF) and a multispectral sensor.
  • TOF time of flight
  • the camera hardware abstraction layer judges that the current zoom factor is between the range of [0.6, 0.9] zoom factor, so it can send instructions to the camera device driver to call the wide-angle camera and the main camera, and the camera algorithm library Start to load the algorithm in the network model used in the embodiment of this application.
  • sensor 1 in the wide-angle camera is invoked to obtain the first image
  • sensor 2 in the main camera captures the second image
  • the first image and the second image are sent to image signal processing for processing
  • Preliminary processing such as registration
  • the camera device driver returns to the hardware abstraction layer, and then uses the algorithm in the loaded camera algorithm library for processing, such as using the segmentation model, the first fusion model, the second fusion model and the third fusion model according to
  • the relevant processing steps provided in the embodiment of the present application are processed to obtain the captured image.
  • the segmentation model, the first fusion model, the second fusion model and the third fusion model can be processed by calling the digital signal processor through the driver of the digital signal processor, and calling the graphics processor through the driver of the graphics processor.
  • the captured images are sent back to the camera application via the camera hardware abstraction layer and the camera access interface for display and storage.
  • FIG. 11 is a schematic structural diagram of an image processing device provided by an embodiment of the present application. As shown in FIG. 11 , the image processing device 300 includes an acquisition module 310 and a processing module 320 .
  • the image processing device 300 can perform the following schemes:
  • the acquisition module 310 is configured to acquire a first image and a second image, the resolution of the first image is lower than that of the second image, the first image includes a first area, and the first area is that the resolution of the first image is smaller than a preset Threshold area.
  • a processing module 320 configured to input the first image into a segmentation model to determine whether a mask block is obtained, wherein the segmentation model is used to segment the first region in the first image and generate a mask block corresponding to the first region , the first region is used to represent the region of missing detail in the first image.
  • the processing module 320 is further configured to fuse the first image and the second image using the first fusion model to obtain a first fusion image.
  • the processing module 320 is further configured to determine the first image block in the first image according to the mask block, determine the second image block in the second image, and combine the first image block and the second image block The blocks are fused using the second fusion model to obtain fused image blocks.
  • the processing module 320 is further configured to fuse the first fusion image and the fusion image block by using the third fusion model to obtain a captured image.
  • the processing module 320 will fuse the first image and the second image using the first fusion model to obtain the first fusion image.
  • the processing module 320 is further configured to register the first image and the second image.
  • the processing module 320 is further configured to register the first image block and the second image block.
  • Registration includes: global registration and/or local registration. Global registration is used to register all content in multiple images, and local registration is used to register local content in multiple images.
  • the processing module 320 is further configured to use the training image set and add random highlight noise to train the first fusion model to obtain the second fusion model, wherein the training image set includes the original image, the original Images are annotated with mask blocks.
  • the third fusion model is a Laplace fusion model.
  • module may be implemented in the form of software and/or hardware, which is not specifically limited.
  • a “module” may be a software program, a hardware circuit or a combination of both to realize the above functions.
  • the hardware circuitry may include application specific integrated circuits (ASICs), electronic circuits, processors (such as shared processors, dedicated processors, or group processors) for executing one or more software or firmware programs. etc.) and memory, incorporating logic, and/or other suitable components to support the described functionality.
  • ASICs application specific integrated circuits
  • processors such as shared processors, dedicated processors, or group processors for executing one or more software or firmware programs. etc.
  • memory incorporating logic, and/or other suitable components to support the described functionality.
  • modules of each example described in the embodiments of the present application can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.
  • the embodiment of the present application also provides another electronic device, including a camera module, a processor, and a memory.
  • the camera module is used to acquire a first image and a second image, the first image and the second image are images taken for the same scene to be photographed, and the definition of the first image is lower than that of the second image.
  • Memory which stores computer programs that run on the processor.
  • a processor configured to execute the processing steps in the above-mentioned image processing method.
  • the camera module includes a wide-angle camera, a main camera and a telephoto camera; the wide-angle camera is used to obtain the first image after the processor obtains the camera instruction; the main camera is used to obtain the first image after the processor obtains the camera instruction , to acquire the second image; or, the main camera is used to acquire the first image after the processor acquires the photographing instruction; the telephoto camera is configured to acquire the second image after the processor acquires the photographing instruction.
  • the image is obtained by the image processor in the color camera and the black and white camera.
  • the image sensor may be, for example, a charge-coupled device (charge-coupled device, CCD), a complementary metal oxide semiconductor (complementary metal oxide semiconductor, CMOS) and the like.
  • the embodiment of the present application also provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium; when the computer-readable storage medium is run on an image processing device, the image processing device executes the following steps: The method shown in Figure 3 and/or Figure 4.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server, or data center Transmission to another website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.).
  • wired such as coaxial cable, optical fiber, digital subscriber line (DSL)
  • wireless such as infrared, wireless, microwave, etc.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, or may be a data storage device including one or more servers, data centers, etc. that can be integrated with the medium.
  • the available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium, or a semiconductor medium (for example, a solid state disk (solid state disk, SSD)) and the like.
  • the embodiment of the present application also provides a computer program product including computer instructions, which, when run on an image processing device, enables the image processing device to execute the method shown in FIG. 3 and/or FIG. 4 .
  • FIG. 12 is a schematic structural diagram of a chip provided by an embodiment of the present application.
  • the chip shown in FIG. 12 may be a general-purpose processor or a special-purpose processor.
  • the chip includes a processor 401 .
  • the processor 401 is configured to support the image processing apparatus to execute the technical solutions shown in FIG. 3 and/or FIG. 4 .
  • the chip further includes a transceiver 402, and the transceiver 402 is configured to be controlled by the processor 401, and configured to support the communication device to execute the technical solution shown in FIG. 3 and/or FIG. 4 .
  • the chip shown in FIG. 12 may further include: a storage medium 403 .
  • the chip shown in Figure 12 can be implemented using the following circuits or devices: one or more field programmable gate arrays (field programmable gate array, FPGA), programmable logic device (programmable logic device, PLD) , controllers, state machines, gate logic, discrete hardware components, any other suitable circuitry, or any combination of circuitry capable of performing the various functions described throughout this application.
  • field programmable gate array field programmable gate array, FPGA
  • programmable logic device programmable logic device
  • controllers state machines, gate logic, discrete hardware components, any other suitable circuitry, or any combination of circuitry capable of performing the various functions described throughout this application.
  • the electronic equipment, image processing device, computer storage medium, computer program product, and chip provided by the above-mentioned embodiments of the present application are all used to execute the method provided above. Therefore, the beneficial effects that it can achieve can refer to the above-mentioned The beneficial effects corresponding to the method will not be repeated here.
  • sequence numbers of the above processes do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.
  • presetting and predefining can be realized by pre-saving corresponding codes, tables or other methods that can be used to indicate related information in devices (for example, including electronic devices) , the present application does not limit its specific implementation.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)

Abstract

La présente invention, qui relève du domaine du traitement d'images, concerne un procédé de traitement d'images et son dispositif associé. Le procédé comprend : la collecte, par une première caméra, d'une première image et la collecte, par une seconde caméra, d'une deuxième image (S10) ; l'obtention d'un bloc de masquage selon la première image ; la fusion de la première image et de la deuxième image de façon à obtenir une première image fusionnée (S30) ; selon le bloc de masquage, la détermination d'un premier bloc d'image dans la première image et la détermination d'un deuxième bloc d'image dans la deuxième image (S40) ; la fusion du premier bloc d'image et du deuxième bloc d'image, de façon à obtenir un bloc d'image fusionné (S50) ; et la fusion de la première image fusionnée et du bloc d'image fusionné de façon à obtenir une troisième image (S60). Au moyen de la fusion du contenu de la deuxième image plus nette et de la même surface dans la première image de faible définition, des détails manquants sont récupérés, et une fusion est réalisée de multiples fois de façon à obtenir une image de haute définition.
PCT/CN2022/091225 2021-08-12 2022-05-06 Procédé de traitement d'images et son dispositif associé WO2023015981A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110923642.9A CN114092364B (zh) 2021-08-12 2021-08-12 图像处理方法及其相关设备
CN202110923642.9 2021-08-12

Publications (1)

Publication Number Publication Date
WO2023015981A1 true WO2023015981A1 (fr) 2023-02-16

Family

ID=80296087

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/091225 WO2023015981A1 (fr) 2021-08-12 2022-05-06 Procédé de traitement d'images et son dispositif associé

Country Status (2)

Country Link
CN (1) CN114092364B (fr)
WO (1) WO2023015981A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117132629A (zh) * 2023-02-17 2023-11-28 荣耀终端有限公司 图像处理方法和电子设备

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114092364B (zh) * 2021-08-12 2023-10-03 荣耀终端有限公司 图像处理方法及其相关设备
CN116723394B (zh) * 2022-02-28 2024-05-10 荣耀终端有限公司 多摄策略调度方法及其相关设备
CN114782296B (zh) * 2022-04-08 2023-06-09 荣耀终端有限公司 图像融合方法、装置及存储介质
CN116051386B (zh) * 2022-05-30 2023-10-20 荣耀终端有限公司 图像处理方法及其相关设备
CN115631098B (zh) * 2022-06-16 2023-10-03 荣耀终端有限公司 去反射方法及装置
CN116245741B (zh) * 2022-06-28 2023-11-17 荣耀终端有限公司 图像处理方法及其相关设备
CN116051368B (zh) * 2022-06-29 2023-10-20 荣耀终端有限公司 图像处理方法及其相关设备
CN116801093B (zh) * 2023-08-25 2023-11-28 荣耀终端有限公司 图像处理方法、设备及存储介质
CN117729445A (zh) * 2024-02-07 2024-03-19 荣耀终端有限公司 一种图像处理方法、电子设备以及计算机可读存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107197169A (zh) * 2017-06-22 2017-09-22 维沃移动通信有限公司 一种高动态范围图像拍摄方法及移动终端
CN107277387A (zh) * 2017-07-26 2017-10-20 维沃移动通信有限公司 高动态范围图像拍摄方法、终端及计算机可读存储介质
US10165194B1 (en) * 2016-12-16 2018-12-25 Amazon Technologies, Inc. Multi-sensor camera system
CN112184609A (zh) * 2020-10-10 2021-01-05 展讯通信（上海）有限公司 图像融合方法及装置、存储介质、终端
CN112995544A (zh) * 2019-12-02 2021-06-18 三星电子株式会社 根据单输入生成多曝光帧的系统和方法
CN113099123A (zh) * 2021-04-07 2021-07-09 中煤科工集团重庆研究院有限公司 一种高动态范围视频图像采集方法
CN114092364A (zh) * 2021-08-12 2022-02-25 荣耀终端有限公司 图像处理方法及其相关设备

Family Cites Families (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9779491B2 (en) * 2014-08-15 2017-10-03 Nikon Corporation Algorithm and device for image processing
US9779227B1 (en) * 2014-10-24 2017-10-03 Amazon Technologies, Inc. Security system using keys encoded in holograms
CN108291867B (zh) * 2015-07-01 2021-07-16 堀场仪器株式会社 用于液体中纳米颗粒的显微观察的专用试管组件和方法
US10012580B2 (en) * 2015-10-14 2018-07-03 MANTA Instruments, Inc. Apparatus and method for measurements of growth or dissolution kinetics of colloidal particles
CN107730528A (zh) * 2017-10-28 2018-02-23 天津大学 一种基于grabcut算法的交互式图像分割与融合方法
WO2019097456A1 (fr) * 2017-11-17 2019-05-23 C 3 Limited Système de mesure d'objet
CN108682015B (zh) * 2018-05-28 2021-10-19 安徽科大讯飞医疗信息技术有限公司 一种生物图像中的病灶分割方法、装置、设备及存储介质
KR102192899B1 (ko) * 2018-08-16 2020-12-18 주식회사 날비컴퍼니 이미지에 보케 효과를 적용하는 방법 및 기록매체
CN111340044A (zh) * 2018-12-19 2020-06-26 北京嘀嘀无限科技发展有限公司 图像处理方法、装置、电子设备及存储介质
CN109951633B (zh) * 2019-02-18 2022-01-11 华为技术有限公司 一种拍摄月亮的方法和电子设备
CN110072070B (zh) * 2019-03-18 2021-03-23 华为技术有限公司 一种多路录像方法及设备、介质
CN110430357B (zh) * 2019-03-26 2021-01-29 华为技术有限公司 一种图像拍摄方法与电子设备
CN110163875A (zh) * 2019-05-23 2019-08-23 南京信息工程大学 一种基于调制网络和特征注意金字塔的半监督视频目标分割方法
CN110246141B (zh) * 2019-06-13 2022-10-21 大连海事大学 一种基于联合角点池化的复杂交通场景下车辆图像分割方法
CN112116624A (zh) * 2019-06-21 2020-12-22 华为技术有限公司 一种图像处理方法和电子设备
CN113132620B (zh) * 2019-12-31 2022-10-11 华为技术有限公司 一种图像拍摄方法及相关装置
CN111341419A (zh) * 2020-02-19 2020-06-26 京东方科技集团股份有限公司 医学图像的处理方法、装置、系统、控制系统及存储介质
CN111582093A (zh) * 2020-04-27 2020-08-25 北京工业大学 一种基于计算机视觉和深度学习的高分辨率图像中小目标自动检测方法
CN112001391A (zh) * 2020-05-11 2020-11-27 江苏鲲博智行科技有限公司 一种图像特征融合图像语义分割的方法
CN111612807B (zh) * 2020-05-15 2023-07-25 北京工业大学 一种基于尺度和边缘信息的小目标图像分割方法
CN111709878B (zh) * 2020-06-17 2023-06-23 北京百度网讯科技有限公司 人脸超分辨率实现方法、装置、电子设备及存储介质
CN112116620B (zh) * 2020-09-16 2023-09-22 北京交通大学 一种室内图像语义分割与涂装展示的方法
CN112507777A (zh) * 2020-10-10 2021-03-16 厦门大学 一种基于深度学习的光学遥感图像舰船检测与分割方法
CN112465843A (zh) * 2020-12-22 2021-03-09 深圳市慧鲤科技有限公司 图像分割方法及装置、电子设备和存储介质
CN112598580B (zh) * 2020-12-29 2023-07-25 广州光锥元信息科技有限公司 提升人像照片清晰度的方法及装置
CN112950606B (zh) * 2021-03-15 2023-04-07 重庆邮电大学 一种基于小样本的手机屏幕缺陷分割方法
CN112926556B (zh) * 2021-04-28 2023-05-02 上海大学 一种基于语义分割的航拍输电线路散断股识别方法及系统
CN113111857A (zh) * 2021-05-10 2021-07-13 金华高等研究院 基于多模态信息融合的人体姿态估计方法
CN113239784B (zh) * 2021-05-11 2022-09-30 广西科学院 一种基于空间序列特征学习的行人重识别系统及方法
CN113240679A (zh) * 2021-05-17 2021-08-10 广州华多网络科技有限公司 图像处理方法、装置、计算机设备及存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10165194B1 (en) * 2016-12-16 2018-12-25 Amazon Technologies, Inc. Multi-sensor camera system
CN107197169A (zh) * 2017-06-22 2017-09-22 维沃移动通信有限公司 一种高动态范围图像拍摄方法及移动终端
CN107277387A (zh) * 2017-07-26 2017-10-20 维沃移动通信有限公司 高动态范围图像拍摄方法、终端及计算机可读存储介质
CN112995544A (zh) * 2019-12-02 2021-06-18 三星电子株式会社 根据单输入生成多曝光帧的系统和方法
CN112184609A (zh) * 2020-10-10 2021-01-05 展讯通信（上海）有限公司 图像融合方法及装置、存储介质、终端
CN113099123A (zh) * 2021-04-07 2021-07-09 中煤科工集团重庆研究院有限公司 一种高动态范围视频图像采集方法
CN114092364A (zh) * 2021-08-12 2022-02-25 荣耀终端有限公司 图像处理方法及其相关设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117132629A (zh) * 2023-02-17 2023-11-28 荣耀终端有限公司 图像处理方法和电子设备

Also Published As

Publication number Publication date
CN114092364A (zh) 2022-02-25
CN114092364B (zh) 2023-10-03

Similar Documents

Publication Publication Date Title
WO2023015981A1 (fr) Procédé de traitement d'images et son dispositif associé
WO2022262260A1 (fr) Procédé de photographie et dispositif électronique
WO2020168956A1 (fr) Procédé pour photographier la lune, et dispositif électronique
WO2021052232A1 (fr) Procédé et dispositif de photographie à intervalle de temps
WO2020073959A1 (fr) Procédé de capture d'image et dispositif électronique
CN113810600B (zh) 终端的图像处理方法、装置和终端设备
CN113452898B (zh) 一种拍照方法及装置
WO2023015991A1 (fr) Procédé de photographie, dispositif électronique, et support de stockage lisible par ordinateur
US20240119566A1 (en) Image processing method and apparatus, and electronic device
CN113660408B (zh) 一种视频拍摄防抖方法与装置
WO2023279863A1 (fr) Procédé et appareil de traitement d'image, et dispositif électronique
CN113630558B (zh) 一种摄像曝光方法及电子设备
US20240137659A1 (en) Point light source image detection method and electronic device
WO2022001258A1 (fr) Procédé et appareil d'affichage à écrans multiples, dispositif terminal et support de stockage
CN115526787B (zh) 视频处理方法和装置
WO2024045670A1 (fr) Procédé de génération de vidéo à grande gamme dynamique, et dispositif électronique
WO2023131070A1 (fr) Procédé de gestion de dispositif électronique, dispositif électronique et support de stockage lisible
WO2023226612A1 (fr) Procédé et appareil de détermination de paramètre d'exposition
WO2022267608A1 (fr) Procédé de réglage d'intensité d'exposition et appareil associé
CN113592751A (zh) 图像处理方法、装置和电子设备
WO2023015985A1 (fr) Procédé de traitement d'image et dispositif électronique
WO2022206589A1 (fr) Procédé de traitement d'image et dispositif associé
CN115705663B (zh) 图像处理方法与电子设备
CN115526786B (zh) 图像处理方法及其相关设备
CN116055855B (zh) 图像处理方法及其相关设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22854977

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE