WO2023077939A1

WO2023077939A1 - Camera switching method and apparatus, and electronic device and storage medium

Info

Publication number: WO2023077939A1
Application number: PCT/CN2022/116759
Authority: WO
Inventors: 侯伟龙; 金杰; 李子荣
Original assignee: 荣耀终端有限公司
Priority date: 2021-11-05
Filing date: 2022-09-02
Publication date: 2023-05-11
Also published as: CN116095476B; CN116095476A

Abstract

The present application relates to the technical field of terminals, and provides a camera switching method and apparatus, and an electronic device and a storage medium. The method comprises: performing video frame output according to first raw data of a first camera, and in the video frame output process, if a focusing operation is detected, starting a second camera before a camera switching condition is satisfied; and according to the first raw data and second raw data of the second camera, performing parameter synchronization on the second camera and the first camera, and when the focusing operation satisfies the camera switching condition, performing video frame output according to third raw data, the third raw data being raw video data of the second camera subjected to parameter synchronization. A camera to be started is started in advance, and parameter synchronization is performed after the camera is started, such that a picture effect after switching is close to a picture effect before switching, thereby avoiding the problem of a large video picture difference before and after switching and achieving smooth switching between cameras.

Description

Camera switching method, device, electronic equipment and storage medium

This application claims the priority of the Chinese patent application submitted to the State Intellectual Property Office on November 5, 2021, with the application number 202111310300.6, and the application name is "Camera Switching Method, Device, Electronic Equipment, and Storage Medium", and claims to be filed in The priority of a Chinese patent application filed with the State Intellectual Property Office on March 10, 2022, with application number 202210248963.8 and titled "Camera switching method, electronic device, and storage medium", the entire contents of which are incorporated by reference in this application middle.

technical field

The present application relates to the technical field of terminals, and in particular to a camera switching method, electronic equipment, and a storage medium.

Background technique

With the rapid development of terminal technology, the shooting capabilities of electronic devices such as mobile phones are gradually improving. Some electronic devices are configured with multiple cameras, and different cameras in the multiple cameras have different shooting capabilities. For example, different cameras have different viewing angles, and the electronic device can collect video data with different viewing angles through different cameras.

Usually, multiple cameras include a primary camera and a secondary camera. During video recording through the cameras, if a user's focus operation is detected, the primary camera and the secondary camera may need to be switched. For example, by default, after the camera application is launched, the electronic device shoots video through the main camera. If a focus adjustment operation is detected and the focus adjustment value exceeds the field of view range of the main camera, the electronic device switches from the main camera to the auxiliary camera, and then the electronic device shoots a video through the auxiliary camera.

However, when zooming, in the case of switching from one camera to another, due to the performance difference between the two cameras, there may be a large difference in the video picture before and after the switch.

Contents of the invention

The present application provides a camera switching method, device, electronic equipment and storage medium, which solves the problem in the prior art that there is a large difference in video images before and after switching when switching from one camera to another.

In order to achieve the above object, the application adopts the following technical solutions:

In a first aspect, a camera switching method is provided, the electronic device includes a first camera and a second camera, and the method includes:

performing video frame output according to the first raw data of the first camera;

In the process of video frame output, if a focusing operation is detected, before the camera switching condition is not met, start the second camera;

Synchronize parameters of the second camera with the first camera according to the first raw data and the second raw data of the second camera;

When the focusing operation satisfies the camera switching condition, the video frame output is performed according to the third original data, and the third original data is the original video data of the second camera whose parameters are synchronized.

In this way, by starting the second camera in advance and synchronizing the parameters of the second camera with the first camera after starting, the difference between the video picture of the second camera and the video picture of the first camera after switching can be small, that is, to solve It solves the problem of large difference in video images before and after switching, thus realizing smooth switching between cameras.

As an example of the present application, the parameter synchronization of the second camera and the first camera according to the first raw data and the second raw data of the second camera includes:

determining first imaging information according to the first raw data;

determining second imaging information according to the second raw data;

adjusting the second imaging information according to the first imaging information to synchronize the second imaging information with the first imaging information;

According to the adjusted second camera information, the parameters of the second camera are adjusted, so as to synchronize the parameters of the second camera with the first camera.

In this way, by synchronizing the first camera information with the second camera information, the parameters of the first camera and the second camera are synchronized, so that the difference between the video picture after switching and the video picture before switching is small.

As an example of the present application, the first imaging information includes first exposure data, first autofocus AF, first automatic white balance AWB, and first field of view FOV, and the second imaging information includes second exposure data , 2nd AF, 2nd AWB, 2nd FOV.

In this way, when synchronizing camera parameters, the exposure data, AF, AWB, and FOV parameters are adjusted synchronously, that is, multiple indicators that affect the effect of the video image are considered, so that the visual difference of the video image before and after switching is as much as possible small.

As an example of the present application, the method also includes:

Based on the second raw data, pre-noise reduction processing is performed through the first target model corresponding to the second camera, the first target model can perform noise reduction processing based on any raw data, and all pre-noise reduction processing The first target model is used to perform noise reduction processing on the third raw data after the focusing operation satisfies the camera switching condition.

In this way, pre-noise reduction processing based on the second original data before switching can make the noise reduction effect of the first target model closer to the noise reduction effect of the third target model, so that after subsequent switching, the noise reduction effect of the first target model The image difference between the processed video frame and the output video frame before switching is small.

As an example of the present application, the pre-noise reduction processing is performed through the first target model corresponding to the second camera based on the second raw data, including:

Performing resolution reduction processing on the second original data according to a first preset ratio, and performing frame rate reduction processing on the second original data after resolution reduction processing according to a second preset ratio, to obtain fourth original data ;

The fourth original data is input into the first object model to perform pre-noise reduction processing on the first object model.

In this way, by performing resolution reduction processing and frame rate reduction processing on the second original data, outputting the fourth original data with small resolution and low frame rate, and then performing pre-noise reduction processing based on the fourth original data, the first objective can be improved. model performance.

As an example of the present application, in the process of video frame output, if a focusing operation is detected, before the camera switching condition is not met, starting the second camera includes:

In the process of video frame output, if a focusing operation is detected, before the camera switch condition is reached, when the focus value corresponding to the focus operation is between the target focus value corresponding to the camera switch condition When the difference between them is less than or equal to the preset value, start the second camera.

If the difference between the focus value corresponding to the focus operation and the target focus value corresponding to the camera switching condition is less than or equal to the preset value, it means that the user may focus to meet the camera switching condition with a high probability, so the second camera, which can improve the timeliness and effectiveness of starting the second camera.

As an example of the present application, the electronic device includes a first image signal processing module and a second image signal processing module;

The video frame output according to the first raw data of the first camera includes:

The image sensor of the first camera outputs the first raw data;

The first image signal processing module acquires the first raw data;

The first image signal processing module copies the first original data to obtain fifth original data;

The first image signal processing module performs image enhancement processing on the first raw data to obtain video enhancement data;

The first image signal processing module sends the video enhancement data and the fifth original data to the second image signal processing module;

The second image signal processing module outputs video frames based on the video enhancement data and the fifth original data.

In this way, image enhancement processing is performed by the first image signal processing module, and the first image signal processing module also provides the second image signal processing module with fifth raw data that can be used to adjust exposure parameters, so that the second image signal can The processing module can determine a clear video frame, which solves the problem that the second image signal processing module cannot adopt a complex multi-frame enhancement processing algorithm similar to taking pictures, so that the display effect of the video picture is often significantly worse than that of the captured image.

As an example of the present application, the first raw data includes long exposure data and short exposure data collected in the same time period, and the first image signal processing module performs image enhancement processing on the first raw data, including:

The first image signal processing module fuses the long exposure data and the short exposure data to obtain fused original data;

The first image signal processing module performs noise reduction processing on the fused original data to obtain the enhanced video data.

In this way, the long-exposure data and the short-exposure data within the same time period can be fused subsequently to output high dynamic video frames.

As an example of the present application, the first image signal processing module fuses the long exposure data and the short exposure data, including:

The first image signal processing module inputs the long-exposure data and the short-exposure data into a second object model, and performs fusion processing by the second object model, and the second object model can Exposure data and short exposure data are fused. In this way, the fusion process can be improved by using the second target model.

As an example of the present application, the first image signal processing module performs noise reduction processing on the fused raw data, including:

The first image signal processing module inputs the fused raw data into a third object model corresponding to the first camera, and performs noise reduction processing by the third object model, and the third object model can perform any The original data is subjected to noise reduction processing. In this way, noise reduction efficiency can be improved by using the third target model to perform noise reduction processing.

As an example of the present application, the first image signal processing module includes a plurality of third object models corresponding to the first camera, and each third object model in the plurality of third object models corresponds to an exposure range of values; the method also includes:

The first image signal processing module receives target exposure data, the target exposure data is determined by the second image signal processing module based on the first exposure data, and the first exposure data is determined by the second image signal processing module The module obtains exposure data statistics based on the fifth raw data, and the target exposure data is used to adjust the exposure parameters of the first camera;

The first image signal processing module selects a third target model from the plurality of third target models according to the target exposure data and the exposure value range corresponding to each third target model, and the selected third target model A three-objective model is used for noise reduction processing.

In this way, according to the exposure value range to which the target exposure data belongs, a third target model for the next noise reduction process is selected from a plurality of third target models, so that a reasonable noise reduction process can be performed on the next video data , thereby improving the noise reduction effect.

As an example of the present application, the second image signal processing module performs video frame output based on the video enhancement data and the fifth original data, including:

The second image signal processing module performs format conversion processing on the video enhancement data to obtain a YUV image;

The second image signal processing module determines target data based on the fifth raw data, and the target data is used to adjust the image quality of the YUV image;

The second image signal processing module adjusts the YUV image based on the target data, and outputs the adjusted YUV image as the video frame.

In this way, the format conversion process is performed on the video enhancement data through the second image signal processing module, and the target data is determined based on the second raw data, and the YUV image obtained after the format conversion process is optimized according to the target data, so as to obtain a clear picture video frame.

As an example of the present application, the second image signal processing module includes an ISP integrated in a system-on-a-chip (SOC), and the first image signal processing module includes an ISP outside the SOC.

In this way, by sharing the task of processing video frames through an external ISP, the burden of the built-in ISP in the SOC can be reduced, thereby realizing real-time processing of video frames, so that video images that meet the requirements can be obtained.

As an example of the present application, the image sensor of the first camera outputs the first raw data, including:

A night scene video shooting instruction is detected through the camera application in the electronic device, and the night scene video shooting instruction is used to indicate video recording in night scene mode;

In response to the night scene video shooting instruction, the image sensor outputs the first raw data.

In this way, in the night scene, the electronic device acquires the first raw data, and processes the first raw data collected by the camera through the method provided in this application, so that the highlighted area of the obtained video frame will not be overexposed and the darkened area will not be overexposed. Not too dark, resulting in a clear video frame.

In a second aspect, a camera switching device is provided, which is configured in an electronic device, and the electronic device includes a first camera and a second camera; the device includes an image sensor node, a first image signal processing module, and a second Image signal processing module;

The first image signal processing module and the second image signal processing module are configured to output video frames according to the first raw data of the first camera;

The image sensor node is used to start the second camera before the camera switching condition is met if a focusing operation is detected during the video frame output process;

The first image signal processing module and the second image signal processing module are configured to, according to the first raw data and the second raw data of the second camera, combine the second camera with the first The camera performs parameter synchronization;

The first image signal processing module and the second image signal processing module are configured to output video frames according to third raw data when the focusing operation satisfies the camera switching condition, and the third raw data is the raw video data of the second camera after parameter synchronization.

As an example of the present application, the first image signal processing module and the second image signal processing module are configured to:

determining first imaging information according to the first raw data;

determining second imaging information according to the second raw data;

As an example of the present application, the first image signal processing module is used for:

Based on the second raw data, pre-noise reduction processing is performed on the first target model corresponding to the second camera, the first target model can perform noise reduction processing on any raw data, and all pre-noise reduction processing The first target model is used to perform noise reduction processing on the third raw data after the focusing operation satisfies the camera switching condition.

As an example of the present application, the image sensor node is used for:

As an example of this application;

The image sensor of the first camera outputs the first raw data;

The first image signal processing module is configured to acquire the first raw data;

The first image signal processing module is configured to copy the first original data to obtain fifth original data;

The first image signal processing module is configured to perform image enhancement processing on the first raw data to obtain video enhancement data;

The first image signal processing module is configured to send the video enhancement data and the fifth original data to the second image signal processing module;

The second image signal processing module is configured to output video frames based on the video enhancement data and the fifth original data.

As an example of the present application, the first raw data includes long exposure data and short exposure data collected in the same time period, and the first image signal processing module is used for:

performing fusion processing on the long exposure data and the short exposure data to obtain fusion original data;

Noise reduction processing is performed on the fused original data to obtain the enhanced video data.

Input the long exposure data and the short exposure data into a second target model, and perform fusion processing by the second target model, and the second target model can fuse any long exposure data and short exposure data deal with.

The fused original data is input into a third object model corresponding to the first camera, and the third object model performs noise reduction processing, and the third object model can perform noise reduction processing on any original data.

As an example of the present application, the first image signal processing module includes a plurality of third object models corresponding to the first camera, and each third object model in the plurality of third object models corresponds to an exposure range of values;

The first image signal processing module is also used for:

receiving target exposure data, the target exposure data is determined by the second image signal processing module based on the first exposure data, the first exposure data is determined by the second image signal processing module based on the fifth raw data Obtained by performing exposure data statistics, the target exposure data is used to adjust the exposure parameters of the first camera;

According to the target exposure data and the exposure value range corresponding to each third target model, select a third target model from the plurality of third target models, and the selected third target model is used for noise reduction processing .

As an example of the present application, the second image signal processing module is used for:

Perform format conversion processing on the video enhancement data to obtain a YUV image;

determining target data based on the fifth raw data, the target data is used to adjust the image quality of the YUV image;

adjusting the YUV image based on the target data, and outputting the adjusted YUV image as the video frame.

As an example of the present application, the image sensor node is used for:

In response to the night scene video shooting instruction, the first raw data is output.

In the third aspect, an electronic device is provided, the structure of the electronic device includes a processor and a memory, and the memory is used to store a program that supports the electronic device to execute the method described in any one of the above-mentioned first aspects, and to store a program for The data involved in implementing the method described in any one of the above first aspects; the processor is configured to execute the program stored in the memory. The electronic device may also include a communication bus for establishing a connection between the processor and the memory.

In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer is made to execute the method described in any one of the above-mentioned first aspects.

According to a fifth aspect, there is provided a computer program product containing instructions, which, when run on a computer, causes the computer to execute the method described in the first aspect above.

The technical effects obtained by the above-mentioned second aspect, third aspect, fourth aspect and fifth aspect are similar to those obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here.

Description of drawings

FIG. 1 is a schematic layout diagram of a camera provided in an embodiment of the present application;

FIG. 2 is a schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

FIG. 4 is an interactive schematic diagram of an application scenario provided by an embodiment of the present application;

FIG. 5 is an interactive schematic diagram of another application scenario provided by the embodiment of the present application;

FIG. 6 is an interactive schematic diagram of another application scenario provided by the embodiment of the present application;

FIG. 7 is a schematic flowchart of a video frame output method provided by an embodiment of the present application;

FIG. 8 is a schematic flowchart of a camera switching method provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of a hardware architecture provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the implementation manner of the present application will be further described in detail below in conjunction with the accompanying drawings.

It should be understood that the "plurality" mentioned in this application means two or more. In the description of this application, unless otherwise specified, "/" means or means, for example, A/B can mean A or B; "and/or" in this article is just a description of the relationship between associated objects, It means that there can be three kinds of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in order to clearly describe the technical solution of the present application, words such as "first" and "second" are used to distinguish the same or similar items with basically the same function and effect. Those skilled in the art can understand that words such as "first" and "second" do not limit the number and execution order, and words such as "first" and "second" do not necessarily limit the difference.

Reference to "one embodiment" or "some embodiments" or the like in the specification of the present application means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically stated otherwise. The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless specifically stated otherwise.

Before introducing the method provided by the embodiment of the present application in detail, the nouns and execution subjects involved in the embodiment of the present application are firstly introduced.

First, the nouns involved in the embodiments of the present application are introduced.

Exposure: According to the length of exposure time, it can be divided into long exposure and short exposure. The longer the exposure time, the greater the amount of light entering the aperture. Conversely, the shorter the exposure time, the smaller the amount of light entering the aperture.

3A statistical algorithm: including automatic exposure (automatic exposure, AE) algorithm, automatic focus (automatic focus, AF) algorithm, and automatic white balance (automatic white balance, AWB) algorithm.

AE: refers to the camera automatically determines the exposure according to the lighting conditions. The imaging system generally has an AE function, which is directly related to the brightness and image quality of the image screen, that is, it will determine the brightness and darkness of the image.

AF: It means that the camera automatically adjusts the focus distance of the camera according to the distance of the subject, that is, the lens in the camera is adjusted to form the focus through distance measurement, so that the image in the camera is clear.

AWB: Mainly used to solve the problem of image color cast. If there is a color cast in the image, it can be corrected by the AWB algorithm.

Field of view: (field angle, FOV), refers to the range that the camera can cover. The larger the FOV, the more scenes the camera can accommodate. It is not difficult to understand that if the subject is not within the FOV of the camera, it will not be captured by the camera.

Image sensor (Sensor): It is the core component of the camera, and its function is to convert optical signals into electrical signals for subsequent processing and storage. The working principle is that the photosensitive element generates charge under the condition of light, and the charge transfer generates a current, and the current is rectified and amplified, and converted into a digital signal to form a digital signal. Image sensors generally include two types: charge coupled device (CCD) and complementary metal oxide semiconductor (complementary metal oxide semiconductor, CMOS).

RAW data: also referred to as raw data in this embodiment of the application, refers to the raw data that the CCD or CMOS image sensor in the camera converts the captured light source signal into a data signal. That is to say, it can be understood as unprocessed data, which can be used to describe the intensity of various lights received by the image sensor.

Next, the execution subject involved in the embodiment of the present application will be introduced.

The method provided in the embodiment of the present application may be executed by an electronic device having a shooting function. The electronic device is configured with multiple cameras, and different cameras in the multiple cameras have different shooting capabilities. For example, multiple cameras may include, but not limited to, a wide-angle camera, a telephoto camera (such as a periscope telephoto camera), a black and white camera, and an ultra-wide-angle camera. Usually, multiple cameras include a main camera and at least one auxiliary camera. For example, please refer to Figure 1. The spatial position distribution of multiple cameras can be shown in Figure 1 (a), or, the spatial distribution of multiple cameras The location distribution can also be shown in (b) in FIG. 1, the plurality of cameras are respectively camera 00, camera 01, camera 02, and camera 03. Exemplarily, camera 00 is the main camera, and the others are auxiliary cameras. After starting the camera application, the electronic device usually shoots through the main camera by default. After the camera is switched, the electronic device selects a suitable auxiliary camera from at least one auxiliary camera according to the switching requirement, and shoots through the selected auxiliary camera. For example, please refer to Figure 1. By default, camera 00 is used to shoot, and after the camera is switched to wide-angle, camera 01 is used to shoot. By way of example and not limitation, an electronic device may be, but not limited to, a mobile phone action camera (GoPro), digital camera, tablet computer, desktop, laptop, handheld computer, notebook computer, vehicle-mounted device, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook, cellular phone, personal digital assistant (personal digital assistant, PDA), augmented reality (augmented reality, AR)\virtual reality (virtual reality, VR) equipment, mobile phone, etc. Not limited.

Please refer to FIG. 2 . FIG. 2 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, ISP 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like. In one example, the number of ISPs 191 included in the electronic device is multiple, and only one is exemplarily shown in FIG. 2 .

It can be understood that the structure shown in this embodiment does not constitute a specific limitation on the electronic device 100 . In other embodiments, the electronic device 100 may include more or fewer components than shown, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU) wait. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

Wherein, the controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL).

It can be understood that the interface connection relationship between the modules shown in this embodiment is only for schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 is charging the battery 142 , it can also supply power to the electronic device 100 through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be disposed in the processor 110 . In some other embodiments, the power management module 141 and the charging management module 140 may also be set in the same device.

The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

The electronic device 100 can realize the shooting function through the ISP 191, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

The ISP 191 is used for processing the data fed back by the camera 193. For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP 191 for processing, and converts it into an image visible to the naked eye. ISP 191 can also optimize the algorithm for image noise, brightness, and skin tone. ISP 191 can also optimize the exposure, color temperature and other parameters of the shooting scene.

In one embodiment, the ISP 191 may include a built-in ISP integrated in the SOC and an external ISP arranged outside the SOC. In comparison, the internal structure of the external ISP is similar or the same as that of the built-in ISP. The difference is , The external ISP and the built-in ISP have different processing tasks for video data. As an example of this application, the external ISP mainly has two functions: on the one hand, it is used to fuse and enhance the original RAW data collected by the camera during the process of video recording by the electronic device 100 through the camera, so as to provide Built-in ISP provides enhanced video data. In addition, the original RAW data collected by the camera is routed to provide a copy of the original RAW data for the built-in ISP, so that the built-in ISP can accurately determine the current exposure data, and then the built-in ISP can dynamically adjust the exposure of the camera according to the exposure data parameter. On the other hand, the external ISP is used to respond to the focus operation, start another camera in advance, and synchronize the parameters of the other camera with the camera before switching, so as to achieve smooth switching. For the specific implementation, please refer to each of the following Example.

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP 191 for conversion into a digital image signal. ISP 191 outputs the digital image signal to DSP for processing. DSP converts digital image signals into standard RGB (red green blue red green blue), YUV and other image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.

The internal memory 121 may be used to store computer-executable program codes including instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 . The internal memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.

The electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .

The earphone interface 170D is used for connecting wired earphones. The earphone interface 170D may be a USB interface 130, or a 3.5mm open mobile terminal platform (open mobile terminal platform, OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 . The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like. The SIM card interface 195 is used for connecting a SIM card. The SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card etc.

In one embodiment, the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the software structure of the electronic device 100 is exemplarily described by taking an Android system with a layered architecture as an example.

FIG. 3 is a block diagram of the software structure of the electronic device 100 provided by the embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is respectively an application program layer, a hardware abstract layer (Hardware Abstract Layer, HAL), a kernel layer, and a hardware layer from top to bottom. In addition, an application framework layer (Application Framework) (not shown in FIG. 3 ) is also included between the application layer and the HAL, which is not highlighted in this embodiment of the present application.

The application layer can consist of a series of application packages. As shown in Figure 3, the application package may include application programs such as a camera and a gallery.

As an example of the present application, the camera application supports a super night video mode, and the electronic device can shoot clear, bright and dark videos in night scenes in the super night video mode.

As an example of this application, the application layer also provides pre-loaded external ISP services. Since the internal memory of the external ISP is usually random access memory (Random Access Memory, RAM), according to the characteristics of RAM, RAM cannot save data in the case of power failure, so the external ISP is usually stored during operation. The required data such as external ISP SDK and models (for example, including the first target model, the second target model, and the third target model described below) are stored in the system memory. When the camera application is started, the application layer starts the preloaded external ISP service, and the external ISP driver controls the external ISP to be powered on in advance by preloading the external ISP service, so that the data required during the operation of the external ISP It is loaded from the system memory to the internal RAM of the external ISP, so that the external ISP can perform corresponding functions (such as data fusion, noise reduction processing, etc.) in the ultra-night video mode.

As an example of this application, the video recorded by the camera can be provided in the gallery application, so that the user can view the recorded video from the gallery application.

The HAL layer mainly includes a video module, which is used to obtain RAW data through the image sensor of the camera, and perform fusion, enhancement, optimization and other processing on the RAW data through the external ISP and the built-in ISP respectively to obtain video with enhanced clarity and noise reduction effect frame. The resulting video frames are then sent to the display for display. In addition, the video module also saves the recorded video to the gallery application for easy viewing by users.

As an example of this application, the video module is also used to start the camera to be started in advance when the camera needs to be switched, and before the camera is switched, the parameters of the camera that is started in advance are synchronized with the camera before the switch, so that Achieve smooth switching.

In one example, the video module includes an image sensor node, a built-in ISP node, and an external ISP node. Each node can be understood as an encapsulation of the functions performed by the underlying hardware, which can be perceived and invoked by the upper layer (application layer). Exemplarily, the image sensor node encapsulates the function of the image sensor in the bottom camera; the built-in ISP node encapsulates the function of the bottom built-in ISP; the external ISP node encapsulates the function of the bottom external ISP. In implementation, the video module implements corresponding functions through the interaction between the image sensor node, the built-in ISP node, and the external ISP node.

The interior of the external ISP node may include multiple submodules, such as a routing submodule, a first preprocessing submodule, and an enhancement submodule. Similarly, each of the multiple sub-modules can be understood as encapsulating the functions of different hardware in the bottom-level external ISP. As an example of this application, the routing sub-module is the function of the routing unit in the bottom-level external ISP. Encapsulation, the first preprocessing sub-module is the encapsulation of one or more IFE functions in the bottom external ISP, and the enhancement sub-module is the function of the neural network processing unit (NPU) in the bottom external ISP package. In implementation, the external ISP node implements corresponding functions through the interaction between multiple sub-modules.

The interior of the built-in ISP node includes a plurality of submodules, for example, a second preprocessing submodule and an optimization processing submodule. Each of the multiple sub-modules can be understood as encapsulating the functions of different hardware in the underlying built-in ISP. As an example of this application, the second preprocessing sub-module is to process one or more images in the underlying built-in ISP Encapsulation of the functions of the front-end engine (image front end, IFE); the optimization processing sub-module is the encapsulation of the functions of the image processing engine (IPE) in the underlying built-in ISP. In implementation, the built-in ISP node realizes the corresponding functions through the interaction of multiple sub-modules.

In addition, the HAL layer also includes an external ISP software development kit (software development kit, SDK), which is used to establish the interaction between multiple sub-modules inside the external ISP node.

The kernel layer is the layer between hardware and software. The kernel layer includes but not limited to camera driver, built-in ISP driver, and external ISP driver.

The hardware layer includes but not limited to camera, built-in ISP, external ISP, display.

In the following, the workflow of the software and hardware of the electronic device 100 will be exemplarily described in conjunction with video recording in the night scene.

In one embodiment, if the camera application detects that video shooting is enabled in the super night video mode, it sends a night scene video shooting request to the video module of the HAL layer. After the video module receives the night scene video shooting request, it establishes a framework for processing the night scene video. Exemplarily, the video module notifies the camera driver to control the main camera to be powered on according to the night scene video shooting request, and notifies the built-in ISP driver to control the built-in ISP to be powered on. Correspondingly, the camera driver drives the main camera. After the main camera is loaded, the camera driver is notified to notify the video module that the main camera has been loaded through the camera driver. In addition, the built-in ISP driver drives the built-in ISP, and notifies the built-in ISP driver after the built-in ISP is loaded, so as to notify the video module that the built-in ISP has been loaded through the built-in ISP driver. After the video module determines that the main camera, built-in ISP and external ISP (for example, the loading time is after the camera application is started), the interaction between the image sensor node, the built-in ISP node and the external ISP node is established. . In this way, the video data can be collected and optimized by calling the video module, and then the optimized video data collected by the main camera can be output to the display screen for display.

As an example of the present application, if the camera application detects the user's focusing operation, before the camera switching condition is met, the camera application sends a camera pre-start instruction to the video module of the HAL layer. In an example, the camera pre-start instruction carries a target camera identifier, and the target camera identifier is used to indicate the auxiliary camera to be started. After receiving the camera pre-start instruction, the video module notifies the camera driver to control the power-on of the auxiliary camera according to the camera pre-start instruction, so as to start the auxiliary camera in advance for video data collection before switching, and then, according to the video data collected by the auxiliary camera and the main camera The video data of the main camera and the auxiliary camera are synchronized. When the camera switching condition is satisfied, the video module optimizes the video data collected by the auxiliary camera after parameter synchronization, and then outputs the optimized video data collected by the auxiliary camera for display on the display screen. For specific implementation, reference may be made to the various embodiments described below.

After introducing the nouns and execution subjects involved in the embodiments of the present application, the application scenarios involved in the embodiments of the present application will be introduced next by taking the electronic device as a mobile phone including multiple rear cameras as an example.

There is a camera application program in the mobile phone, please refer to (a) figure in Figure 4, in one embodiment, the user wants to take night scene video by mobile phone, at this moment, the user can click the application icon of the camera application program in the mobile phone. In response to the user's trigger operation on the application icon of the camera application, the mobile phone starts the main camera in the rear camera, and displays the first interface as shown in (b) in FIG. 4 for the user.

As an example of the present application, a "night scene" option 41 is provided in the first interface, the user can trigger the "night scene" option 41, and in response to the user's trigger operation on the "night scene" option 41, the mobile phone displays the operation interface in the night scene mode ( referred to as the second interface), for example, the second interface is shown in (c) in FIG. 4 . The second interface provides a first switching option 42 and a second switching option 43, wherein the first switching option 43 is used to switch between the front camera and the rear camera. The second switch option 43 is used to switch between the camera mode and the video capture mode.

In one example, after entering the night scene mode, that is, after switching from (b) in FIG. 4 to (c) in FIG. Example), when the user wants to shoot a night scene video, the second switch option 43 can be triggered, and in response to the trigger operation of the second switch option 43 by the user, the mobile phone switches from the camera mode to the video capture mode.

In another example, after entering the night scene mode, that is, after switching from (b) in Figure 4 to (c) in Figure 4, the mobile phone can also be in the video shooting mode by default, in this case , if the user wants to take a night scene image, the second switch option 43 can be triggered, and in response to the trigger operation of the second switch option 43 by the user, the mobile phone switches from the video shooting mode to the camera mode.

In one embodiment, a shooting option 44 is also provided in the second interface, and the user can trigger the shooting option 44 . In response to the user's trigger operation on the shooting option 44, the mobile phone records a video through a camera (such as a main camera). For example, please continue to refer to FIG. 4 , and the video recording interface is shown in (d) in FIG. 4 . As an example of the present application, in a night scene, the mobile phone processes the video data collected by the camera through the method provided in the present application, so that a clear video frame can be captured finally. The clarity of the picture mentioned here means that the highlighted areas will not be overexposed, and the dark areas will not be too dark.

In an example, please continue to refer to figure (d) in FIG. 4 , a pause option 45 is provided in the video recording interface. During the video recording process, when the user wants to pause the video recording, the pause option 45 can be triggered, and in response to the user's trigger operation on the pause option 45, the mobile phone pauses the video recording.

In an example, please continue to refer to figure (d) in FIG. 4 , a snapshot option 46 is provided in the video recording interface. During the video recording process, when the user wants to capture a certain video frame, the capture option 46 can be triggered. In response to the trigger operation of the capture option 46 by the user, the mobile phone performs a capture operation and stores the captured video frame.

Please refer to FIG. 5 , there is a "more" option 51 in the first interface. As an example of the present application, when the user wants to take a night scene video, the "more" option 51 can be triggered. In response to the user's trigger operation on the "more" option 51, the mobile phone displays a third interface, for example, the third interface is shown in (b) of FIG. 5 . As an example of the present application, a "night scene recording" option 52 is provided in the third interface, and the "night scene recording" option 52 is used to trigger the video recording function under the night scene scene, that is, compared to the example shown in FIG. 4, Here you can also set up an option for shooting night scene videos separately. When the user wants to record the night scene video through the mobile phone, the option 52 of "night scene recording" can be triggered. In response to the user's trigger operation on the "night scene recording" option 52, the mobile phone displays an operation interface (called the fourth interface) in the night scene mode. Exemplarily, the fourth interface is shown in (c) in FIG. 5 .

In one embodiment, a shooting option 53 is provided on the fourth interface, and the user can trigger the shooting option 53 . In response to the user's trigger operation on the shooting option 53, the mobile phone records a video through a camera (such as a main camera). For example, the video recording interface is shown in (d) in FIG. 5 . In addition, a first switching option 54 may also be provided in the fourth interface, and the first switching option 54 is used to switch between the front camera and the rear camera. Different from the embodiment shown in FIG. 4 , there is no need to provide the second switching option in the fourth interface, that is, the “night scene recording” option 52 for triggering night scene video recording is provided separately under the “more” option.

As an example of the present application, please refer to diagram (a) in FIG. 6 , the video recording interface provides a focusing item 47 for focusing. During the video recording process, when the user wants to focus, the focus adjustment item 47 can be triggered, such as adjusting from 1x focus to telephoto, such as adjusting to multi-fold focus (such as 2x focus), or from 1x focus to wide-angle Adjustment, such as adjusting to 0.8 times focus. In response to the user's trigger operation on the focusing item 47, the mobile phone focuses on the main camera, or switches to other auxiliary cameras for video collection. For example, when the user adjusts from 1x focus to nx focus, when n is greater than 1 and less than 2, the mobile phone will focus on the main camera, and when n is greater than or equal to 2, the mobile phone will switch from the main camera to the telephoto camera. For another example, when the user adjusts from 1x focus to wide-angle, the mobile phone switches from the main camera to the wide-angle camera. During the focusing process, in order to facilitate the user to know how much focus is currently adjusted, the mobile phone displays the current focusing result. For example, please refer to (b) in Figure 6. Take the user’s adjustment to the telephoto as an example. The current focusing result is displayed near item 47, and the display effect is shown as 61 in FIG. 6 .

As another example of the present application, when the mobile phone is shooting video through the auxiliary camera, when a user's focusing operation is detected based on the focusing item 47, the mobile phone switches from the auxiliary camera to the main camera. Exemplarily, when the mobile phone is shooting video with the telephoto camera, when the user's focusing operation is detected based on the focusing item 47, and the focusing result is 1x focus, the telephoto camera is switched back to the main camera.

In this embodiment of the application, when switching from one camera to another camera, if there is a reserved time between the two cameras, that is, switching from one camera to another camera is not performed immediately, but when the focus adjustment result Switching is only performed when the switching conditions are met, for example, when adjusting from 1x focus to 3x focus, switch from the main camera to the telephoto camera, but do not switch before 3x focus. In this case, the mobile phone can achieve smooth switching. From the perspective of the user's vision, there is not much difference between the video images before and after focusing, and the user is not aware of the camera switching moment. In addition, no matter before or after the focus adjustment, the video data collected by the camera currently performing video recording can be processed, so that a clear video frame can be finally obtained.

Of course, it should be noted that the above is described by taking night scene shooting as an example. In another embodiment, the method provided by the embodiment of the present application can also be applied to a conventional video recording scene. For example, refer to (a) in FIG. 5 As shown in the figure, after the user triggers the electronic device to perform video recording through the "recording" option, the electronic device can still use the method provided by the embodiment of the present application to switch between cameras and optimize the collected video data. In another embodiment, the method can also be applied to the camera preview scene, that is, when the electronic device starts the camera and enters the preview state, it can switch the camera and process the preview image using the method provided in the embodiment of the present application.

Next, in combination with the system architecture shown in FIG. 3 , the implementation process of the camera switching method provided by the embodiment of the present application will be introduced in detail. As an example but not a limitation, the method is applied to an electronic device, and the electronic device is implemented through the interaction between various nodes shown in FIG. 3 . Here, switching from the primary camera to the secondary camera is taken as an example for illustration. Please refer to Figure 7, firstly introduce the processing flow of the video data of the main camera by the electronic device, which may specifically include the following implementation steps:

701: The image sensor node acquires first RAW data.

As an example of the present application, the first RAW data is the RAW data output by the image sensor of the main camera. In implementation, after the electronic device starts the camera application, if the camera application detects a trigger operation of video shooting in the super night video mode, in response to the trigger operation, the camera application sends a night scene video shooting request to the video module. Exemplarily, please refer to the interaction flow in the embodiment of FIG. Generate a night scene video shooting request, and send the night scene video shooting request to the video module. After the video module receives the night scene video shooting request, it establishes a framework for processing the night scene video, and the specific implementation can be referred to above. Afterwards, the image sensor node collects and captures the light source signal through the image sensor in the main camera, and converts the captured light source signal into a data signal to obtain the first RAW data. Exemplarily, the first RAW data is 4K60 interlaced high dynamic range (staggered high dynamic range, SHDR) data, where 4K60 means that the resolution is 4K and the frame rate is 60 frames per second.

In one embodiment, the first RAW data includes long exposure data and short exposure data, the long exposure data refers to the data collected by the image sensor through the long exposure method, and the short exposure data refers to the data collected by the image sensor through the short exposure method . That is, two exposures are performed within one exposure time to obtain the first RAW data. Exemplarily, taking the first RAW data as 4K60 SHDR data as an example, the main camera exposes twice within each 33ms, thus obtaining 60 frames of video data.

It is worth mentioning that the combination of long and short exposures can effectively improve the dynamic range of the video frame. Short exposures are used to prevent overexposure in highlighted areas, and long exposures are used to brighten dark areas to prevent underexposure.

702: The image sensor node sends the first RAW data to the external ISP node.

Exemplarily, the image sensor node sends the 4K60 SHDR data to the external ISP node for processing such as fusion and enhancement through the external ISP node.

As an example of the present application, the first RAW data first arrives at the routing submodule in the external ISP node.

703: The routing submodule performs copying and routing processing on the first RAW data.

When an electronic device shoots a video in a night scene, in order to obtain a clear video frame, on the one hand, it can perform processing such as enhancement on the first RAW data; on the other hand, it can calculate the exposure data according to the first RAW data to obtain the first exposure data, and then dynamically adjust the exposure parameters of the main camera according to the first exposure data.

Therefore, as an example of the present application, the routing sub-module in the external ISP node performs copying and routing processing on the first RAW data. In an implementation, the routing submodule copies the first RAW data to obtain another copy of RAW data, which is called fifth RAW data here. Afterwards, routing processing is performed on the first RAW data and the fifth RAW data. Specifically, the routing submodule transmits a piece of RAW data (such as the first RAW data) in the first RAW data and the fifth RAW data to the first preset The processing sub-module performs processing, and another piece of RAW data (for example, the fifth RAW data) is used for subsequent built-in ISP nodes to count the first exposure data.

It should be noted that the above is an example of transmitting the first RAW data to the first preprocessing submodule and transmitting the fifth RAW data to the built-in ISP node. In another embodiment, the first RAW data may also be transmitted to the built-in ISP node, and the fifth RAW data may be transmitted to the first preprocessing submodule, which is not limited in this embodiment.

Next, the routing sub-module transmits the first RAW data to the first pre-processing sub-module, and transmits the fifth RAW data to the built-in ISP node for the purpose of counting the first exposure data as an example.

704: The first preprocessing submodule performs preprocessing on the first RAW data.

Due to the influence of factors such as the unsatisfactory physical components of the main camera, there may be certain defects in the first RAW data, such as the influence of dark current, the brightness attenuation around the image, and the existence of dead pixels. Before performing fusion and noise reduction processing on the RAW data, the first preprocessing submodule first preprocesses the first RAW data, so as to correct the first RAW data.

As an example and not limitation, preprocessing includes but is not limited to lens shading correction (LSC) processing, black level compensation (black level compensation, BLC) processing, bad pixel correction (bad pixel correction, BPC processing), color interpolation At least one of the processing.

705: The first preprocessing submodule sends the preprocessed first RAW data to the enhancement submodule.

For example, the first preprocessing submodule sends the preprocessed 4K60 SHDR data to the enhancement submodule.

706: The enhancement submodule performs fusion and noise reduction processing on the preprocessed first RAW data.

As an example of the present application, the specific implementation of performing fusion processing on the preprocessed first RAW data may include: inputting the preprocessed first RAW data into the second target model for processing, and outputting the fusion RAW data. The second target model can perform fusion processing on arbitrary long-exposure data and short-exposure data.

Exemplarily, if the preprocessed first RAW data is 4K60 SHDR data, the 4K60 SHDR data is input into the second target model, and the fused RAW data obtained after fusion processing is 4K30 data. That is to say, when the second target model performs fusion processing, it fuses the long-exposure data and short-exposure data obtained through two consecutive exposures in the same period of time, so the 60 frames of data before fusion become 30 frames. In this way, the signal-to-noise ratio and dynamic range of video data can be improved through fusion processing.

Wherein, the second target model may be a pre-trained fusion network model. For example, the second target model may be obtained after training the second network model based on the exposure sample data. In one example, the second network model may include, but is not limited to, HDRnet.

Afterwards, noise reduction processing is performed on the fused RAW data obtained after fusion. As an example of the present application, the implementation of denoising the fused RAW data may include: inputting the fused RAW data into a third object model corresponding to the main camera for processing, and outputting the noise-reduced video data. The third target model can perform noise reduction processing on arbitrary video data.

Wherein, the third target model may be a pre-trained denoising network model. For example, the third target model may be obtained after training the third network model based on the RAW sample data. In one example, the third network model may include, but is not limited to, Unet.

It should be noted that it is optional for the external ISP node to preprocess the first RAW data through the first preprocessing submodule. In another embodiment, the first RAW may also be directly sent to the enhancement submodule, by The enhancement sub-module performs fusion and noise reduction processing on the first RAW data.

707: The enhancement submodule outputs the video data after noise reduction processing, and the routing submodule outputs fifth RAW data.

Specifically, the enhancement submodule sends the noise-reduced video data to the built-in ISP node, and the routing submodule also sends the fifth RAW data to the built-in ISP node. The built-in ISP node receives the video data and the fifth RAW data through the second preprocessing submodule. It is not difficult to understand that the video data output by the enhancement sub-module is 4K30 data, which is used for browsing and recording; the fifth RAW data output by the routing sub-module is 4K60 data, which is used for calculating 3A and possible camera needs.

It should be noted that since the external ISP node performs fusion and noise reduction processing on the first RAW data of the main camera, there is generally a gap between the video data output by the external ISP node and the first RAW data output by the main camera. There is a certain delay. For example, there is a delay of one frame. For example, if the main camera outputs the first RAW data at time t, the external ISP node outputs video data at time t-1 at the same time.

In addition, the external ISP node controls the synchronous output of the enhancement sub-module and the routing sub-module, that is, the noise-reduced video data and the fifth RAW data are synchronously transmitted to the second pre-processing sub-module.

708: The second preprocessing submodule processes the video data output by the enhancement submodule, and calculates the first exposure data based on the fifth RAW data, and adjusts the exposure parameters.

As an example of the present application, the processing of the video data output by the enhancement submodule by the second preprocessing submodule includes: performing preprocessing again on the video data output by the enhancement submodule, such as but not limited to LSC processing, BLC processing, At least one of BPC processing and color interpolation processing to further reduce the noise of the video data. Afterwards, RGB conversion is performed on the preprocessed video data again, and the video image obtained after the RGB conversion is compressed to obtain a YUV image.

It should be noted that it is optional for the second preprocessing submodule described in the embodiment of the present application to preprocess the video data output by the enhancement submodule again. In another embodiment, the second preprocessing submodule also The RGB conversion may be performed directly based on the video data output by the enhancement sub-module, which is not limited in this embodiment of the present application.

In addition, the second preprocessing sub-module determines the first exposure data based on the fifth RAW data, and determines whether the current exposure level is reasonable according to the first exposure data, and then adjusts the exposure parameters of the main camera if it is not reasonable. The value range of the first exposure data is (0, 255). In one example, the second preprocessing submodule compares the first exposure data with the exposure threshold, and if the difference between the first exposure data and the exposure threshold is greater than the threshold range, gradually adjusts the first Exposure data to obtain target exposure data. The second preprocessing sub-module sends the target exposure data to the main camera, so that the main camera adjusts the exposure parameters of the image sensor. The ultimate goal is to make the exposure data calculated according to the fifth RAW data output by the main camera close to or same.

Wherein, the adjustment step size may be set according to actual requirements. The exposure threshold can be set according to actual needs. The threshold range can also be set according to actual needs.

For example, the exposure threshold is 128, the threshold range is [0,5], and the adjustment step is 4. If the first exposure data is 86, it means that the exposure parameter needs to be increased. At this time, the first exposure data can be adjusted according to the adjustment step to obtain a target exposure data of 90. The second preprocessing sub-module sends the target exposure data 90 to the main camera, so that the main camera adjusts the exposure parameter of the image sensor to 90. The exposure data is again counted according to the fifth RAW data received next time, and the exposure parameters of the image sensor are adjusted according to the above method until the counted exposure data is close to or equal to 128.

It is worth mentioning that by adjusting the exposure data step by step so that the exposure parameters of the main camera are close to or the same as the exposure threshold, the exposure changes of the video frames can be smoothly transitioned.

As an example of the present application, in addition to counting the first exposure data based on the fifth RAW data, the second preprocessing submodule may also count the first AWB, the first AF and the first FOV based on the fifth RAW data. In an example, the second preprocessing submodule sends the first AWB to the optimization processing submodule, so that the optimization processing submodule can perform white balance adjustment during image optimization processing. In addition, the second preprocessing sub-module sends the first AF to the main camera, so that the main camera performs adjustment processing according to the first AF. In an example, the first AWB, the first exposure data, the first AF, and the first FOV are used as the first imaging information, and the first imaging information may be used for subsequent parameter synchronization during camera switching.

709: The second preprocessing submodule sends the YUV image and target exposure data to the optimization processing submodule.

According to the foregoing, it can be known that the target exposure data is determined according to the first exposure data. For example, if the first exposure data is 100, and the second preprocessing submodule determines that the exposure parameter of the main camera needs to be adjusted to 200, then the target exposure data is 200.

As an example of the present application, since the second preprocessing submodule adjusts the exposure parameters of the main camera, the gain of the video data obtained through the main camera subsequently changes. The YUV image obtained is reasonably de-noised. The second pre-processing sub-module adjusts the exposure parameters of the main camera and sends the target exposure data to the optimization processing sub-module, which is convenient for the optimization processing sub-module to determine the noise reduction parameters. The noise reduction parameter performs reasonable noise reduction processing on the next received YUV image.

As an example of this application, the external ISP node includes multiple third target models corresponding to the main camera, each third target model in the multiple third target models corresponds to an exposure value range, and each third target The number of exposure value ranges corresponding to the model may be one or more. According to the foregoing, it can be seen that the third target model can be used for noise reduction processing. Similarly, in order to perform reasonable noise reduction processing on the next video data, the second preprocessing sub-module can also send the target exposure data to external Set the ISP node so that the external ISP node can determine the exposure value range to which the target exposure data fed back by the second preprocessing sub-module belongs, so that according to the determined exposure value range, the corresponding third target model can be selected from multiple third target models. The target model, the selected third target model is used for the next noise reduction process.

710: The optimization processing submodule performs image optimization processing based on the received data.

The optimization processing sub-module optimizes the YUV image according to the target exposure data, such as performing noise reduction processing on the YUV image, so as to obtain a clear and bright video frame.

711: The optimization processing submodule sends the obtained video frame to display.

That is, the optimization processing sub-module sends the video frames obtained after the image optimization processing to the display screen for display.

It is worth mentioning that in the super night video mode, the video data is fused and image enhanced through the external ISP, the processed video data is sent to the built-in ISP, and the original video data is provided for the built-in ISP. In this way, the built-in ISP can generate clear video frames based on the video data provided by the external ISP, which reduces the operating burden of the built-in ISP, thereby reducing the power consumption of the SOC.

When the focus is not adjusted, the electronic device processes the video data of the main camera according to the procedures of the above embodiments, and outputs high-definition video frames. In one embodiment, please refer to FIG. 8, when the electronic device detects a focusing operation, it performs the following operations:

801: The image sensor node starts the auxiliary camera according to the camera pre-start instruction.

As an example of this application, the camera pre-start instruction is issued by the camera application program. In one example, the camera application detects a focus operation, because in a possible situation, the focus operation may be adjusted to the field of view of the secondary camera, that is, it needs to switch from the primary camera to the secondary camera for shooting. Therefore, in order to avoid the stuck problem when the camera is switched, the camera application program sends a camera pre-start command to the video module before the camera switching condition is met, so as to notify the video module to start the auxiliary camera in advance. In an example, the camera pre-start instruction carries a target camera identifier, and the target camera identifier is used to uniquely identify an auxiliary camera. After the video module receives the camera pre-start instruction, it can start the auxiliary camera through the image sensor node. The specific process of starting the auxiliary camera by the image sensor node can be referred to above, and will not be repeated here.

In an example, the camera switching condition may be determined according to the field of view angle of the main camera and/or the field of view angle of the auxiliary camera. For example, the camera switching condition may mean that the focus value corresponding to the focus operation exceeds the target focus value, and the field of view corresponding to the target focus value exceeds the field of view of the main camera but is smaller than the field of view of the auxiliary camera.

In one embodiment, if a focusing operation is detected, before the camera switching condition is met, when the difference between the focusing value corresponding to the focusing operation and the target focusing value corresponding to the camera switching condition is less than or equal to the preset When a value is set, the secondary camera is activated.

Wherein, the preset value can be set according to actual needs.

For example, the auxiliary camera is a telephoto camera, and the target focus value is 3x focus, that is, when the focus value corresponding to the focus operation reaches 3x focus, it will automatically switch from the main camera to the telephoto camera. In addition, assuming that the preset value is 0.3, when the focus value corresponding to the focus operation reaches 2.7 times focus, the telephoto camera starts to be activated.

It should be noted that when there are multiple auxiliary cameras, since the field of view angles corresponding to different auxiliary cameras in the multiple auxiliary cameras are different, different auxiliary cameras correspond to a camera switching condition, or in other words, different auxiliary cameras correspond to a target Focus value.

802: The image sensor node acquires second RAW data.

The second RAW data is output through the image sensor of the auxiliary camera. That is, after the electronic device starts the auxiliary camera, the auxiliary camera starts to collect video data, and the image sensing node obtains the video data collected by the auxiliary camera to obtain the second RAW data. In one example, the second RAW data may be SHDR data, and in another example, the second RAW data may also be SDR data.

In one embodiment, the second RAW data includes long exposure data and short exposure data. That is to say, the image sensor of the auxiliary camera can expose twice in each exposure time period, one long exposure and one short exposure, for example, a CMOS image sensor can be used to expose twice in the same time period. In this way, the long-exposure data and the short-exposure data in the same time period can be fused subsequently to output high dynamic video frames. Exemplarily, taking the second RAW data as 4K60 SHDR data as an example, the auxiliary camera exposes twice in every 33ms, so as to obtain video data of 60 frames per second.

In another embodiment, the second RAW data may also be 4K30 video data, that is, the secondary camera exposes once in each exposure time period, and outputs video data at 30 frames per second. In this case, subsequent fusion processing is not required.

It should be noted that, during the process of the image sensor node collecting the second RAW data through the auxiliary camera, the electronic device still continues to execute the above step 701 to step 711 .

803: The image sensor node sends the second RAW data to the external ISP node.

804: The external ISP node replicates the second RAW data.

As an example of the present application, the external ISP node copies the second RAW data through the routing sub-module to obtain the sixth RAW data, and the sixth RAW data is used to count the current exposure data of the secondary camera and other information.

805: The external ISP node performs resolution reduction processing and frame rate reduction processing on the second RAW data to obtain fourth RAW data.

As an example of this application, the external ISP node transmits the second RAW data to the first preprocessing submodule, and the first preprocessing submodule performs resolution reduction and frame rate reduction processing on the second RAW data, thereby obtaining Fourth RAW data with small resolution and low frame rate. In an example, the first preprocessing submodule may perform resolution reduction processing on the second RAW data according to a first preset ratio, and then perform frame reduction processing on the second RAW data after resolution reduction processing according to a second preset ratio rate processing. Wherein, the first preset ratio and the second preset ratio can be set according to actual needs.

It is worth mentioning that by performing resolution reduction and frame rate reduction processing on the second RAW data, the power consumption of the subsequent pre-noise reduction processing can be reduced.

806: The external ISP node performs pre-noise reduction processing based on the fourth RAW data.

In an embodiment, the first preprocessing submodule in the external ISP node sends the fourth RAW data to the enhancement submodule. In an example, when the second RAW data includes long-exposure data and short-exposure data, the enhancement sub-module performs fusion processing on the fourth RAW data, for example, through the fusion of the second target model, and outputs a small-resolution, low-frame-rate Fusion data. Afterwards, the enhancement sub-module performs noise reduction processing based on the obtained fusion data with small resolution and low frame rate. Fusion data for pre-noise reduction.

It is worth mentioning that pre-noise reduction processing based on the video data collected by the auxiliary camera before switching can make the noise reduction effect of the first object model closer to that of the third object model, so that after subsequent switching The image difference between the video frame after the noise reduction processing of the first target model and the output video frame before switching is small.

Wherein, the first target model may be a pre-trained denoising network model. For example, the first target model may be obtained after training the first network model based on RAW sample data. In one example, the first network model may include but not limited to Unet.

In one embodiment, before the first preprocessing submodule sends the fourth RAW data to the enhancement submodule, it can also preprocess the fourth RAW data. For example, the preprocessing can include but not limited to LSC processing, BLC processing, At least one of BPC processing and color interpolation processing. Or, in another embodiment, the first preprocessing submodule may also perform preprocessing on the second RAW data before performing processing on the second RAW data to reduce resolution and frame rate. The embodiment of the present application does not limit the timing of preprocessing.

807: The external ISP node outputs the sixth RAW data to the internal ISP node.

In an example, the built-in ISP node receives the sixth RAW data through the second preprocessing submodule.

As an example of the present application, the timing for the routing submodule to output the sixth RAW data may occur after the copy operation, that is, there is no strict execution sequence between step 807 and step 804 .

808: The built-in ISP node collects statistics on the second camera information based on the sixth RAW data.

In one example, the second imaging information includes second exposure data, second AF, second AWB and second FOV.

The built-in ISP node collects statistics of the second camera information based on the sixth RAW data through the second preprocessing submodule.

809: The built-in ISP node adjusts the second camera information based on the first camera information.

In an example, the ISP node adjusts each parameter in the second camera information to be the same as or close to the corresponding parameter in the first camera information through the second preprocessing submodule, so as to perform information synchronization. For example, information synchronization between the first exposure data and the second exposure data, information synchronization between the first AF and the second AF, information synchronization between the first AWB and the second AWB, and information synchronization between the first FOV and the second FOV Information synchronization.

810: The built-in ISP node adjusts parameters of the auxiliary camera based on the adjusted second camera information.

For example, the built-in ISP node sends the adjusted second camera information to the auxiliary camera through the second preprocessing submodule, and instructs the auxiliary camera to adjust parameters according to the second camera information.

811: When the focusing operation satisfies the camera switching condition, the external ISP node processes the third RAW data output by the auxiliary camera whose parameters have been synchronized as foreground data.

In an example, when the camera application detects that the focus operation meets the camera switching condition, for example, when the focus value corresponding to the focus operation reaches the target focus value, the external ISP node is notified. The external ISP node processes the third RAW data output by the auxiliary camera according to the execution manner of steps 701 to 711 above, that is, processes the third RAW data output by the auxiliary camera as foreground data at this time.

For example, when the focusing value corresponding to the focusing operation reaches 3X, the external ISP node sends the third RAW data to the first pre-processing sub-module through the routing sub-module for pre-processing, and then the first pre-processing The sub-module sends the preprocessed third RAW data to the enhancement sub-module for fusion processing, and inputs the RAW data obtained after fusion processing to the first target model for noise reduction processing. In addition, the routing submodule copies the third RAW data.

812: The external ISP node sends the processed video data to the built-in ISP node.

The external ISP node outputs the noise-reduced video data to the built-in ISP, and outputs the RAW data copied from the third RAW to the built-in ISP node. Afterwards, the built-in ISP node processes the received video data through the second preprocessing submodule according to the processing method for the first RAW data, and outputs video frames with higher definition after optimized processing by the optimization processing submodule.

In a possible implementation manner, after switching from the main camera to the auxiliary camera, the main camera may be controlled to be powered off. In another possible implementation manner, after switching from the main camera to the auxiliary camera, the main camera may also be controlled to be in a power-on state. In yet another possible implementation, after switching from the main camera to the auxiliary camera, the main camera can also be controlled to be in the power-on state within a certain duration threshold, if the auxiliary camera is not switched back to the main camera within the duration threshold, then Control the power off of the main camera. Wherein, the duration threshold may be set according to actual requirements.

When switching back to the main camera from the auxiliary camera, if the main camera is powered off, it can be switched according to the above switching process; if the main camera is powered on, it can be switched directly, that is, there is no need to switch according to the above switching process .

In one embodiment, the external ISP stores a plurality of first object models corresponding to the auxiliary camera, and each first object model in the plurality of first object models corresponds to an exposure value range. In this case, after the second preprocessing sub-module of the built-in ISP synchronously processes the second exposure data, it feeds back the synchronized second exposure data to the external ISP. In this way, the external ISP determines the exposure value range to which the synchronized second exposure data belongs, and then selects the first target model corresponding to the determined exposure value range from a plurality of first target models, and the selected first target model is used for for the next pre-noise reduction process.

In the embodiment of the present application, during the focusing process, the camera to be started is started in advance, compared with direct switching, it can avoid the jamming problem caused by the spatial position difference between the cameras. In addition, after starting, synchronize the parameters of the auxiliary camera with the main camera, so that after switching, the picture effect of the video frame captured by the secondary camera is close to the picture effect of the video frame collected by the main camera, avoiding the large difference in the video picture before and after switching problem, achieved smooth switching between cameras.

For ease of understanding, the flow of the method provided by the embodiment of the present application will be introduced next with reference to the hardware framework diagram shown in FIG. 9 . As shown in FIG. 9 , the hardware involved in this embodiment of the present application mainly includes multiple cameras (for example, including a main camera and a secondary camera), a SOC, an external ISP, and a built-in ISP.

The external ISP includes multiple interfaces, a routing unit, a first external ISP front-end unit, a second external ISP front-end unit, and an external ISP back-end unit. The routing unit is connected to the first external ISP front-end unit and the second external The ISP front-end unit is connected, the first external ISP front-end unit is connected with the external ISP back-end unit, and the second external ISP front-end unit is connected with the external ISP back-end unit. The routing unit is used to perform the function of the routing submodule in each of the above-mentioned embodiments, the first external ISP front-end unit and the second external ISP front-end unit are used to perform the function of the first preprocessing sub-module in the above-mentioned various embodiments, and the external The ISP back-end unit is used to implement the functions of the enhanced sub-modules in each of the above embodiments. In one example, the first external ISP front-end unit is IFE0 in the external ISP, the second external ISP front-end unit is IFE1 in the external ISP, and the external ISP back-end unit is an NPU in the external ISP.

The built-in ISP includes a first built-in ISP front-end unit, a second built-in ISP front-end unit, a third built-in ISP front-end unit and a built-in ISP back-end unit. The first built-in ISP front-end unit is connected to the built-in ISP back-end unit, and the second built-in ISP front-end unit is connected to the built-in ISP back-end unit. The first built-in ISP front-end unit, the second built-in ISP front-end unit and the third built-in ISP front-end unit are used to execute the function of the second preprocessing submodule in each of the above-mentioned embodiments, and the built-in ISP back-end unit is used to execute the above-mentioned each embodiment Optimize the function of processing submodules in . In one example, the first built-in ISP front-end unit is IFE0 in the built-in ISP, the second built-in ISP front-end unit is IFE1 in the built-in ISP, the third built-in ISP front-end unit is IFE2 in the built-in ISP, and the built-in ISP back-end unit is Built-in IPE in ISP.

It should be noted that, the foregoing is only an exemplary illustration of the multiple units included in the external ISP and the internal ISP, but does not constitute a limitation to the structural components thereof. In some embodiments, the external ISP or the internal ISP may further include other units, which is not limited in this embodiment of the present application.

Next, in conjunction with the hardware frame diagram shown in Figure 9, the flow of the camera switching method provided by the embodiment of the present application is introduced, specifically:

901: The external ISP receives the first RAW data.

As an example of the present application, the first RAW data is from a main camera of the electronic device, specifically, an image sensor of the main camera outputs the first RAW data to an external ISP.

As an example but not a limitation, as shown in FIG. 9, the external ISP receives the first RAW data from the main camera through a mobile industry processor interface (mobile industry processor interface, Mipi)0.

902: The external ISP copies and routes the first RAW data through the routing unit.

As an example of the present application, the external ISP first copies the first RAW data through the routing unit to obtain the fifth RAW data. Afterwards, the routing unit performs routing processing on the two sets of RAW data. For example, the first RAW data is transmitted to the first external ISP front-end unit, and the first external ISP front-end unit performs preprocessing based on the first RAW data. And send the first RAW data obtained after preprocessing to the external ISP back-end unit, and the external ISP back-end unit performs fusion and noise reduction processing; the fifth RAW data is directly output to the built-in ISP.

For the preprocessing of the first RAW data by the first external ISP front-end unit, refer to the embodiment shown in FIG. 7 above. For the fusion and noise reduction processing of the preprocessed first RAW data by the external ISP back-end unit, please refer to the above Figure 7 shows the embodiment.

903: The external ISP back-end unit outputs the video data after noise reduction processing, and outputs the fifth RAW data through the routing unit.

Exemplarily, the external ISP back-end unit sends the noise-reduced video data to the built-in ISP through the Mipi0 interface of the external ISP, and the routing unit sends the fifth RAW data to the built-in ISP through the Mipi1 interface of the external ISP. Please refer to FIG. 9 , before the camera is switched, the external ISP processes the first RAW data of the main camera through the first branch 1 .

904: The built-in ISP receives the video data output by the external ISP back-end unit, and the fifth RAW data output by the routing unit.

In one example, the built-in ISP receives the video data output by the external ISP back-end unit through the first built-in ISP front-end unit, and then the first built-in ISP front-end unit processes the video data, such as preprocessing again, and then performing RGB Convert and compress the converted RGB image to obtain a YUV image. Then transmit the YUV image to the built-in ISP back-end unit for processing.

In an example, the built-in ISP receives the fifth RAW data output by the routing unit through the second built-in ISP front-end unit. After that, the second built-in ISP front-end unit determines the first exposure data based on the fifth RAW data, and determines whether the current exposure level is reasonable according to the first exposure data, and if not, determines the target exposure data, and adjusts the camera according to the target exposure data exposure parameters. In one example, the second built-in ISP front-end unit adjusts the exposure data of the camera through the I2C interface.

In addition, the second built-in ISP front-end unit also counts AWB, color and other data based on the fifth RAW data. The second built-in ISP front-end unit transmits data such as 3A, color, etc. to the built-in ISP back-end unit, so that the built-in ISP back-end unit can optimize the YUV image according to the data transmitted by the second built-in ISP front-end unit, such as denoising the YUV image processing, so as to obtain a clear and clear video frame.

In addition, the second built-in ISP front-end unit can also send the target exposure data to the external ISP through the peripheral interface, for example, to the external ISP back-end unit, so that the external ISP back-end unit can use the target exposure data from A third target model is selected from a plurality of third target models used for noise reduction processing, so that next video data is subjected to noise reduction processing according to the selected third target model. Exemplarily, the peripheral interface may be a secure digital input and output (secure digital input and output, SDIO) interface.

Further, the second built-in ISP front-end unit determines the first AF, the first AWB, and the first FOV based on the fifth RAW data, so as to obtain the first imaging information. For its specific implementation, reference may be made to the embodiment shown in FIG. 7 .

905: Built-in ISP output video frame.

Specifically, the built-in ISP outputs optimized video frames through the built-in ISP back-end unit, and displays the video frames on the display screen.

It should be noted that, before the switching operation is performed, the electronic device outputs the video frames of the main camera according to the above process. When the user adjusts the focus, perform the following switching operations:

906: In the process of video frame output, when the camera application program detects a focusing operation, before the camera switching condition is not met, the SOC starts the secondary camera.

In an example, when the user's focus operation is detected, before the camera switch condition is met, if the difference between the focus value corresponding to the focus operation and the target focus value corresponding to the camera switch condition is less than the preset value, the auxiliary camera is activated. Afterwards, the electronic device collects video data through the auxiliary camera to obtain second RAW data.

It should be noted that after the electronic device starts the auxiliary camera, the auxiliary camera starts to collect video data to obtain the second RAW data. Before switching, the video data of the main camera is still output, that is, the second RAW data collected by the auxiliary camera is not output. For the processing process of the second RAW data, please refer to the following steps.

907: The external ISP receives the second RAW data.

In an example, the external ISP receives the second RAW data through the Mipi1 interface.

908: The external ISP copies and routes the second RAW data through the routing unit.

In an example, the routing unit copies the second RAW data to obtain sixth RAW data. The routing unit sends the second RAW data to the second external ISP front-end unit, and sends the sixth RAW data to the built-in ISP through the Mipi2 interface.

909: The external ISP performs resolution reduction and frame rate reduction processing on the second RAW data through the second external ISP front-end unit to obtain fourth RAW data.

In an embodiment, before the second external ISP front-end unit reduces the resolution and frame rate of the second RAW data, it may also perform preprocessing on the second RAW data, which is not limited in this embodiment of the present application.

910: The second external ISP front-end unit sends the fourth RAW data to the external ISP back-end unit.

911: The external ISP back-end unit performs pre-noise reduction processing based on the fourth RAW data.

Please refer to FIG. 9 , for an external ISP, after the auxiliary camera is started and before the camera is switched, the second RAW data of the auxiliary camera is processed through the second branch 2 .

912: The built-in ISP receives sixth RAW data through the third built-in ISP front-end unit.

913: The third built-in ISP front-end unit determines second camera information based on the sixth RAW data.

914: The third built-in ISP front-end unit performs information synchronization on the first camera information and the second camera information.

In one example, the third built-in ISP front-end unit obtains the first camera information from the second built-in ISP front-end unit of the built-in ISP, and then adjusts the second camera information according to the first camera information, so as to combine the second camera information with the Information synchronization is performed on the first camera information.

915: The built-in ISP controls the secondary camera to adjust parameters according to the synchronized second camera information.

In one example, the third built-in ISP front-end unit controls the camera through the I2C interface to adjust parameters according to the synchronized second camera information.

In one embodiment, when the external ISP includes a plurality of first object models corresponding to the auxiliary camera, the built-in ISP synchronously processes the second exposure data, and feeds back the synchronized second exposure data to the external ISP, for example , can be fed back to the external ISP back-end unit. In this way, the external ISP back-end unit determines the exposure value range to which the synchronized second exposure data belongs, and then selects the first target model corresponding to the determined exposure value range from a plurality of first target models, and the selected first The target model is used for the next pre-noise reduction processing on the video data collected by the auxiliary camera.

916: When the camera switching condition is satisfied, the external ISP uses the third RAW data as data to be output.

The third RAW data is RAW data output by the secondary camera after parameter synchronization.

In an implementation, the external ISP copies the third RAW data through the routing unit. Afterwards, the routing unit sends the third RAW data to the first external ISP front-end unit, which is preprocessed by the first external ISP front-end unit, and then sent to the external ISP back-end unit, which is fused by the external ISP back-end unit And noise reduction processing, the video data after noise reduction processing is output to the built-in ISP through the Mipi0 interface for subsequent optimization processing. In addition, the external ISP sends the RAW data copied from the third RAW data to the built-in ISP through the routing unit, and the front-end unit of the third built-in ISP in the built-in ISP determines the exposure data.

Please refer to FIG. 9 , when the camera switching condition is satisfied, the external ISP processes the third RAW data of the secondary camera through the first branch 1 .

As an example but not a limitation, the present application provides a camera switching method according to another embodiment, and the method may be applied to the above-mentioned electronic device, where the electronic device includes at least a first camera and a second camera. In one embodiment, the electronic device includes a first image signal processing module and a second image signal processing module, for example, the second image signal processing module is an ISP integrated in the SOC (abbreviation: built-in ISP), and the first image signal processing module includes The ISP outside the SOC (referred to as: external ISP). The method may include the following implementation steps:

Step 1001: Output video frames according to the first RAW data of the first camera, where the first RAW data is original video data.

In one example, the first camera may be the main camera. In another example, the first camera may also be an auxiliary camera.

As an example of the present application, specific implementation of step 1001 may include: the image sensor of the first camera outputs first RAW data, and the first image signal processing module acquires the first RAW data. The first image signal processing module copies the first RAW data to obtain fifth RAW data. The first image signal processing module performs image enhancement processing on the first RAW data to obtain video enhancement data. The first image signal processing module sends the video enhancement data and the fifth RAW data to the second image signal processing module. The second image signal processing module outputs video frames based on the video enhancement data and the fifth RAW data.

As an example of the present application, the specific implementation of outputting the first RAW data by the image sensor of the first camera may include: detecting a night scene video shooting instruction through the camera application in the electronic device, and the night scene video shooting instruction is used to indicate that in the night scene mode Make a video recording. In response to the night scene video shooting instruction, the image sensor outputs first RAW data.

As an example of the present application, the first RAW data includes long exposure data and short exposure data collected in the same time period, and the first image signal processing module performs image enhancement processing on the first RAW data, including: the first image signal processing module The long-exposure data and short-exposure data are fused to obtain fused RAW data. The first image signal processing module performs noise reduction processing on the fused RAW data to obtain video enhancement data.

As an example of the present application, the first image signal processing module fuses the long exposure data and the short exposure data, including: the first image signal processing module inputs the long exposure data and the short exposure data into the second target model, and the The second target model performs fusion processing, and the second target model can perform fusion processing on arbitrary long exposure data and short exposure data.

As an example of the present application, the first image signal processing module performs noise reduction processing on the fused RAW data, including: the first image signal processing module inputs the fused RAW data into the third target model corresponding to the first camera, and the third The target model performs noise reduction processing, and the third target model can perform noise reduction processing on arbitrary RAW data.

As an example of the present application, the first image signal processing module includes multiple third target models corresponding to the first camera, and each third target model in the multiple third target models corresponds to an exposure value range. In this case, the first image signal processing module receives target exposure data, the target exposure data is determined by the second image signal processing module based on the first exposure data, and the first exposure data is determined by the second image signal processing module based on the fifth The RAW data is obtained through exposure data statistics, and the target exposure data is used to adjust the exposure parameters of the first camera. The first image signal processing module selects a third target model from a plurality of third target models according to the target exposure data and the exposure value range corresponding to each third target model, and the selected third target model is used for noise reduction processing .

As an example of the present application, the second image signal processing module performs format conversion processing on the video enhancement data to obtain a YUV image. The specific implementation may include: the second image signal processing module determines the target data based on the fifth RAW data, and the target data uses Used to adjust the image quality of YUV images. The second image signal processing module adjusts the YUV image based on the target data, and outputs the adjusted YUV image as a video frame.

Step 1002: During the process of video frame output, if a focus operation is detected, start the second camera before the camera switching condition is not met.

As an example of this application, if a focusing operation is detected, before the camera switching condition is reached, when the difference between the focusing value corresponding to the focusing operation and the target focusing value corresponding to the camera switching condition is less than or equal to When the value is preset, activate the second camera.

Step 1003, according to the first RAW data and the second RAW data of the second camera, synchronize the parameters of the second camera with the first camera.

In one example, the second camera can be a secondary camera. In another example, the second camera can also be the main camera. The second camera is different from the first camera.

As an example of the present application, specific implementation of step 1003 may include: determining first imaging information according to the first RAW data. Based on the second RAW data, second imaging information is determined. According to the first imaging information, the second imaging information is adjusted to synchronize the second imaging information with the first imaging information. According to the adjusted second camera information, the parameters of the second camera are adjusted, so as to synchronize the parameters of the second camera with the first camera.

Optionally, the first imaging information includes the first exposure data, the first autofocus AF, the first automatic white balance AWB, and the first field of view FOV, and the second imaging information includes the second exposure data, the second AF, the second AWB, second FOV.

As an example of the present application, the electronic device may also perform pre-noise reduction processing on the first target model corresponding to the second camera based on the second RAW data. The first target model can perform noise reduction processing based on arbitrary RAW data. The noise-processed first target model is used to perform noise reduction processing on the third RAW data after the focusing operation meets the camera switching condition.

As an example of the present application, based on the second RAW data, the specific implementation of performing pre-noise reduction processing through the first target model corresponding to the second camera may include: performing resolution reduction processing on the second RAW data according to a first preset ratio , and perform frame rate reduction processing on the second RAW data after the resolution reduction processing according to the second preset ratio to obtain the fourth RAW data, and input the fourth RAW data into the first target model, so that the first target model Perform pre-noise reduction processing.

Both the first preset ratio and the second preset ratio can be set according to actual needs.

In another embodiment, the second RAW data may be subjected to frame rate reduction processing according to the second preset ratio, and then the resolution processing may be performed on the second RAW data after the frame rate reduction processing according to the first preset ratio. .

Step 1004 , when the focusing operation satisfies the camera switching condition, output the video frame according to the third RAW data, the third RAW data is the raw video data of the second camera whose parameters have been synchronized.

In the embodiment of the present application, the video frame output is performed according to the first RAW data of the first camera. During the video frame output process, if a focusing operation is detected, the second camera is started before the camera switching condition is met. According to the first RAW data and the second RAW data of the second camera, the parameters of the second camera are synchronized with the first camera. When the focusing operation satisfies the camera switching condition, video frame output is performed according to the third RAW data, and the third RAW data is the raw video data of the second camera whose parameters have been synchronized. In this way, by starting the second camera in advance, and synchronizing the parameters of the second camera and the first camera after starting, the difference between the video picture of the second camera and the video picture of the first camera after switching can be small, that is, to solve It solves the problem of large difference in video images before and after switching, thus realizing smooth switching between cameras.

It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used for illustration. In practical applications, the above-mentioned functions can be assigned to different functional units, Completion of modules means that the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated into one processing unit, or each unit may exist separately physically, or two or more units may be integrated into one unit, and the above-mentioned integrated units may adopt hardware It can also be implemented in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the above system, reference may be made to the corresponding process in the foregoing method embodiments, and details will not be repeated here.

In the above-mentioned embodiments, the descriptions of each embodiment have their own emphases, and for parts that are not detailed or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In the embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the procedures in the methods of the above embodiments in the present application can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a computer-readable storage medium. The computer program When executed by a processor, the steps in the above-mentioned various method embodiments can be realized. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form. The computer readable medium may include at least: any entity or means capable of carrying computer program codes to electronic equipment, recording media, computer memory, ROM, RAM, electrical carrier signals, telecommunication signals, and software distribution media. Such as U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunication signals under legislation and patent practice.

Finally, it should be noted that: the above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application shall be covered by this application. within the scope of the application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims

A camera switching method, characterized in that it is applied to an electronic device, the electronic device includes a first camera and a second camera, and the method includes:

performing video frame output according to the first raw data of the first camera;

In the process of video frame output, if a focusing operation is detected, before the camera switching condition is not met, start the second camera;

Synchronize parameters of the second camera with the first camera according to the first raw data and the second raw data of the second camera;

When the focusing operation satisfies the camera switching condition, the video frame output is performed according to the third original data, and the third original data is the original video data of the second camera whose parameters are synchronized.
The method according to claim 1, wherein, according to the first raw data and the second raw data of the second camera, the parameter synchronization of the second camera and the first camera is performed, include:

determining first imaging information according to the first raw data;

determining second imaging information according to the second raw data;

adjusting the second imaging information according to the first imaging information to synchronize the second imaging information with the first imaging information;

According to the adjusted second camera information, the parameters of the second camera are adjusted, so as to synchronize the parameters of the second camera with the first camera.
The method according to claim 2, wherein the first imaging information includes the first exposure data, the first autofocus AF, the first automatic white balance AWB, and the first field of view FOV, and the second imaging information includes: The information includes second exposure data, second AF, second AWB, and second FOV.
The method according to any one of claims 1-3, further comprising:

Based on the second raw data, pre-noise reduction processing is performed on the first target model corresponding to the second camera, the first target model can perform noise reduction processing on any raw data, and all pre-noise reduction processing The first target model is used to perform noise reduction processing on the third raw data after the focusing operation satisfies the camera switching condition.
The method according to claim 4, wherein, based on the second raw data, performing pre-noise reduction processing through the first target model corresponding to the second camera includes:

Performing resolution reduction processing on the second original data according to a first preset ratio, and performing frame rate reduction processing on the second original data after resolution reduction processing according to a second preset ratio, to obtain fourth original data ;

The fourth original data is input into the first object model to perform pre-noise reduction processing on the first object model.
The method according to claim 1, wherein, in the process of outputting the video frame, if a focusing operation is detected, before the camera switching condition is not met, starting the second camera includes:

In the process of video frame output, if a focusing operation is detected, before the camera switch condition is reached, when the focus value corresponding to the focus operation is between the target focus value corresponding to the camera switch condition When the difference between them is less than or equal to the preset value, start the second camera.
The method according to any one of claims 1-6, wherein the electronic device includes a first image signal processing module and a second image signal processing module;

The video frame output according to the first raw data of the first camera includes:

The image sensor of the first camera outputs the first raw data;

The first image signal processing module acquires the first raw data;

The first image signal processing module copies the first original data to obtain fifth original data;

The first image signal processing module performs image enhancement processing on the first raw data to obtain video enhancement data;

The first image signal processing module sends the video enhancement data and the fifth original data to the second image signal processing module;

The second image signal processing module outputs video frames based on the video enhancement data and the fifth original data.
The method according to claim 7, wherein the first raw data includes long exposure data and short exposure data collected in the same time period, and the first image signal processing module performs processing on the first raw data Image enhancement processing to obtain video enhancement data, including:

The first image signal processing module fuses the long exposure data and the short exposure data to obtain fused original data;

The first image signal processing module performs noise reduction processing on the fused original data to obtain the enhanced video data.
The method according to claim 8, wherein the first image signal processing module fuses the long exposure data and the short exposure data, including:

The first image signal processing module inputs the long-exposure data and the short-exposure data into a second object model, and performs fusion processing by the second object model, and the second object model can Exposure data and short exposure data are fused.
The method according to claim 8 or 9, wherein the first image signal processing module performs noise reduction processing on the fused original data, including:

The first image signal processing module inputs the fused raw data into a third object model corresponding to the first camera, and performs noise reduction processing by the third object model, and the third object model can perform any The original data is subjected to noise reduction processing.
The method according to claim 10, wherein the first image signal processing module includes a plurality of third object models corresponding to the first camera, and each of the third object models in the plurality of third object models The three-target model corresponds to an exposure value range; the method also includes:

The first image signal processing module receives target exposure data, the target exposure data is determined by the second image signal processing module based on the first exposure data, and the first exposure data is determined by the second image signal processing module The module obtains exposure data statistics based on the fifth raw data, and the target exposure data is used to adjust the exposure parameters of the first camera;

The first image signal processing module selects a third target model from the plurality of third target models according to the target exposure data and the exposure value range corresponding to each third target model, and the selected third target model A three-objective model is used for noise reduction processing.
The method according to any one of claims 7-11, wherein the second image signal processing module performs video frame output based on the video enhancement data and the fifth original data, comprising:

The second image signal processing module performs format conversion processing on the video enhancement data to obtain a YUV image;

The second image signal processing module determines target data based on the fifth raw data, and the target data is used to adjust the image quality of the YUV image;

The second image signal processing module adjusts the YUV image based on the target data, and outputs the adjusted YUV image as the video frame.
The method according to any one of claims 7-12, wherein the second image signal processing module includes an ISP integrated in a chip-level system SOC, and the first image signal processing module includes an ISP outside the SOC .
The method according to any one of claims 7-13, wherein the image sensor of the first camera outputs the first raw data, comprising:

A night scene video shooting instruction is detected through the camera application in the electronic device, and the night scene video shooting instruction is used to indicate video recording in night scene mode;

In response to the night scene video shooting instruction, the image sensor outputs the first raw data.
A camera switching device, characterized in that it is configured in an electronic device, the electronic device includes a first camera and a second camera; the device includes an image sensor node, a first image signal processing module, and a second image signal processing module processing module;

The first image signal processing module and the second image signal processing module are configured to output video frames according to the first raw data of the first camera;

The image sensor node is used to start the second camera before the camera switching condition is met if a focusing operation is detected during the video frame output process;

The first image signal processing module and the second image signal processing module are configured to, according to the first raw data and the second raw data of the second camera, combine the second camera with the first The camera performs parameter synchronization;

The first image signal processing module and the second image signal processing module are configured to output video frames according to third raw data when the focusing operation satisfies the camera switching condition, and the third raw data is the raw video data of the second camera after parameter synchronization.
An electronic device, characterized in that the electronic device includes a memory and a processor;

The memory is used to store a program that supports the electronic device to execute the method described in any one of claims 1-14, and to store data involved in implementing the method described in any one of claims 1-14; The processor is configured to execute programs stored in the memory.
A computer-readable storage medium, with instructions stored in the computer-readable storage medium, is characterized in that, when the computer-readable storage medium is run on a computer, it causes the computer to execute the method described in any one of claims 1-14.