WO2023077938A1

WO2023077938A1 - Video frame generation method and apparatus, electronic device, and storage medium

Info

Publication number: WO2023077938A1
Application number: PCT/CN2022/116757
Authority: WO
Inventors: 侯伟龙; 金杰; 李子荣
Original assignee: 荣耀终端有限公司
Priority date: 2021-11-05
Filing date: 2022-09-02
Publication date: 2023-05-11
Also published as: CN116095509A; CN116095509B

Abstract

The present application relates to the technical field of terminals, and discloses a video frame generation method and apparatus, an electronic device, and a storage medium. The method comprises: an image sensor outputs first raw data; a first image signal processing module obtains the first raw data; the first image signal processing module copies the first raw data to obtain second raw data; the first image signal processing module performs image enhancement processing on the first raw data to obtain video enhancement data; the first image signal processing module sends the video enhancement data and the second raw data to a second image signal processing module; and the second image signal processing module generates a video frame on the basis of the video enhancement data and the second raw data. According to the present application, the image enhancement processing is performed by means of the first image signal processing module, and the first image signal processing module further provides, for the second image signal processing module, the second raw data that can be used for adjusting an exposure parameter, so that it can be ensured that a clear video frame can be obtained.

Description

Method, device, electronic device and storage medium for generating video frames

This application claims the priority of the Chinese patent application submitted to the State Intellectual Property Office on November 5, 2021, with the application number 202111316825.0, and the application name is "Method for Generating Video Frames, Electronic Devices, and Storage Medium", and claims that it will be filed in 2022 The priority of a Chinese patent application filed with the State Intellectual Property Office on March 10, 202210254250.2 and titled "Method, device, electronic device, and storage medium for generating video frames", the entire contents of which are incorporated by reference in In this application.

technical field

The present application relates to the technical field of terminals, and in particular to a method, device, electronic equipment and storage medium for generating video frames.

Background technique

With the rapid development of terminal technology, the shooting level of electronic devices such as mobile phones is gradually improving. Especially in night scenes, when electronic equipment is shooting images, it can ensure that the highlights are not overexposed and the darks are not underexposed, which means that the details of the captured images are very clear regardless of the highlights or dark parts.

In related technologies, in order to achieve the above effects, the electronic device takes images through the camera and processes them through the image signal processor (Image Signal Processing, ISP) (hereinafter referred to as the built-in ISP) integrated in the system on chip (System on Chip, SOC). Take an image, perform image enhancement processing on the captured image, including but not limited to exposure parameter adjustment, white balance, focus, noise reduction and sharpening, etc., and then obtain a captured image with clear details after multi-frame processing.

However, in the case of video capture by electronic equipment, due to the real-time nature of video, complex multi-frame enhancement processing algorithms similar to those of taking photos cannot be used, and the display effect of video images is often significantly worse than that of captured images.

Contents of the invention

The present application provides a method, device, electronic equipment and storage medium for generating video frames, which solves the problem that complex multi-frame enhancement processing algorithms similar to photographing cannot be used in the prior art, and the display effect of video images is often significantly worse than that of captured images. question.

In order to achieve the above object, the application adopts the following technical solutions:

In a first aspect, a method for generating a video frame is provided, which is applied to an electronic device, and the electronic device includes an image sensor, a first image signal processing module, and a second image signal processing module, and the method includes:

The image sensor outputs first raw data;

The first image signal processing module acquires the first raw data;

The first image signal processing module copies the first original data to obtain second original data;

The first image signal processing module performs image enhancement processing on the first raw data to obtain video enhancement data;

The first image signal processing module sends the video enhancement data and the second original data to the second image signal processing module;

The second image signal processing module generates a video frame based on the video enhancement data and the second original data.

In this way, image enhancement processing is performed by the first image signal processing module, and the first image signal processing module also provides the second image signal processing module with second raw data that can be used to adjust exposure parameters, thus ensuring that clear video frame.

As an example of the present application, the first raw data includes long exposure data and short exposure data collected in the same time period, and the first image signal processing module performs image enhancement processing on the first raw data, including:

The first image signal processing module fuses the long exposure data and the short exposure data to obtain fused original data;

The first image signal processing module performs noise reduction processing on the fusion original data.

In this way, the long-exposure data and the short-exposure data within the same time period can be fused subsequently to output high dynamic video frames.

As an example of the present application, the first image signal processing module fuses the long exposure data and the short exposure data, including:

The first image signal processing module inputs the long-exposure data and the short-exposure data into a first object model, and performs fusion processing by the first object model, and the first object model can Exposure data and short exposure data are fused.

In this way, fusion processing is performed through the first target model, which can improve fusion efficiency.

As an example of the present application, the first image signal processing module performs noise reduction processing on the fused raw data, including:

The first image signal processing module inputs the fused raw data into a second target model, and the second target model performs noise reduction processing, and the second target model can perform noise reduction processing on any raw data .

In this way, noise reduction efficiency can be improved by using the second target model to perform noise reduction processing.

As an example of the present application, the first image signal processing module includes a plurality of second target models, and each second target model in the plurality of second target models corresponds to an exposure value range;

The method also includes:

The first image signal processing module receives target exposure data, the target exposure data is determined by the second image signal processing module based on the first exposure data, and the first exposure data is determined by the second image signal processing module The module obtains exposure data statistics based on the second raw data, and the target exposure data is used to adjust the exposure parameters of the image sensor;

The first image signal processing module selects a second target model from the plurality of second target models according to the target exposure data and the exposure value range corresponding to each second target model, and the selected second target model A two-objective model is used for denoising.

In this way, according to the exposure value range to which the target exposure data belongs, a second target model for the next noise reduction process is selected from multiple second target models, so that a reasonable noise reduction process can be performed on the next video data , thereby improving the noise reduction effect.

As an example of the present application, before the first image signal processing module fuses the long exposure data and the short exposure data, it further includes:

The first image signal processing module preprocesses the long exposure data and the short exposure data, and the preprocessing includes lens correction LSC processing, black level compensation BLC processing, bad pixel correction BPC processing, and color interpolation processing at least one of the

The first image signal processing module fuses the long exposure data and the short exposure data, including:

The first image signal processing module fuses the preprocessed long exposure data and the short exposure data.

In this way, by performing preprocessing on the long-exposure data and the short-exposure data respectively, the definition of subsequent video frames can be improved.

As an example of the present application, the second image signal processing module generates a video frame based on the video enhancement data and the second original data, including:

The second image signal processing module performs format conversion processing on the video enhancement data to obtain a YUV image;

The second image signal processing module determines target data based on the second raw data, and the target data is used to adjust the image quality of the YUV image;

The second image signal processing module adjusts the YUV image based on the target data, and uses the adjusted YUV image as the video frame.

In this way, the format conversion process is performed on the video enhancement data through the second image signal processing module, and the target data is determined based on the second raw data, and the YUV image obtained after the format conversion process is optimized according to the target data, so as to obtain a clear picture video frame.

As an example of the present application, the image sensor outputs first raw data, including:

A night scene video shooting instruction is detected through the camera application in the electronic device, and the night scene video shooting instruction is used to indicate video recording in night scene mode;

In response to the night scene video shooting instruction, the image sensor outputs the first raw data.

In this way, in the night scene, the electronic device acquires the first raw data, and processes the first raw data collected by the camera through the method provided in this application, so that the highlighted area of the obtained video frame will not be overexposed and the darkened area will not be overexposed. Not too dark, resulting in a clear video frame.

As an example of the present application, the second image signal processing module includes an ISP integrated in a system-on-a-chip (SOC), and the first image signal processing module includes an ISP outside the SOC.

In this way, by sharing the task of processing video frames through an external ISP, the burden of the built-in ISP in the SOC can be reduced, thereby realizing real-time processing of video frames, so that video images that meet the requirements can be obtained.

In a second aspect, a device for generating a video frame is provided, the device comprising: an image sensor node, a first image signal processing module, and a second image signal processing module;

The image sensor node is configured to output first raw data;

The first image signal processing module is configured to acquire the first original data, copy the first original data to obtain second original data, and perform image enhancement processing on the first original data to obtain video enhanced data, sending the enhanced video data and the second raw data to the second image signal processing module;

The second image signal processing module is configured to generate a video frame based on the video enhancement data and the second original data.

As an example of the present application, the first raw data includes long exposure data and short exposure data collected in the same time period; the first image signal processing module is used for:

performing fusion processing on the long exposure data and the short exposure data to obtain fusion original data;

Perform noise reduction processing on the fused raw data.

As an example of the present application, the first image signal processing module is used for:

Inputting the long exposure data and the short exposure data into a first target model, and performing fusion processing by the first target model, and the first target model can fuse any long exposure data and short exposure data deal with.

The fused original data is input into a second target model, and the noise reduction process is performed by the second target model, and the second target model can perform noise reduction process on any original data.

As an example of the present application, the first image signal processing module includes a plurality of second target models, and each second target model in the plurality of second target models corresponds to an exposure value range; the first Image signal processors are also used to:

receiving target exposure data, the target exposure data is determined by the second image signal processing module based on the first exposure data, the first exposure data is determined by the second image signal processing module based on the second raw data Obtained by performing statistics on exposure data, the target exposure data is used to adjust the exposure parameters of the image sensor;

According to the target exposure data and the exposure value range corresponding to each second target model, select a second target model from the plurality of second target models, and the selected second target model is used for noise reduction processing .

Preprocessing the long exposure data and the short exposure data, the preprocessing includes at least one of lens correction LSC processing, black level compensation BLC processing, bad pixel correction BPC processing, and color interpolation processing;

The preprocessed long exposure data and the short exposure data are fused.

As an example of the present application, the second image signal processing module is used for:

Perform format conversion processing on the video enhancement data to obtain a YUV image;

determining target data based on the second raw data, the target data being used to adjust the image quality of the YUV image;

adjusting the YUV image based on the target data, and using the adjusted YUV image as the video frame.

As an example of the present application, the image sensor node is used for:

In response to the night scene video shooting instruction, the first raw data is output.

As an example of the present application, the second image signal processing module is an ISP integrated in a system-on-a-chip (SOC), and the first image signal processing module is an ISP outside the SOC.

In the third aspect, an electronic device is provided, the structure of the electronic device includes a processor and a memory, and the memory is used to store a program that supports the electronic device to execute the method described in any one of the above-mentioned first aspects, and to store a program for The data involved in implementing the method described in any one of the above first aspects; the processor is configured to execute the program stored in the memory. The electronic device may also include a communication bus for establishing a connection between the processor and the memory.

In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer is made to execute the method described in any one of the above-mentioned first aspects.

According to a fifth aspect, there is provided a computer program product containing instructions, which, when run on a computer, causes the computer to execute the method described in the first aspect above.

The technical effects obtained by the above-mentioned second aspect, third aspect, fourth aspect and fifth aspect are similar to those obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here.

Description of drawings

FIG. 1 is a schematic diagram of spatial position distribution of a camera provided in an embodiment of the present application;

FIG. 2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of a software framework of an electronic device provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of another application scenario provided by the embodiment of the present application;

FIG. 6 is a schematic flowchart of a method for generating a video frame provided in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a hardware provided by an embodiment of the present application;

FIG. 8 is a schematic flowchart of another method for generating video frames provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the implementation manner of the present application will be further described in detail below in conjunction with the accompanying drawings.

It should be understood that the "plurality" mentioned in this application means two or more. In the description of this application, unless otherwise specified, "/" means or means, for example, A/B can mean A or B; "and/or" in this article is just a description of the relationship between associated objects, It means that there can be three kinds of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in order to clearly describe the technical solution of the present application, words such as "first" and "second" are used to distinguish the same or similar items with basically the same function and effect. Those skilled in the art can understand that words such as "first" and "second" do not limit the number and execution order, and words such as "first" and "second" do not necessarily limit the difference.

Reference to "one embodiment" or "some embodiments" or the like in the specification of the present application means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically stated otherwise. The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless specifically stated otherwise.

Before introducing the method provided by the embodiment of the present application in detail, the nouns and execution subjects involved in the embodiment of the present application are firstly introduced.

First, the nouns involved in the embodiments of the present application are introduced.

Exposure: According to the length of exposure time, it can be divided into long exposure and short exposure. The longer the exposure time, the greater the amount of light entering the aperture. Conversely, the shorter the exposure time, the smaller the amount of light entering the aperture.

3A statistical algorithm: including automatic exposure (automatic exposure, AE) algorithm, automatic focus (automatic focus, AF) algorithm, and automatic white balance (automatic white balance, AWB) algorithm.

AE: refers to the camera automatically determines the exposure according to the lighting conditions. The imaging system generally has an AE function, which is directly related to the brightness and image quality of the image screen, that is, it will determine the brightness and darkness of the image.

AF: It means that the camera automatically adjusts the focus distance of the camera according to the distance of the subject, that is, the lens in the camera is adjusted to form the focus through distance measurement, so that the image in the camera is clear.

AWB: Mainly used to solve the problem of image color cast. If there is a color cast in the image, it can be corrected by the AWB algorithm.

Field of view: (field angle, FOV), refers to the range that the camera can cover. The larger the FOV, the more scenes the camera can accommodate. It is not difficult to understand that if the subject is not within the FOV of the camera, it will not be captured by the camera.

Image sensor (Sensor): It is the core component of the camera, and its function is to convert optical signals into electrical signals for subsequent processing and storage. The working principle is that the photosensitive element generates charge under the condition of light, and the charge transfer generates a current, and the current is rectified and amplified, and converted into a digital signal to form a digital signal. Image sensors generally include two types: charge coupled device (CCD) and complementary metal oxide semiconductor (complementary metal oxide semiconductor, CMOS).

RAW data: also referred to as raw data in this embodiment of the application, refers to the raw data that the CCD or CMOS image sensor in the camera converts the captured light source signal into a data signal. That is to say, it can be understood as unprocessed data, which can be used to describe the intensity of various lights received by the image sensor.

Next, the execution subject involved in the embodiment of the present application will be introduced.

The method provided in the embodiment of the present application may be executed by an electronic device having a shooting function. The electronic device is configured with one or more cameras, and different cameras among the multiple cameras have different shooting functions. For example, the electronic device is configured with at least one of a wide-angle camera, a telephoto camera (such as a periscope telephoto camera), a black-and-white camera, and an ultra-wide-angle camera. The one or more cameras may include a front camera and/or a rear camera. As an example of the present application, the electronic device is equipped with multiple rear cameras, and the multiple rear cameras include a main camera and at least one auxiliary camera. For example, please refer to FIG. 1, the spatial distribution of multiple rear cameras can be shown in FIG. 1 As shown in figure (a) in Figure 1, or, the spatial distribution of multiple rear cameras can also be shown in figure (b) in Figure 1, the multiple rear cameras are respectively camera 00, camera 01, camera 02, The camera 03, for example, the camera 00 is the main camera, and the others are auxiliary cameras. After starting the camera application, the electronic device usually shoots through the main camera by default. After the camera is switched, the electronic device selects a suitable auxiliary camera from at least one auxiliary camera according to the switching requirement, and shoots through the selected auxiliary camera. By way of example and not limitation, an electronic device may be, but not limited to, a mobile phone action camera (GoPro), digital camera, tablet computer, desktop, laptop, handheld computer, notebook computer, vehicle-mounted device, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook, cellular phone, personal digital assistant (personal digital assistant, PDA), augmented reality (augmented reality, AR)\virtual reality (virtual reality, VR) equipment, mobile phone, etc. Not limited.

Please refer to FIG. 2 . FIG. 2 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2. Mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, ISP191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like. In one example, the number of ISPs 191 included in the electronic device is multiple, and only one is exemplarily shown in FIG. 2 .

It can be understood that the structure shown in this embodiment does not constitute a specific limitation on the electronic device 100 . In other embodiments, the electronic device 100 may include more or fewer components than shown, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an ISP, a controller , memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

Wherein, the controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 is charging the battery 142 , it can also supply power to the electronic device 100 through the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be disposed in the processor 110 . In some other embodiments, the power management module 141 and the charging management module 140 may also be set in the same device.

The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

The electronic device 100 can realize the shooting function through the ISP 191, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

The ISP 191 is used for processing the data fed back by the camera 193. For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP 191 for processing, and converts it into an image visible to the naked eye. ISP 191 can also optimize the algorithm for image noise, brightness, and skin tone. ISP 191 can also optimize the exposure, color temperature and other parameters of the shooting scene.

In some embodiments, the ISP 191 may include a built-in ISP integrated in the SOC and an external ISP provided outside the SOC. In comparison, the internal structure of the external ISP is similar or the same as that of the built-in ISP. The difference is that the methods used by the external ISP and the built-in ISP to process video data are different. As an example of this application, the external ISP uses artificial intelligence methods (such as through network models, etc.) to process video data, while the built-in ISP uses other algorithms to process video data.

As an example of this application, the external ISP mainly has two functions: on the one hand, it is used to perform fusion processing and image enhancement processing on the RAW data collected by the camera, so as to provide enhanced video data for the built-in ISP; on the other hand, It is used to route the RAW data collected by the camera to provide a copy of RAW data for the built-in ISP, so that the built-in ISP can accurately determine the current exposure data, and then facilitate the built-in ISP to dynamically adjust the exposure parameters of the camera according to the exposure data.

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP 191 for conversion into a digital image signal. ISP 191 outputs the digital image signal to DSP for processing. DSP converts digital image signals into standard RGB (red green blue red green blue), YUV and other image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.

The internal memory 121 may be used to store computer-executable program codes including instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 . The internal memory 121 may include an area for storing programs and an area for storing data.

Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.

The electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .

The earphone interface 170D is used for connecting wired earphones. The earphone interface 170D may be a USB interface 130, or a 3.5mm open mobile terminal platform (open mobile terminal platform, OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 . The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like. The SIM card interface 195 is used for connecting a SIM card. The SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card etc.

In one embodiment, the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the software structure of the electronic device 100 is exemplarily described by taking an Android system with a layered architecture as an example.

FIG. 3 is a block diagram of the software structure of the electronic device 100 provided by the embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is respectively an application program layer, a hardware abstract layer (Hardware Abstract Layer, HAL), a kernel layer, and a hardware layer from top to bottom. In addition, an application framework layer (Application Framework) (not shown in FIG. 3 ) is also included between the application layer and the HAL, which is not highlighted in this embodiment of the present application.

The application layer can consist of a series of application packages. As shown in Figure 3, the application package may include application programs such as a camera and a gallery.

As an example of the present application, the camera application supports a super night video mode, and in the super night video mode, the electronic device can shoot clear, bright and dark videos in night scenes.

As an example of this application, the application layer also provides pre-loaded external ISP services. Since the internal memory of the external ISP is usually random access memory (Random Access Memory, RAM), according to the characteristics of RAM, RAM cannot save data in the case of power failure, so the external ISP is usually stored during operation. The required data such as external ISP SDK and models (for example, including the first target model and the second target model described below) are stored in the system memory. When the camera application is started, the application layer starts the preloading external ISP service, so that the external ISP driver controls the external ISP to be powered on in advance, and loads the data required for the external ISP during operation from the system memory to the external ISP The internal RAM is convenient for the external ISP to perform corresponding functions (such as data fusion, image enhancement processing, etc.) in the super night video mode.

As an example of this application, the video recorded by the camera can be provided in the gallery application, so that the user can view the recorded video from the gallery application.

The HAL layer mainly includes a video module, which is used to obtain RAW data through the image sensor of the camera, and perform fusion, enhancement, optimization and other processing on the RAW data through the external ISP and the built-in ISP respectively to obtain video with enhanced clarity and noise reduction effect frame. The resulting video frames are then sent to the display for display. In addition, the video module also saves the recorded video to the gallery application for easy viewing by users.

In one example, the video module includes an image sensor node, a built-in ISP node, and an external ISP node. Each node can be understood as an encapsulation of the functions performed by the underlying hardware, which can be perceived and invoked by the upper layer (application layer). Exemplarily, the image sensor node encapsulates the function of the image sensor in the bottom camera; the built-in ISP node encapsulates the function of the bottom built-in ISP; the external ISP node encapsulates the function of the bottom external ISP. In implementation, the video module implements the above functions through the interaction between the image sensor node, the built-in ISP node, and the external ISP node.

The external ISP node includes a plurality of submodules, for example, a routing submodule, a first preprocessing submodule, and an enhancement submodule. In the same way, each sub-module in the multiple sub-modules can be understood as the encapsulation of the functions of different hardware in the bottom external ISP respectively. As an example of this application, the routing sub-module is the function of the SIF in the bottom external ISP. Encapsulation, the first preprocessing sub-module is the encapsulation of the function of the IFE in the bottom external ISP, and the enhancement sub-module is the encapsulation of the function of the neural network processing unit (NPU) in the bottom external ISP. In implementation, the external ISP implements corresponding functions through the interaction of multiple sub-modules.

The interior of the built-in ISP node includes multiple submodules, for example, the second preprocessing submodule and the optimization processing submodule. Each of the multiple sub-modules can be understood as encapsulating the functions of different hardware in the underlying built-in ISP. As an example of this application, the second preprocessing sub-module is for one or more image front-ends in the built-in ISP Encapsulation of the function of the engine (image front end, IFE); the optimization processing sub-module is the encapsulation of the function of the image processor (image processing engine, IPE) in the built-in ISP in the bottom layer. In implementation, the built-in ISP realizes the corresponding functions through the interaction of multiple sub-modules.

In addition, the HAL layer also includes an external ISP software development kit (software development kit, SDK), which is used to establish the interaction between multiple sub-modules inside the external ISP node.

The kernel layer is the layer between hardware and software. The kernel layer includes but not limited to camera driver, built-in ISP driver, and external ISP driver.

The hardware layer includes but not limited to camera, built-in ISP, external ISP, display.

In the following, the workflow of the software and hardware of the electronic device 100 will be exemplarily described in conjunction with video recording in the night scene.

In one embodiment, if the camera application detects that video shooting is enabled in the super night video mode, it sends a notification to the video module at the HAL layer. After the video module receives the notification, it establishes a framework for processing the night scene video. Specifically, the video module notifies the camera driver to control the camera to be powered on, and notifies the built-in ISP driver to control the built-in ISP to be powered on. Correspondingly, the camera driver drives the camera, and after the camera is loaded, the camera driver is notified, and the camera driver notifies the video module that the camera has been loaded. In addition, the built-in ISP driver drives the built-in ISP, and after the built-in ISP is loaded, the built-in ISP driver is notified, and the built-in ISP driver notifies the video module that the built-in ISP has been loaded.

After the video module determines that the camera, built-in ISP and external ISP (for example, the loading time is after the camera application starts) are all loaded, set up the interaction between the image sensor node, the built-in ISP node and the external ISP node . In this way, the video data can be collected and processed by calling the video module, and the specific implementation can be referred to below.

After introducing the nouns and execution subjects involved in the embodiments of the present application, the application scenarios involved in the embodiments of the present application will be introduced next by taking the electronic device as a mobile phone including multiple rear cameras as an example.

There is a camera application in the mobile phone, please refer to the (a) figure in Figure 4, in one embodiment, the user wants to take a night scene video through the mobile phone, at this time, the user can click the application icon of the camera application in the mobile phone. In response to the user's trigger operation on the application icon of the camera application, the mobile phone starts the main camera in the rear camera, and displays the first interface shown in (b) of FIG. 4 to the user.

As an example of the present application, a "night scene" option 41 is provided in the first interface, the user can trigger the "night scene" option 41, and in response to the user's trigger operation on the "night scene" option 41, the mobile phone displays the operation interface in the night scene mode ( referred to as the second interface), for example, the second interface is shown in (c) in FIG. 4 . The second interface provides a first switching option 42 and a second switching option 43, wherein the first switching option 43 is used to switch between the front camera and the rear camera. The second switch option 43 is used to switch between the camera mode and the video capture mode.

In one example, after entering the night scene mode, that is, after switching from (b) in FIG. 4 to (c) in FIG. Example), when the user wants to shoot a night scene video, the second switch option 43 can be triggered, and in response to the trigger operation of the second switch option 43 by the user, the mobile phone switches from the camera mode to the video capture mode.

In another example, after entering the night scene mode, that is, after switching from (b) in Figure 4 to (c) in Figure 4, the mobile phone can also be in the video shooting mode by default, in this case , if the user wants to take a night scene image, the second switch option 43 can be triggered, and in response to the trigger operation of the second switch option 43 by the user, the mobile phone switches from the video shooting mode to the camera mode.

In one embodiment, a shooting option 44 is also provided in the second interface, and the user can trigger the shooting option 44 . In response to the user's trigger operation on the shooting option 44, the mobile phone records a video through a camera (such as a main camera). For example, please continue to refer to FIG. 4 , and the video recording interface is shown in (d) in FIG. 4 . As an example of the present application, in a night scene, the mobile phone processes the video data collected by the camera through the method provided in the present application, so that a clear video frame can be captured finally. The clarity of the picture mentioned here means that the highlighted areas will not be overexposed, and the dark areas will not be too dark.

In an example, please continue to refer to figure (d) in FIG. 4 , a pause option 45 is provided in the video recording interface. During the video recording process, when the user wants to pause the video recording, the pause option 45 can be triggered, and in response to the user's trigger operation on the pause option 45, the mobile phone pauses the video recording.

In an example, please continue to refer to figure (d) in FIG. 4 , a snapshot option 46 is provided in the video recording interface. During the video recording process, when the user wants to capture a certain video frame, the capture option 46 can be triggered. In response to the trigger operation of the capture option 46 by the user, the mobile phone performs a capture operation and stores the captured video frame.

In one example, please continue to refer to figure (d) in FIG. 4 , the video recording interface also provides a focusing item 47 for focusing. During the video recording process, when the user wants to focus, the focus adjustment item 47 can be triggered, such as adjusting from 1x focus to telephoto, such as adjusting to multi-fold focus (such as 2x focus), or from 1x focus to wide-angle Adjustment, such as adjusting to 0.8 times focus. In response to the user's trigger operation on the focusing item 47, the mobile phone focuses on the main camera, or switches to other auxiliary cameras for video collection. For example, when the user adjusts from 1x focus to nx focus, when n is greater than 1 and less than 2, the mobile phone will focus on the main camera, and when n is greater than or equal to 2, the mobile phone will switch from the main camera to other auxiliary cameras (such as switching to the telephoto camera). For another example, when the user adjusts from 1x focus to wide-angle, the mobile phone switches from the main camera to the wide-angle camera. In the embodiment of the present application, no matter before or after the focus adjustment, the video data collected by the camera currently performing video recording can be processed, so that a video frame with a clear picture can be finally obtained.

Please refer to FIG. 5 , there is a "more" option 51 in the first interface. As an example of the present application, when the user wants to shoot a night scene video, the "more" option 51 can be triggered. In response to the user's trigger operation on the "more" option 51, the mobile phone displays a third interface, for example, the third interface is shown in (b) of FIG. 5 . As an example of the present application, a "night scene recording" option 52 is provided in the third interface, and the "night scene recording" option 52 is used to trigger the video recording function under the night scene scene, that is, compared to the example shown in FIG. 4, Here you can also set up an option for shooting night scene videos separately. When the user wants to record the night scene video through the mobile phone, the option 52 of "night scene recording" can be triggered. In response to the user's trigger operation on the "night scene recording" option 52, the mobile phone displays an operation interface (called the fourth interface) in the night scene mode. Exemplarily, the fourth interface is shown in (c) in FIG. 5 .

In one embodiment, a shooting option 53 is provided on the fourth interface, and the user can trigger the shooting option 53 . In response to the user's trigger operation on the shooting option 53, the mobile phone records a video through a camera (such as a main camera). For example, the video recording interface is shown in (d) in FIG. 5 . In addition, a first switching option 54 may also be provided in the fourth interface, and the first switching option 54 is used to switch between the front camera and the rear camera. Unlike the embodiment shown in Figure 4, there is no need to provide the second switching option in the fourth interface, that is, the "night scene recording" option 52 for triggering night scene video recording is provided under the "more" option alone.

Of course, it should be noted that the above is described by taking night scene shooting as an example. In another embodiment, the method provided by the embodiment of the present application can also be applied to a conventional video recording scene. For example, refer to (a) in FIG. 5 As shown in the figure, after the user triggers the electronic device to perform video recording through the "recording" option, the electronic device can still process the collected video data by using the method provided by the embodiment of the present application. In another embodiment, the method can also be applied to a camera preview scene, that is, when the electronic device starts the camera and enters the preview state, it can use the method provided by the embodiment of the present application to process the preview image.

Please refer to FIG. 6 , and then describe in detail the implementation process of the method for generating video frames provided by the embodiment of the present application in combination with the system architecture shown in FIG. 3 . As an example but not a limitation, the method is applied to an electronic device, and the electronic device is implemented through the interaction between various nodes shown in FIG. 3. The method may include the following implementation steps:

601: The image sensor node acquires first RAW data.

In an implementation, the electronic device launches a camera application. Exemplarily, as shown in (a) of FIG. 4 , an application icon of a camera application is provided on a display interface of the electronic device. When detecting a user's trigger operation on the application icon, the electronic device starts the camera application in response to the user's trigger operation.

In an example, the camera application determines that a night scene video shooting instruction is received when a trigger operation of video shooting in the super night video mode is detected. The camera application sends a night scene video shooting request to the video module. Exemplarily, please refer to the interaction flow in the embodiment of FIG. Generate a night scene video shooting request, and send the night scene video shooting request to the video module.

After the video module receives the night scene video shooting request, it establishes a framework for processing the night scene video, and the specific implementation can be referred to above. Afterwards, the image sensor node collects and captures the light source signal through the image sensor in the camera, and converts the captured light source signal into a data signal to obtain the first RAW data. Exemplarily, the first RAW data is 4K60 interlaced high dynamic range (staggered high dynamic range, SHDR) data, where 4K60 means that the resolution is 4K and the frame rate is 60 frames per second.

In one embodiment, the first RAW data includes long exposure data and short exposure data, the long exposure data refers to the data collected by the image sensor through the long exposure method, and the short exposure data refers to the data collected by the image sensor through the short exposure method . That is, two exposures are performed within one exposure time to obtain the first RAW data. Exemplarily, taking the first RAW data as 4K60 SHDR data as an example, the camera exposes twice within each 33ms, thus obtaining 60 frames of video data.

It is worth mentioning that the combination of long and short exposures can effectively improve the dynamic range of the video frame. Short exposures are used to prevent overexposure in highlighted areas, and long exposures are used to brighten dark areas to prevent underexposure.

602: The image sensor node sends the first RAW data to the external ISP node.

Exemplarily, the image sensor node sends the 4K60 SHDR data to the external ISP node for processing such as fusion and enhancement through the external ISP node.

As an example of the present application, the first RAW data first arrives at the routing submodule in the external ISP node.

603: The routing submodule performs copying and routing processing on the first RAW data.

When an electronic device shoots a video in a night scene, in order to obtain a clear video frame, on the one hand, the first RAW data collected can be enhanced, and on the other hand, the exposure data can be calculated according to the first RAW data collected by the camera. , to obtain the first exposure data, and then dynamically adjust the exposure parameters of the camera according to the first exposure data.

Therefore, as an example of the present application, the routing sub-module in the external ISP node performs copying and routing processing on the first RAW data. In implementation, the routing sub-module copies the first RAW data. For the convenience of distinction, the copied RAW data is called the second RAW data. In this way, two copies of RAW data (including the first RAW data and the second RAW data) can be obtained. RAW data). Afterwards, the two obtained RAW data are routed. Specifically, the routing submodule transmits one part of the RAW data (for example, the first RAW data) to the first preprocessing submodule for processing, and the other part of the RAW data ( For example, the second RAW data) is used for subsequent built-in ISP nodes to count the first exposure data.

604: The first preprocessing submodule performs preprocessing on the first RAW data.

Due to the influence of factors such as unsatisfactory physical components of the camera, there may be certain defects in the first RAW data, such as the influence of dark current, the brightness attenuation around the image, and the existence of dead pixels. Before performing fusion and noise reduction processing on the RAW data, the first preprocessing submodule first preprocesses the first RAW data, so as to correct the first RAW data.

As an example and not limitation, preprocessing includes but is not limited to lens shading correction (LSC) processing, black level compensation (black level compensation, BLC) processing, bad pixel correction (bad pixel correction, BPC processing), color interpolation At least one of the processing.

605: The first preprocessing submodule sends the preprocessed first RAW data to the enhancement submodule.

For example, the first preprocessing submodule sends the preprocessed 4K60 SHDR data to the enhancement submodule.

606: The enhancement submodule performs fusion and noise reduction processing on the preprocessed first RAW data.

As an example of the present application, the specific implementation of performing fusion processing on the preprocessed first RAW data may include: inputting the preprocessed first RAW data into the first target model for processing, and outputting the fused data. The first target model can determine fusion data based on arbitrary long exposure data and short exposure data.

Exemplarily, if the preprocessed first RAW data is 4K60 SHDR data, the 4K60 SHDR data is input into the first target model, and the data obtained after fusion processing is 4K30 data. That is to say, when the first target model performs fusion processing, it fuses the long-exposure data and short-exposure data obtained through two consecutive exposures in the same time period, so the 60-frame data before fusion becomes 30 frames.

Wherein, the first target model may be a pre-trained fusion network model. For example, the first target model may be obtained after training the first network model based on the exposure sample data. In one example, the first network model may include, but is not limited to, HDRnet.

Afterwards, noise reduction processing is performed on the fused video data. As an example of the present application, the implementation of denoising the fused video data may include: inputting the fused video data into the second target model for processing, and outputting the denoised video data. The second target model can perform noise reduction processing on arbitrary video data.

Wherein, the second target model may be a pre-trained denoising network model. For example, the second target model may be obtained after training the second network model based on the RAW sample data. In one example, the second network model may include, but is not limited to, Unet.

607: The enhancement submodule outputs the video data after noise reduction processing, and the routing submodule outputs the second RAW data.

Specifically, the enhancement submodule sends the noise-reduced video data to the second preprocessing submodule, and the routing submodule also sends the second RAW data to the second preprocessing submodule. It is not difficult to understand that the video data output by the enhancement sub-module is 4K30 data, which is used for browsing and recording; the second RAW data output by the routing sub-module is 4K60 data, which is used for calculating 3A and possible camera needs.

It should be noted that since the external ISP node performs fusion and noise reduction processing on the first RAW data collected by the image sensor, the difference between the video data output by the external ISP node and the first RAW data output by the image sensor Generally, there will be a certain delay. For example, there is a delay of one frame. For example, if the image sensor outputs the first RAW data at time t, the external ISP node outputs video data at time t-1 at the same time.

In addition, the external ISP node controls the synchronous output of the enhancement sub-module and the routing sub-module, that is, the noise-reduced video data and the second RAW data are synchronously transmitted to the second pre-processing sub-module.

608: The second preprocessing submodule processes the video data output by the enhancement submodule, and calculates the first exposure data based on the second RAW data, and adjusts an exposure parameter.

As an example of the present application, the processing of the video data output by the enhancement submodule by the second preprocessing submodule includes: performing preprocessing again on the video data output by the enhancement submodule, such as but not limited to LSC processing, BLC processing, At least one of BPC processing and color interpolation processing to further reduce the noise of the video data. Afterwards, RGB conversion is performed on the preprocessed video data again, and the video image obtained after the RGB conversion is compressed to obtain a YUV image.

It should be noted that it is optional for the second preprocessing submodule described in the embodiment of the present application to preprocess the video data output by the enhancement submodule again. In another embodiment, the second preprocessing submodule also The RGB conversion may be performed directly based on the video data output by the enhancement sub-module, which is not limited in this embodiment of the present application.

In addition, the second preprocessing sub-module determines the first exposure data based on the received second RAW data, and determines whether the current exposure level is reasonable according to the first exposure data, and then adjusts the exposure parameters of the camera if it is not reasonable. The value range of the first exposure data is (0, 255). In one example, the second preprocessing submodule compares the first exposure data with the exposure threshold, and if the difference between the first exposure data and the exposure threshold is greater than the threshold range, gradually adjusts the first Exposure data to obtain target exposure data. The second preprocessing sub-module sends the target exposure data to the camera, so that the camera adjusts the exposure parameters of the image sensor, and the ultimate goal is to make the exposure data calculated according to the second RAW data close to or equal to the exposure threshold.

Wherein, the adjustment step size may be set according to actual requirements. The exposure threshold can be set according to actual needs. The threshold range can also be set according to actual needs.

For example, the exposure threshold is 128, the threshold range is [0,5], and the adjustment step is 4. If the first exposure data is 86, it means that the exposure parameter needs to be increased. At this time, the first exposure data can be adjusted according to the adjustment step to obtain a target exposure data of 90. The second preprocessing sub-module sends the target exposure data 90 to the camera, so that the camera adjusts the exposure parameter of the image sensor to 90. The exposure data is again counted according to the second RAW data received next time, and the exposure parameters of the image sensor are adjusted according to the above method until the counted exposure data is close to or equal to 128.

It is worth mentioning that by adjusting the exposure data step by step so that the exposure parameters of the camera are close to or the same as the exposure threshold, the exposure changes of the video frames can be smoothly transitioned.

609: The second preprocessing submodule sends the YUV image and target exposure data to the optimization processing submodule.

According to the foregoing, it can be seen that the target exposure data is the adjusted exposure data determined according to the first exposure data. For example, if the first exposure data is 100, and the second preprocessing sub-module determines that the exposure parameter of the image sensor needs to be adjusted to 200, then the target exposure data is 200.

As an example of the present application, since the second preprocessing sub-module adjusts the exposure parameters of the image sensor, the gain of the video data obtained through the image sensor subsequently changes. In order to facilitate the optimization processing sub-module The YUV image received once is subjected to reasonable noise reduction processing. While adjusting the exposure parameters of the image sensor, the second preprocessing sub-module sends the target exposure data to the optimization processing sub-module, which is convenient for the optimization processing sub-module to determine the noise reduction parameters. , so as to perform reasonable noise reduction processing on the next received YUV image according to the noise reduction parameters.

As an example of this application, the external ISP node includes multiple second target models, each of the multiple second target models corresponds to an exposure value range, and the exposure value corresponding to each second target model The number of ranges can be one or more. According to the foregoing, it can be seen that the second target model can be used for noise reduction processing. Similarly, in order to perform reasonable noise reduction processing on the next video data, the second preprocessing sub-module can also send the target exposure data to external Set the ISP node so that the external ISP node can determine the exposure value range to which the target exposure data fed back by the second preprocessing sub-module belongs, so that according to the determined exposure value range, the corresponding second target model can be selected from multiple second target models. The target model, the selected second target model is used for the next noise reduction process.

610: The optimization processing submodule performs image optimization processing based on the received data.

The optimization processing sub-module optimizes the YUV image according to the target exposure data, such as performing noise reduction processing on the YUV image, so as to obtain a clear and bright video frame.

611: The optimization processing submodule sends the obtained video frames to display.

That is, the optimization processing sub-module sends the video frames obtained after the image optimization processing to the display screen for display.

It should be noted that, the above is described by taking the second preprocessing sub-module to count the first exposure data after receiving the second RAW data, and adjust the exposure parameters of the image sensor as an example. In implementation, the second preprocessing submodule also counts the AWB and AF based on the second RAW data, and sends the AWB to the optimization processing submodule, so that the optimization processing submodule can perform white balance adjustment during image optimization processing. In addition, the second preprocessing submodule sends the AF to the camera, so that the camera can perform adjustment processing according to the AF.

In the embodiment of this application, in the super night video mode, the video data is fused and denoised through the external ISP, the processed video data is sent to the built-in ISP, and the original video data is provided for the built-in ISP, so that Because the built-in ISP can determine the target exposure data based on the original video data. In this way, the built-in ISP can generate clear video frames based on the processed video data provided by the external ISP and the determined target exposure data.

Please refer to FIG. 7 . FIG. 7 is a schematic structural diagram of hardware according to an exemplary embodiment. The hardware involved in this embodiment of the present application mainly includes a camera, an external ISP, and a built-in ISP. The external ISP includes multiple interfaces, a routing unit, a front-end processing unit, and a back-end processing unit. The routing unit is connected to the front-end processing unit, and the front-end processing unit is connected to the back-end processing unit. The routing unit is used to execute the implementation shown in Figure 6. In the function of the routing submodule in the example, the front-end processing unit is used to perform the function of the first preprocessing submodule in the embodiment of FIG. 6 , and the back-end processing unit is used to perform the function of the enhanced submodule in FIG. 6 . The processing unit includes an IFE, and the back-end processing unit includes an NPU; the built-in ISP includes a first ISP front-end unit, a second ISP front-end unit, and an ISP back-end unit, the first ISP front-end unit is connected to the ISP back-end unit, and the second ISP front-end unit The unit is connected with the ISP back-end unit, and the first ISP front-end unit and the second ISP front-end unit are used to perform the function of the second preprocessing submodule in Figure 6, and the ISP back-end unit is used to perform the optimization processing submodule in Figure 6 Function. In one example, the first ISP front-end unit includes IFE0, the second ISP front-end unit includes IFE1, and the ISP back-end unit includes IPE.

It should be noted that, the foregoing is only an exemplary illustration of the multiple units included in the external ISP and the internal ISP, but does not constitute a limitation to the structural components thereof. In some embodiments, the external ISP or the internal ISP may further include other units, which is not limited in this embodiment of the present application.

Next, in conjunction with the hardware framework diagram shown in accompanying drawing 7, the flow of the method for generating video frames provided by the embodiment of the present application is introduced, specifically:

701: The external ISP receives the first RAW data from the camera.

In one embodiment, the first RAW data may be from the main camera of the electronic device. In another embodiment, the first RAW data may also be from an auxiliary camera of the electronic device. For example, in the case of 1x focus, the first RAW data comes from the main camera of the electronic device, and in the case of 3x focus, the first RAW data comes from the auxiliary camera of the electronic device, which is not limited in this embodiment of the present application .

As an example but not a limitation, as shown in FIG. 7, the external ISP receives the first RAW data from the camera through a mobile industry processor interface (mobile industry processor interface, Mipi)0.

702: The external ISP copies and routes the first RAW data through the routing unit.

As an example of the present application, the external ISP first copies the first RAW data through the routing unit, so as to obtain another copy of the RAW data, which is referred to as the second RAW data here. Afterwards, the routing unit performs routing processing on the two sets of RAW data. For example, one of the RAW data (such as the first RAW data) is transmitted to the front-end processing unit in the external ISP, and the front-end processing unit is based on this one. The first RAW data is preprocessed, and the first RAW data obtained after preprocessing is sent to the back-end processing unit in the external ISP, and the back-end processing unit performs fusion and noise reduction processing; another RAW data (such as 2nd RAW data) for direct output to the built-in ISP.

The preprocessing of the RAW data by the front-end processing unit may refer to the embodiment shown in FIG. 6 above, and the fusion and noise reduction processing of the preprocessed RAW data by the back-end processing unit may also refer to the embodiment shown in FIG. 6 above.

It should be noted that, the above is only described by taking the example of transmitting the first RAW data to the front-end processing unit in the external ISP and directly outputting the second RAW data to the built-in ISP. In another embodiment, the second RAW data may also be transmitted to the front-end processing unit in the external ISP, and the first RAW data may be directly output to the built-in ISP, which is not limited in this embodiment of the present application.

703: The back-end processing unit outputs the video data after the noise reduction processing, and the routing unit outputs the second RAW data.

Exemplarily, the back-end processing unit sends the noise-reduced video data to the built-in ISP through the Mipi0 interface of the external ISP, and the routing unit sends another 60-frame second RAW data through the Mipi1 interface of the external ISP to the built-in ISP.

704: The built-in ISP receives the video data output by the back-end processing unit and the second RAW data output by the routing unit.

In one example, the built-in ISP receives the video data output by the back-end processing unit through the first ISP front-end unit, and then the first ISP front-end unit processes the video data output by the back-end processing unit, such as RGB conversion, and converts The final RGB image is compressed to obtain a YUV image. In some embodiments, the first ISP front-end unit may also perform preprocessing on the received video data before performing data format conversion. For example, preprocessing includes but is not limited to color correction, downsampling, demosaicing, etc.

Afterwards, the first ISP front-end unit transmits the YUV image to the ISP back-end unit for processing. In one example, the ISP back-end unit is mainly used for image processing tasks such as hardware noise reduction, image cropping, noise reduction, color processing, detail enhancement, etc. Exemplarily, hardware noise reduction includes multi-frame noise reduction (multi-frame noise reduction, MFNR), multi-frame super resolution (multi-frame super resolution, MFSR).

In an example, the built-in ISP receives the second RAW data output by the routing unit through the second ISP front-end unit. After receiving the second RAW data, the second ISP front-end unit determines the first exposure data based on the second RAW data, and determines whether the current exposure degree is reasonable according to the first exposure data, and if not, determines the target exposure data, and according to The target exposure data is used to adjust the exposure parameters of the camera. Exemplarily, the second ISP front-end unit sends the target exposure data to the camera through the I2C interface, so as to control the camera. For its specific implementation, reference may be made to the embodiment shown in FIG. 6 .

Further, the second ISP front-end unit also counts the AF based on the second RAW data, and sends the AF to the camera through the I2C interface, so that the camera can perform adjustment processing according to the AF.

In addition, the second ISP front-end unit also counts AWB, color and other data based on the second RAW data. The second ISP front-end unit transmits data such as 3A and color to the ISP back-end unit, so that the ISP back-end unit can optimize the YUV image according to the data transmitted by the second ISP front-end unit, such as performing noise reduction processing on the YUV image, thereby obtaining Clear, crisp video frames.

In addition, the second ISP front-end unit can also send the target exposure data to the external ISP through the peripheral interface, so that the external ISP can select one of multiple second target models for noise reduction according to the target exposure data. The second target model, so that the next video data is subjected to noise reduction processing according to the selected second target model. Exemplarily, the peripheral interface may be a secure digital input and output (secure digital input and output, SDIO) interface.

705: Built-in ISP output video frame.

Specifically, the built-in ISP outputs the video frame through the ISP back-end unit, so as to display the video frame on the display screen.

Please refer to FIG. 8 . FIG. 8 is a schematic diagram showing a flow of a method for generating video frames according to an exemplary embodiment. As an example but not a limitation, the method may be applied to the above-mentioned electronic device, and the electronic device includes an image sensor, a first image signal processing module, and a second image signal processing module. The second image signal processing module is a module with a video data processing function, and the first image signal processing module is a module capable of acquiring video data and having a video data processing function. As an example of the present application, the second image signal processing module includes an ISP integrated in the SOC, that is, includes the above-mentioned built-in ISP, and the first image signal processing module includes an ISP outside the SOC, that is, includes the above-mentioned external ISP. The electronic device implements a method for generating a video frame through an image sensor, a first image signal processing module, and a second image signal processing module, and the method may include the following:

Step 801: The image sensor outputs first RAW data.

For example, please refer to step 601 in the embodiment shown in FIG. 6 , the first RAW data is the original RAW data acquired by the image sensor, that is, the first RAW data is the original video data.

In an example, the specific implementation of step 801 may include: detecting a night scene video shooting instruction through a camera application in the electronic device, where the night scene video shooting instruction is used to instruct video recording in a night scene mode. In response to the night scene video shooting instruction, the image sensor outputs first RAW data.

Step 802: the first image signal processing module acquires first RAW data.

Step 803: the first image signal processing module copies the first RAW data to obtain the second RAW data.

For example, please refer to step 603 in the embodiment shown in FIG. 6 , the second RAW data may be another copy of RAW data obtained by copying the RAW data by the routing submodule.

Step 804: The first image signal processing module performs image enhancement processing on the first RAW data to obtain video enhancement data.

As an example of the present application, the first RAW data includes long exposure data and short exposure data collected in the same time period. In this case, the specific implementation of step 804 may include: the first image signal processing module fuses the long-exposure data and the short-exposure data to obtain fused RAW data. The first image signal processing module performs noise reduction processing on the fused RAW data to obtain video enhancement data.

As an example of the present application, the specific implementation of the fusion processing of the long exposure data and the short exposure data by the first image signal processing module may include: the first image signal processing module inputs the long exposure data and the short exposure data into the first object model In the method, fusion processing is performed by the first object model, and the first object model can perform fusion processing on arbitrary long exposure and short exposure data.

As an example of the present application, the first image signal processing module fuses the long exposure data and the short exposure data, and before obtaining the fused RAW data, it also includes: the first image signal processing module performs preprocessing Processing, the preprocessing includes at least one of LSC processing, BLC processing, BPC processing, and color interpolation processing. In this case, the specific implementation of the fusion processing of the long exposure data and the short exposure data by the first image signal processing module includes: the first image signal processing module performs fusion processing on the preprocessed long exposure data and short exposure data to obtain Fusion of RAW data.

As an example of the present application, the specific implementation of the first image signal processing module to perform noise reduction processing on the fused RAW data may include: the first image signal processing module inputs the fused RAW data into the second target model, and the second target model Perform noise reduction processing, and the second target model can perform noise reduction processing on any RAW data.

As an example of the present application, the first image signal processing module includes a plurality of second target models, and each second target model in the plurality of second target models corresponds to an exposure value range. In this case, before the first image signal processing module inputs the fused RAW data into the second target model, it further includes: the first image signal processing module receives target exposure data, and the target exposure data is generated by the second image signal processing module Determined based on the first exposure data, the first exposure data is obtained by the second image signal processing module based on the second RAW data to perform statistics on the exposure data, and the target exposure data is used to adjust the exposure parameters of the image sensor of the electronic device. The first image signal processing module selects a second target model from multiple second target models according to the target exposure data and the exposure value range corresponding to each second target model, and the selected second target model is used for noise reduction processing .

For example, please refer to the embodiment shown in FIG. 6 , the above preprocessing process is performed by the first preprocessing submodule, and the fusion and noise reduction processing processes are performed by the enhancement submodule. In this case, the video enhancement data refers to the video data output by the enhancement sub-module, and the second RAW data refers to the video data output by the routing sub-module.

Step 805: The first image signal processing module sends the video enhancement data and the second RAW data to the second image signal processing module.

For example, referring to the embodiment shown in FIG. 6 , the first image signal processing module sends the video enhancement data and the second RAW data to the second preprocessing submodule respectively.

Step 806: The second image signal processing module generates a video frame based on the video enhancement data and the second RAW data.

In an example, the specific implementation of step 806 includes: the second image signal processing module performs format conversion processing on the video enhancement data to obtain a YUV image. The second image signal processing module determines target data based on the second RAW data, and the target data is used to adjust the image quality of the YUV image. The second image signal processing module adjusts the YUV image based on the target data, and uses the adjusted YUV image as a video frame.

In one example, target data includes but not limited to 3A data, color.

In the embodiment of the present application, the first RAW data is acquired by the first image signal processing module, and the first RAW data is copied to obtain the second RAW data. Image enhancement processing is performed based on the first RAW data to obtain video enhancement data. Send the enhanced video data and the second RAW data to the second image signal processing module. The second image signal processing module generates video frames based on the video enhancement data and the second RAW data. That is, image enhancement processing is performed by the first image signal processing module, and the first image signal processing module also provides the second image signal processing module with second RAW data that can be used to adjust the exposure parameters, so as to ensure that clear video frame.

It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used for illustration. In practical applications, the above-mentioned functions can be assigned to different functional units, Completion of modules means that the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated into one processing unit, or each unit may exist separately physically, or two or more units may be integrated into one unit, and the above-mentioned integrated units may adopt hardware It can also be implemented in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working processes of the units and modules in the above system, reference may be made to the corresponding processes in the aforementioned method embodiments, and details will not be repeated here.

In the above-mentioned embodiments, the descriptions of each embodiment have their own emphases, and for parts that are not detailed or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In the embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the procedures in the methods of the above embodiments in the present application can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a computer-readable storage medium. The computer program When executed by a processor, the steps in the above-mentioned various method embodiments can be realized. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program codes to electronic equipment, recording media, computer memory, read-only memory (Read-Only Memory, ROM), RAM, electrical carrier signal, telecommunications signals and software distribution media. Such as U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunication signals under legislation and patent practice.

Finally, it should be noted that: the above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application shall be covered by this application. within the scope of the application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims

A method for generating a video frame, characterized in that it is applied to an electronic device, the electronic device includes an image sensor, a first image signal processing module, and a second image signal processing module, and the method includes:

The image sensor outputs first raw data;

The first image signal processing module acquires the first raw data;

The first image signal processing module copies the first original data to obtain second original data;

The first image signal processing module performs image enhancement processing on the first raw data to obtain video enhancement data;

The first image signal processing module sends the video enhancement data and the second original data to the second image signal processing module;

The second image signal processing module generates video frames based on the video enhancement data and the second original data.
The method according to claim 1, wherein the first raw data includes long exposure data and short exposure data collected in the same time period, and the first image signal processing module performs processing on the first raw data Image enhancement processing, including:

The first image signal processing module fuses the long exposure data and the short exposure data to obtain fused original data;

The first image signal processing module performs noise reduction processing on the fusion original data.
The method according to claim 2, wherein the first image signal processing module fuses the long exposure data and the short exposure data, including:

The first image signal processing module inputs the long-exposure data and the short-exposure data into a first object model, and performs fusion processing by the first object model, and the first object model can Exposure data and short exposure data are fused.
The method according to claim 2, wherein the first image signal processing module performs noise reduction processing on the fused raw data, including:

The first image signal processing module inputs the fused raw data into a second target model, and the second target model performs noise reduction processing, and the second target model can perform noise reduction processing on any raw data .
The method according to claim 4, wherein the first image signal processing module includes a plurality of second object models, and each second object model in the plurality of second object models corresponds to an exposure value scope;

The method also includes:

The first image signal processing module receives target exposure data, the target exposure data is determined by the second image signal processing module based on the first exposure data, and the first exposure data is determined by the second image signal processing module The module obtains exposure data statistics based on the second raw data, and the target exposure data is used to adjust the exposure parameters of the image sensor;

The first image signal processing module selects a second target model from the plurality of second target models according to the target exposure data and the exposure value range corresponding to each second target model, and the selected second target model A two-objective model is used for denoising.
The method according to claim 2, wherein, before the first image signal processing module fuses the long exposure data and the short exposure data, it further includes:

The first image signal processing module preprocesses the long exposure data and the short exposure data, and the preprocessing includes lens correction LSC processing, black level compensation BLC processing, bad pixel correction BPC processing, and color interpolation processing at least one of the

The first image signal processing module fuses the long exposure data and the short exposure data, including:

The first image signal processing module fuses the preprocessed long exposure data and the short exposure data.
The method according to any one of claims 1-6, wherein the second image signal processing module generates a video frame based on the video enhancement data and the second original data, comprising:

The second image signal processing module performs format conversion processing on the video enhancement data to obtain a YUV image;

The second image signal processing module determines target data based on the second raw data, and the target data is used to adjust the image quality of the YUV image;

The second image signal processing module adjusts the YUV image based on the target data, and uses the adjusted YUV image as the video frame.
The method according to any one of claims 1-7, wherein the image sensor outputs the first raw data, comprising:

A night scene video shooting instruction is detected through the camera application in the electronic device, and the night scene video shooting instruction is used to indicate video recording in night scene mode;

In response to the night scene video shooting instruction, the image sensor outputs the first raw data.
The method according to any one of claims 1-8, wherein the second image signal processing module includes an image signal processor ISP integrated in a chip-level system SOC, and the first image signal processing module includes the ISP outside the SOC.
A device for generating video frames, characterized in that the device includes: an image sensor node, a first image signal processing module, and a second image signal processing module;

The image sensor node is configured to output first raw data;

The first image signal processing module is configured to acquire the first original data, copy the first original data to obtain second original data, and perform image enhancement processing on the first original data to obtain video enhanced data, sending the enhanced video data and the second raw data to the second image signal processing module;

The second image signal processing module is configured to generate a video frame based on the video enhancement data and the second original data.
An electronic device, characterized in that the electronic device includes a memory and a processor;

The memory is used to store a program that supports the electronic device to execute the method described in any one of claims 1-9, and to store data involved in implementing the method described in any one of claims 1-9; The processor is configured to execute programs stored in the memory.
A computer-readable storage medium, with instructions stored in the computer-readable storage medium, is characterized in that, when the computer-readable storage medium is run on a computer, it causes the computer to execute the method described in any one of claims 1-9.