WO2023077938A1 - Procédé et appareil de génération de trame vidéo, dispositif électronique et support de stockage - Google Patents

Procédé et appareil de génération de trame vidéo, dispositif électronique et support de stockage Download PDF

Info

Publication number
WO2023077938A1
WO2023077938A1 PCT/CN2022/116757 CN2022116757W WO2023077938A1 WO 2023077938 A1 WO2023077938 A1 WO 2023077938A1 CN 2022116757 W CN2022116757 W CN 2022116757W WO 2023077938 A1 WO2023077938 A1 WO 2023077938A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
image signal
signal processing
processing module
video
Prior art date
Application number
PCT/CN2022/116757
Other languages
English (en)
Chinese (zh)
Inventor
侯伟龙
金杰
李子荣
Original Assignee
荣耀终端有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 荣耀终端有限公司 filed Critical 荣耀终端有限公司
Publication of WO2023077938A1 publication Critical patent/WO2023077938A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems

Definitions

  • the present application relates to the technical field of terminals, and in particular to a method, device, electronic equipment and storage medium for generating video frames.
  • the electronic device takes images through the camera and processes them through the image signal processor (Image Signal Processing, ISP) (hereinafter referred to as the built-in ISP) integrated in the system on chip (System on Chip, SOC).
  • ISP Image Signal Processing
  • SOC System on Chip
  • the present application provides a method, device, electronic equipment and storage medium for generating video frames, which solves the problem that complex multi-frame enhancement processing algorithms similar to photographing cannot be used in the prior art, and the display effect of video images is often significantly worse than that of captured images. question.
  • a method for generating a video frame which is applied to an electronic device, and the electronic device includes an image sensor, a first image signal processing module, and a second image signal processing module, and the method includes:
  • the image sensor outputs first raw data
  • the first image signal processing module acquires the first raw data
  • the first image signal processing module copies the first original data to obtain second original data
  • the first image signal processing module performs image enhancement processing on the first raw data to obtain video enhancement data
  • the first image signal processing module sends the video enhancement data and the second original data to the second image signal processing module
  • the second image signal processing module generates a video frame based on the video enhancement data and the second original data.
  • image enhancement processing is performed by the first image signal processing module, and the first image signal processing module also provides the second image signal processing module with second raw data that can be used to adjust exposure parameters, thus ensuring that clear video frame.
  • the first raw data includes long exposure data and short exposure data collected in the same time period
  • the first image signal processing module performs image enhancement processing on the first raw data, including:
  • the first image signal processing module fuses the long exposure data and the short exposure data to obtain fused original data
  • the first image signal processing module performs noise reduction processing on the fusion original data.
  • the long-exposure data and the short-exposure data within the same time period can be fused subsequently to output high dynamic video frames.
  • the first image signal processing module fuses the long exposure data and the short exposure data, including:
  • the first image signal processing module inputs the long-exposure data and the short-exposure data into a first object model, and performs fusion processing by the first object model, and the first object model can Exposure data and short exposure data are fused.
  • the first image signal processing module performs noise reduction processing on the fused raw data, including:
  • the first image signal processing module inputs the fused raw data into a second target model, and the second target model performs noise reduction processing, and the second target model can perform noise reduction processing on any raw data .
  • noise reduction efficiency can be improved by using the second target model to perform noise reduction processing.
  • the first image signal processing module includes a plurality of second target models, and each second target model in the plurality of second target models corresponds to an exposure value range;
  • the method also includes:
  • the first image signal processing module receives target exposure data, the target exposure data is determined by the second image signal processing module based on the first exposure data, and the first exposure data is determined by the second image signal processing module
  • the module obtains exposure data statistics based on the second raw data, and the target exposure data is used to adjust the exposure parameters of the image sensor;
  • the first image signal processing module selects a second target model from the plurality of second target models according to the target exposure data and the exposure value range corresponding to each second target model, and the selected second target model A two-objective model is used for denoising.
  • a second target model for the next noise reduction process is selected from multiple second target models, so that a reasonable noise reduction process can be performed on the next video data , thereby improving the noise reduction effect.
  • the first image signal processing module before the first image signal processing module fuses the long exposure data and the short exposure data, it further includes:
  • the first image signal processing module preprocesses the long exposure data and the short exposure data, and the preprocessing includes lens correction LSC processing, black level compensation BLC processing, bad pixel correction BPC processing, and color interpolation processing at least one of the
  • the first image signal processing module fuses the long exposure data and the short exposure data, including:
  • the first image signal processing module fuses the preprocessed long exposure data and the short exposure data.
  • the second image signal processing module generates a video frame based on the video enhancement data and the second original data, including:
  • the second image signal processing module performs format conversion processing on the video enhancement data to obtain a YUV image
  • the second image signal processing module determines target data based on the second raw data, and the target data is used to adjust the image quality of the YUV image;
  • the second image signal processing module adjusts the YUV image based on the target data, and uses the adjusted YUV image as the video frame.
  • the format conversion process is performed on the video enhancement data through the second image signal processing module, and the target data is determined based on the second raw data, and the YUV image obtained after the format conversion process is optimized according to the target data, so as to obtain a clear picture video frame.
  • the image sensor outputs first raw data, including:
  • a night scene video shooting instruction is detected through the camera application in the electronic device, and the night scene video shooting instruction is used to indicate video recording in night scene mode;
  • the image sensor In response to the night scene video shooting instruction, the image sensor outputs the first raw data.
  • the electronic device acquires the first raw data, and processes the first raw data collected by the camera through the method provided in this application, so that the highlighted area of the obtained video frame will not be overexposed and the darkened area will not be overexposed. Not too dark, resulting in a clear video frame.
  • the second image signal processing module includes an ISP integrated in a system-on-a-chip (SOC), and the first image signal processing module includes an ISP outside the SOC.
  • SOC system-on-a-chip
  • a device for generating a video frame comprising: an image sensor node, a first image signal processing module, and a second image signal processing module;
  • the image sensor node is configured to output first raw data
  • the first image signal processing module is configured to acquire the first original data, copy the first original data to obtain second original data, and perform image enhancement processing on the first original data to obtain video enhanced data, sending the enhanced video data and the second raw data to the second image signal processing module;
  • the second image signal processing module is configured to generate a video frame based on the video enhancement data and the second original data.
  • the first raw data includes long exposure data and short exposure data collected in the same time period;
  • the first image signal processing module is used for:
  • the first image signal processing module is used for:
  • Inputting the long exposure data and the short exposure data into a first target model, and performing fusion processing by the first target model, and the first target model can fuse any long exposure data and short exposure data deal with.
  • the first image signal processing module is used for:
  • the fused original data is input into a second target model, and the noise reduction process is performed by the second target model, and the second target model can perform noise reduction process on any original data.
  • the first image signal processing module includes a plurality of second target models, and each second target model in the plurality of second target models corresponds to an exposure value range; the first Image signal processors are also used to:
  • the target exposure data is determined by the second image signal processing module based on the first exposure data
  • the first exposure data is determined by the second image signal processing module based on the second raw data Obtained by performing statistics on exposure data
  • the target exposure data is used to adjust the exposure parameters of the image sensor
  • the target exposure data and the exposure value range corresponding to each second target model select a second target model from the plurality of second target models, and the selected second target model is used for noise reduction processing .
  • the first image signal processing module is used for:
  • the preprocessing includes at least one of lens correction LSC processing, black level compensation BLC processing, bad pixel correction BPC processing, and color interpolation processing;
  • the preprocessed long exposure data and the short exposure data are fused.
  • the second image signal processing module is used for:
  • the target data being used to adjust the image quality of the YUV image
  • the image sensor node is used for:
  • a night scene video shooting instruction is detected through the camera application in the electronic device, and the night scene video shooting instruction is used to indicate video recording in night scene mode;
  • the first raw data is output.
  • the second image signal processing module is an ISP integrated in a system-on-a-chip (SOC), and the first image signal processing module is an ISP outside the SOC.
  • SOC system-on-a-chip
  • an electronic device in the third aspect, includes a processor and a memory, and the memory is used to store a program that supports the electronic device to execute the method described in any one of the above-mentioned first aspects, and to store a program for The data involved in implementing the method described in any one of the above first aspects; the processor is configured to execute the program stored in the memory.
  • the electronic device may also include a communication bus for establishing a connection between the processor and the memory.
  • a computer-readable storage medium wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer is made to execute the method described in any one of the above-mentioned first aspects.
  • a computer program product containing instructions, which, when run on a computer, causes the computer to execute the method described in the first aspect above.
  • FIG. 1 is a schematic diagram of spatial position distribution of a camera provided in an embodiment of the present application
  • FIG. 2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
  • FIG. 3 is a schematic diagram of a software framework of an electronic device provided in an embodiment of the present application.
  • FIG. 4 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of another application scenario provided by the embodiment of the present application.
  • FIG. 6 is a schematic flowchart of a method for generating a video frame provided in an embodiment of the present application
  • FIG. 7 is a schematic structural diagram of a hardware provided by an embodiment of the present application.
  • FIG. 8 is a schematic flowchart of another method for generating video frames provided by an embodiment of the present application.
  • references to "one embodiment” or “some embodiments” or the like in the specification of the present application means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application.
  • appearances of the phrases “in one embodiment,” “in some embodiments,” “in other embodiments,” “in other embodiments,” etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean “one or more but not all embodiments” unless specifically stated otherwise.
  • the terms “including”, “comprising”, “having” and variations thereof mean “including but not limited to”, unless specifically stated otherwise.
  • Exposure According to the length of exposure time, it can be divided into long exposure and short exposure. The longer the exposure time, the greater the amount of light entering the aperture. Conversely, the shorter the exposure time, the smaller the amount of light entering the aperture.
  • 3A statistical algorithm including automatic exposure (automatic exposure, AE) algorithm, automatic focus (automatic focus, AF) algorithm, and automatic white balance (automatic white balance, AWB) algorithm.
  • AE refers to the camera automatically determines the exposure according to the lighting conditions.
  • the imaging system generally has an AE function, which is directly related to the brightness and image quality of the image screen, that is, it will determine the brightness and darkness of the image.
  • AF It means that the camera automatically adjusts the focus distance of the camera according to the distance of the subject, that is, the lens in the camera is adjusted to form the focus through distance measurement, so that the image in the camera is clear.
  • AWB Mainly used to solve the problem of image color cast. If there is a color cast in the image, it can be corrected by the AWB algorithm.
  • Field of view (field angle, FOV), refers to the range that the camera can cover. The larger the FOV, the more scenes the camera can accommodate. It is not difficult to understand that if the subject is not within the FOV of the camera, it will not be captured by the camera.
  • Image sensor (Sensor): It is the core component of the camera, and its function is to convert optical signals into electrical signals for subsequent processing and storage. The working principle is that the photosensitive element generates charge under the condition of light, and the charge transfer generates a current, and the current is rectified and amplified, and converted into a digital signal to form a digital signal.
  • Image sensors generally include two types: charge coupled device (CCD) and complementary metal oxide semiconductor (complementary metal oxide semiconductor, CMOS).
  • RAW data also referred to as raw data in this embodiment of the application, refers to the raw data that the CCD or CMOS image sensor in the camera converts the captured light source signal into a data signal. That is to say, it can be understood as unprocessed data, which can be used to describe the intensity of various lights received by the image sensor.
  • the method provided in the embodiment of the present application may be executed by an electronic device having a shooting function.
  • the electronic device is configured with one or more cameras, and different cameras among the multiple cameras have different shooting functions.
  • the electronic device is configured with at least one of a wide-angle camera, a telephoto camera (such as a periscope telephoto camera), a black-and-white camera, and an ultra-wide-angle camera.
  • the one or more cameras may include a front camera and/or a rear camera.
  • the electronic device is equipped with multiple rear cameras, and the multiple rear cameras include a main camera and at least one auxiliary camera.
  • FIG. 1 the spatial distribution of multiple rear cameras can be shown in FIG.
  • the spatial distribution of multiple rear cameras can also be shown in figure (b) in Figure 1, the multiple rear cameras are respectively camera 00, camera 01, camera 02,
  • the camera 03 for example, the camera 00 is the main camera, and the others are auxiliary cameras.
  • the electronic device After starting the camera application, the electronic device usually shoots through the main camera by default. After the camera is switched, the electronic device selects a suitable auxiliary camera from at least one auxiliary camera according to the switching requirement, and shoots through the selected auxiliary camera.
  • an electronic device may be, but not limited to, a mobile phone action camera (GoPro), digital camera, tablet computer, desktop, laptop, handheld computer, notebook computer, vehicle-mounted device, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook, cellular phone, personal digital assistant (personal digital assistant, PDA), augmented reality (augmented reality, AR) ⁇ virtual reality (virtual reality, VR) equipment, mobile phone, etc. Not limited.
  • GoPro mobile phone action camera
  • digital camera tablet computer
  • desktop laptop
  • handheld computer notebook computer
  • vehicle-mounted device ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook, cellular phone, personal digital assistant (personal digital assistant, PDA), augmented reality (augmented reality, AR) ⁇ virtual reality (virtual reality, VR) equipment, mobile phone, etc.
  • PDA personal digital assistant
  • AR augmented reality
  • VR virtual reality
  • FIG. 2 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2.
  • Mobile communication module 150 wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, ISP191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like.
  • the number of ISPs 191 included in the electronic device is multiple, and only one is exemplarily shown in FIG. 2 .
  • the structure shown in this embodiment does not constitute a specific limitation on the electronic device 100 .
  • the electronic device 100 may include more or fewer components than shown, or combine certain components, or separate certain components, or arrange different components.
  • the illustrated components can be realized in hardware, software or a combination of software and hardware.
  • the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an ISP, a controller , memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • application processor application processor, AP
  • modem processor graphics processing unit
  • GPU graphics processing unit
  • ISP input signal processor
  • controller memory
  • video codec digital signal processor
  • DSP digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit
  • the controller may be the nerve center and command center of the electronic device 100 .
  • the controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
  • a memory may also be provided in the processor 110 for storing instructions and data.
  • the memory in processor 110 is a cache memory.
  • the memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.
  • processor 110 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input and output
  • subscriber identity module subscriber identity module
  • SIM subscriber identity module
  • USB universal serial bus
  • the interface connection relationship between the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 .
  • the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.
  • the charging management module 140 is configured to receive a charging input from a charger.
  • the charger may be a wireless charger or a wired charger.
  • the charging management module 140 can receive charging input from the wired charger through the USB interface 130 .
  • the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 is charging the battery 142 , it can also supply power to the electronic device 100 through the power management module 141 .
  • the power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 .
  • the power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 .
  • the power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance).
  • the power management module 141 may also be disposed in the processor 110 .
  • the power management module 141 and the charging management module 140 may also be set in the same device.
  • the wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.
  • the electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor.
  • the GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 194 is used to display images, videos and the like.
  • the display screen 194 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc.
  • the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.
  • the electronic device 100 can realize the shooting function through the ISP 191, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.
  • the ISP 191 is used for processing the data fed back by the camera 193. For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP 191 for processing, and converts it into an image visible to the naked eye.
  • ISP 191 can also optimize the algorithm for image noise, brightness, and skin tone. ISP 191 can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP 191 may include a built-in ISP integrated in the SOC and an external ISP provided outside the SOC.
  • the internal structure of the external ISP is similar or the same as that of the built-in ISP. The difference is that the methods used by the external ISP and the built-in ISP to process video data are different.
  • the external ISP uses artificial intelligence methods (such as through network models, etc.) to process video data, while the built-in ISP uses other algorithms to process video data.
  • the external ISP mainly has two functions: on the one hand, it is used to perform fusion processing and image enhancement processing on the RAW data collected by the camera, so as to provide enhanced video data for the built-in ISP; on the other hand, It is used to route the RAW data collected by the camera to provide a copy of RAW data for the built-in ISP, so that the built-in ISP can accurately determine the current exposure data, and then facilitate the built-in ISP to dynamically adjust the exposure parameters of the camera according to the exposure data.
  • Camera 193 is used to capture still images or video.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP 191 for conversion into a digital image signal.
  • ISP 191 outputs the digital image signal to DSP for processing.
  • DSP converts digital image signals into standard RGB (red green blue red green blue), YUV and other image signals.
  • the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the electronic device 100 may support one or more video codecs.
  • the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.
  • MPEG moving picture experts group
  • the NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • Applications such as intelligent cognition of the electronic device 100 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.
  • the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.
  • the internal memory 121 may be used to store computer-executable program codes including instructions.
  • the processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 .
  • the internal memory 121 may include an area for storing programs and an area for storing data.
  • the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like.
  • the storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like.
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
  • the electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.
  • the audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal.
  • the audio module 170 may also be used to encode and decode audio signals.
  • the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .
  • the earphone interface 170D is used for connecting wired earphones.
  • the earphone interface 170D may be a USB interface 130, or a 3.5mm open mobile terminal platform (open mobile terminal platform, OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA
  • the keys 190 include a power key, a volume key and the like.
  • the key 190 may be a mechanical key. It can also be a touch button.
  • the electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 .
  • the indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.
  • the SIM card interface 195 is used for connecting a SIM card.
  • the SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 .
  • the electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1.
  • SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card etc.
  • the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.
  • the software structure of the electronic device 100 is exemplarily described by taking an Android system with a layered architecture as an example.
  • FIG. 3 is a block diagram of the software structure of the electronic device 100 provided by the embodiment of the present application.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces.
  • the Android system is respectively an application program layer, a hardware abstract layer (Hardware Abstract Layer, HAL), a kernel layer, and a hardware layer from top to bottom.
  • an application framework layer (Application Framework) (not shown in FIG. 3 ) is also included between the application layer and the HAL, which is not highlighted in this embodiment of the present application.
  • the application layer can consist of a series of application packages. As shown in Figure 3, the application package may include application programs such as a camera and a gallery.
  • the camera application supports a super night video mode, and in the super night video mode, the electronic device can shoot clear, bright and dark videos in night scenes.
  • the application layer also provides pre-loaded external ISP services. Since the internal memory of the external ISP is usually random access memory (Random Access Memory, RAM), according to the characteristics of RAM, RAM cannot save data in the case of power failure, so the external ISP is usually stored during operation.
  • the required data such as external ISP SDK and models (for example, including the first target model and the second target model described below) are stored in the system memory.
  • the application layer starts the preloading external ISP service, so that the external ISP driver controls the external ISP to be powered on in advance, and loads the data required for the external ISP during operation from the system memory to the external ISP
  • the internal RAM is convenient for the external ISP to perform corresponding functions (such as data fusion, image enhancement processing, etc.) in the super night video mode.
  • the video recorded by the camera can be provided in the gallery application, so that the user can view the recorded video from the gallery application.
  • the HAL layer mainly includes a video module, which is used to obtain RAW data through the image sensor of the camera, and perform fusion, enhancement, optimization and other processing on the RAW data through the external ISP and the built-in ISP respectively to obtain video with enhanced clarity and noise reduction effect frame.
  • the resulting video frames are then sent to the display for display.
  • the video module also saves the recorded video to the gallery application for easy viewing by users.
  • the video module includes an image sensor node, a built-in ISP node, and an external ISP node.
  • Each node can be understood as an encapsulation of the functions performed by the underlying hardware, which can be perceived and invoked by the upper layer (application layer).
  • the image sensor node encapsulates the function of the image sensor in the bottom camera;
  • the built-in ISP node encapsulates the function of the bottom built-in ISP;
  • the external ISP node encapsulates the function of the bottom external ISP.
  • the video module implements the above functions through the interaction between the image sensor node, the built-in ISP node, and the external ISP node.
  • the external ISP node includes a plurality of submodules, for example, a routing submodule, a first preprocessing submodule, and an enhancement submodule.
  • each sub-module in the multiple sub-modules can be understood as the encapsulation of the functions of different hardware in the bottom external ISP respectively.
  • the routing sub-module is the function of the SIF in the bottom external ISP.
  • the first preprocessing sub-module is the encapsulation of the function of the IFE in the bottom external ISP
  • the enhancement sub-module is the encapsulation of the function of the neural network processing unit (NPU) in the bottom external ISP.
  • the external ISP implements corresponding functions through the interaction of multiple sub-modules.
  • the interior of the built-in ISP node includes multiple submodules, for example, the second preprocessing submodule and the optimization processing submodule.
  • Each of the multiple sub-modules can be understood as encapsulating the functions of different hardware in the underlying built-in ISP.
  • the second preprocessing sub-module is for one or more image front-ends in the built-in ISP Encapsulation of the function of the engine (image front end, IFE);
  • the optimization processing sub-module is the encapsulation of the function of the image processor (image processing engine, IPE) in the built-in ISP in the bottom layer.
  • the built-in ISP realizes the corresponding functions through the interaction of multiple sub-modules.
  • the HAL layer also includes an external ISP software development kit (software development kit, SDK), which is used to establish the interaction between multiple sub-modules inside the external ISP node.
  • SDK software development kit
  • the kernel layer is the layer between hardware and software.
  • the kernel layer includes but not limited to camera driver, built-in ISP driver, and external ISP driver.
  • the hardware layer includes but not limited to camera, built-in ISP, external ISP, display.
  • the camera application detects that video shooting is enabled in the super night video mode, it sends a notification to the video module at the HAL layer.
  • the video module After the video module receives the notification, it establishes a framework for processing the night scene video. Specifically, the video module notifies the camera driver to control the camera to be powered on, and notifies the built-in ISP driver to control the built-in ISP to be powered on.
  • the camera driver drives the camera, and after the camera is loaded, the camera driver is notified, and the camera driver notifies the video module that the camera has been loaded.
  • the built-in ISP driver drives the built-in ISP, and after the built-in ISP is loaded, the built-in ISP driver is notified, and the built-in ISP driver notifies the video module that the built-in ISP has been loaded.
  • the video module determines that the camera, built-in ISP and external ISP (for example, the loading time is after the camera application starts) are all loaded, set up the interaction between the image sensor node, the built-in ISP node and the external ISP node .
  • the video data can be collected and processed by calling the video module, and the specific implementation can be referred to below.
  • the application scenarios involved in the embodiments of the present application will be introduced next by taking the electronic device as a mobile phone including multiple rear cameras as an example.
  • the user wants to take a night scene video through the mobile phone, at this time, the user can click the application icon of the camera application in the mobile phone.
  • the mobile phone starts the main camera in the rear camera, and displays the first interface shown in (b) of FIG. 4 to the user.
  • a "night scene” option 41 is provided in the first interface, the user can trigger the "night scene” option 41, and in response to the user's trigger operation on the "night scene” option 41, the mobile phone displays the operation interface in the night scene mode ( referred to as the second interface), for example, the second interface is shown in (c) in FIG. 4 .
  • the second interface provides a first switching option 42 and a second switching option 43, wherein the first switching option 43 is used to switch between the front camera and the rear camera.
  • the second switch option 43 is used to switch between the camera mode and the video capture mode.
  • the second switch option 43 can be triggered, and in response to the trigger operation of the second switch option 43 by the user, the mobile phone switches from the camera mode to the video capture mode.
  • the mobile phone after entering the night scene mode, that is, after switching from (b) in Figure 4 to (c) in Figure 4, can also be in the video shooting mode by default, in this case , if the user wants to take a night scene image, the second switch option 43 can be triggered, and in response to the trigger operation of the second switch option 43 by the user, the mobile phone switches from the video shooting mode to the camera mode.
  • a shooting option 44 is also provided in the second interface, and the user can trigger the shooting option 44 .
  • the mobile phone records a video through a camera (such as a main camera).
  • a camera such as a main camera
  • the video recording interface is shown in (d) in FIG. 4 .
  • the mobile phone processes the video data collected by the camera through the method provided in the present application, so that a clear video frame can be captured finally.
  • the clarity of the picture mentioned here means that the highlighted areas will not be overexposed, and the dark areas will not be too dark.
  • a pause option 45 is provided in the video recording interface.
  • the pause option 45 can be triggered, and in response to the user's trigger operation on the pause option 45, the mobile phone pauses the video recording.
  • a snapshot option 46 is provided in the video recording interface.
  • the capture option 46 can be triggered.
  • the mobile phone performs a capture operation and stores the captured video frame.
  • the video recording interface also provides a focusing item 47 for focusing.
  • the focus adjustment item 47 can be triggered, such as adjusting from 1x focus to telephoto, such as adjusting to multi-fold focus (such as 2x focus), or from 1x focus to wide-angle Adjustment, such as adjusting to 0.8 times focus.
  • the mobile phone focuses on the main camera, or switches to other auxiliary cameras for video collection.
  • the mobile phone when the user adjusts from 1x focus to nx focus, when n is greater than 1 and less than 2, the mobile phone will focus on the main camera, and when n is greater than or equal to 2, the mobile phone will switch from the main camera to other auxiliary cameras (such as switching to the telephoto camera). For another example, when the user adjusts from 1x focus to wide-angle, the mobile phone switches from the main camera to the wide-angle camera.
  • the video data collected by the camera currently performing video recording can be processed, so that a video frame with a clear picture can be finally obtained.
  • FIG. 5 there is a "more” option 51 in the first interface.
  • the "more” option 51 can be triggered.
  • the mobile phone displays a third interface, for example, the third interface is shown in (b) of FIG. 5 .
  • a "night scene recording” option 52 is provided in the third interface, and the "night scene recording” option 52 is used to trigger the video recording function under the night scene scene, that is, compared to the example shown in FIG. 4, Here you can also set up an option for shooting night scene videos separately.
  • the option 52 of "night scene recording” can be triggered.
  • the mobile phone displays an operation interface (called the fourth interface) in the night scene mode.
  • the fourth interface is shown in (c) in FIG. 5 .
  • a shooting option 53 is provided on the fourth interface, and the user can trigger the shooting option 53 .
  • the mobile phone records a video through a camera (such as a main camera).
  • a camera such as a main camera
  • the video recording interface is shown in (d) in FIG. 5 .
  • a first switching option 54 may also be provided in the fourth interface, and the first switching option 54 is used to switch between the front camera and the rear camera.
  • the second switching option in the fourth interface that is, the "night scene recording" option 52 for triggering night scene video recording is provided under the "more" option alone.
  • the method provided by the embodiment of the present application can also be applied to a conventional video recording scene.
  • a conventional video recording scene For example, refer to (a) in FIG. 5
  • the electronic device can still process the collected video data by using the method provided by the embodiment of the present application.
  • the method can also be applied to a camera preview scene, that is, when the electronic device starts the camera and enters the preview state, it can use the method provided by the embodiment of the present application to process the preview image.
  • the method is applied to an electronic device, and the electronic device is implemented through the interaction between various nodes shown in FIG. 3.
  • the method may include the following implementation steps:
  • the image sensor node acquires first RAW data.
  • the electronic device launches a camera application.
  • a camera application Exemplarily, as shown in (a) of FIG. 4 , an application icon of a camera application is provided on a display interface of the electronic device.
  • the electronic device starts the camera application in response to the user's trigger operation.
  • the camera application determines that a night scene video shooting instruction is received when a trigger operation of video shooting in the super night video mode is detected.
  • the camera application sends a night scene video shooting request to the video module.
  • a night scene video shooting instruction is received when a trigger operation of video shooting in the super night video mode is detected.
  • the camera application sends a night scene video shooting request to the video module.
  • the video module After the night scene video shooting request, it establishes a framework for processing the night scene video, and the specific implementation can be referred to above.
  • the image sensor node collects and captures the light source signal through the image sensor in the camera, and converts the captured light source signal into a data signal to obtain the first RAW data.
  • the first RAW data is 4K60 interlaced high dynamic range (staggered high dynamic range, SHDR) data, where 4K60 means that the resolution is 4K and the frame rate is 60 frames per second.
  • the first RAW data includes long exposure data and short exposure data
  • the long exposure data refers to the data collected by the image sensor through the long exposure method
  • the short exposure data refers to the data collected by the image sensor through the short exposure method . That is, two exposures are performed within one exposure time to obtain the first RAW data.
  • the camera exposes twice within each 33ms, thus obtaining 60 frames of video data.
  • Short exposures are used to prevent overexposure in highlighted areas, and long exposures are used to brighten dark areas to prevent underexposure.
  • the image sensor node sends the first RAW data to the external ISP node.
  • the image sensor node sends the 4K60 SHDR data to the external ISP node for processing such as fusion and enhancement through the external ISP node.
  • the first RAW data first arrives at the routing submodule in the external ISP node.
  • the routing submodule performs copying and routing processing on the first RAW data.
  • the first RAW data collected can be enhanced, and on the other hand, the exposure data can be calculated according to the first RAW data collected by the camera. , to obtain the first exposure data, and then dynamically adjust the exposure parameters of the camera according to the first exposure data.
  • the routing sub-module in the external ISP node performs copying and routing processing on the first RAW data.
  • the routing sub-module copies the first RAW data.
  • the copied RAW data is called the second RAW data.
  • RAW data two copies of RAW data (including the first RAW data and the second RAW data) can be obtained. RAW data). Afterwards, the two obtained RAW data are routed.
  • the routing submodule transmits one part of the RAW data (for example, the first RAW data) to the first preprocessing submodule for processing, and the other part of the RAW data (for example, the second RAW data) is used for subsequent built-in ISP nodes to count the first exposure data.
  • the RAW data for example, the first RAW data
  • the other part of the RAW data for example, the second RAW data
  • the first preprocessing submodule performs preprocessing on the first RAW data.
  • the first preprocessing submodule Before performing fusion and noise reduction processing on the RAW data, the first preprocessing submodule first preprocesses the first RAW data, so as to correct the first RAW data.
  • preprocessing includes but is not limited to lens shading correction (LSC) processing, black level compensation (black level compensation, BLC) processing, bad pixel correction (bad pixel correction, BPC processing), color interpolation At least one of the processing.
  • LSC lens shading correction
  • BLC black level compensation
  • BPC bad pixel correction
  • color interpolation At least one of the processing.
  • the first preprocessing submodule sends the preprocessed first RAW data to the enhancement submodule.
  • the first preprocessing submodule sends the preprocessed 4K60 SHDR data to the enhancement submodule.
  • the enhancement submodule performs fusion and noise reduction processing on the preprocessed first RAW data.
  • the specific implementation of performing fusion processing on the preprocessed first RAW data may include: inputting the preprocessed first RAW data into the first target model for processing, and outputting the fused data.
  • the first target model can determine fusion data based on arbitrary long exposure data and short exposure data.
  • the preprocessed first RAW data is 4K60 SHDR data
  • the 4K60 SHDR data is input into the first target model, and the data obtained after fusion processing is 4K30 data. That is to say, when the first target model performs fusion processing, it fuses the long-exposure data and short-exposure data obtained through two consecutive exposures in the same time period, so the 60-frame data before fusion becomes 30 frames.
  • the first target model may be a pre-trained fusion network model.
  • the first target model may be obtained after training the first network model based on the exposure sample data.
  • the first network model may include, but is not limited to, HDRnet.
  • the implementation of denoising the fused video data may include: inputting the fused video data into the second target model for processing, and outputting the denoised video data.
  • the second target model can perform noise reduction processing on arbitrary video data.
  • the second target model may be a pre-trained denoising network model.
  • the second target model may be obtained after training the second network model based on the RAW sample data.
  • the second network model may include, but is not limited to, Unet.
  • the enhancement submodule outputs the video data after noise reduction processing, and the routing submodule outputs the second RAW data.
  • the enhancement submodule sends the noise-reduced video data to the second preprocessing submodule, and the routing submodule also sends the second RAW data to the second preprocessing submodule.
  • the video data output by the enhancement sub-module is 4K30 data, which is used for browsing and recording;
  • the second RAW data output by the routing sub-module is 4K60 data, which is used for calculating 3A and possible camera needs.
  • the external ISP node performs fusion and noise reduction processing on the first RAW data collected by the image sensor
  • the difference between the video data output by the external ISP node and the first RAW data output by the image sensor Generally, there will be a certain delay. For example, there is a delay of one frame. For example, if the image sensor outputs the first RAW data at time t, the external ISP node outputs video data at time t-1 at the same time.
  • the external ISP node controls the synchronous output of the enhancement sub-module and the routing sub-module, that is, the noise-reduced video data and the second RAW data are synchronously transmitted to the second pre-processing sub-module.
  • the second preprocessing submodule processes the video data output by the enhancement submodule, and calculates the first exposure data based on the second RAW data, and adjusts an exposure parameter.
  • the processing of the video data output by the enhancement submodule by the second preprocessing submodule includes: performing preprocessing again on the video data output by the enhancement submodule, such as but not limited to LSC processing, BLC processing, At least one of BPC processing and color interpolation processing to further reduce the noise of the video data. Afterwards, RGB conversion is performed on the preprocessed video data again, and the video image obtained after the RGB conversion is compressed to obtain a YUV image.
  • preprocessing again on the video data output by the enhancement submodule such as but not limited to LSC processing, BLC processing, At least one of BPC processing and color interpolation processing to further reduce the noise of the video data.
  • RGB conversion is performed on the preprocessed video data again, and the video image obtained after the RGB conversion is compressed to obtain a YUV image.
  • the second preprocessing submodule described in the embodiment of the present application preprocess the video data output by the enhancement submodule again.
  • the second preprocessing submodule also The RGB conversion may be performed directly based on the video data output by the enhancement sub-module, which is not limited in this embodiment of the present application.
  • the second preprocessing sub-module determines the first exposure data based on the received second RAW data, and determines whether the current exposure level is reasonable according to the first exposure data, and then adjusts the exposure parameters of the camera if it is not reasonable.
  • the value range of the first exposure data is (0, 255).
  • the second preprocessing submodule compares the first exposure data with the exposure threshold, and if the difference between the first exposure data and the exposure threshold is greater than the threshold range, gradually adjusts the first Exposure data to obtain target exposure data.
  • the second preprocessing sub-module sends the target exposure data to the camera, so that the camera adjusts the exposure parameters of the image sensor, and the ultimate goal is to make the exposure data calculated according to the second RAW data close to or equal to the exposure threshold.
  • the adjustment step size may be set according to actual requirements.
  • the exposure threshold can be set according to actual needs.
  • the threshold range can also be set according to actual needs.
  • the exposure threshold is 128, the threshold range is [0,5], and the adjustment step is 4. If the first exposure data is 86, it means that the exposure parameter needs to be increased. At this time, the first exposure data can be adjusted according to the adjustment step to obtain a target exposure data of 90.
  • the second preprocessing sub-module sends the target exposure data 90 to the camera, so that the camera adjusts the exposure parameter of the image sensor to 90.
  • the exposure data is again counted according to the second RAW data received next time, and the exposure parameters of the image sensor are adjusted according to the above method until the counted exposure data is close to or equal to 128.
  • the exposure changes of the video frames can be smoothly transitioned.
  • the second preprocessing submodule sends the YUV image and target exposure data to the optimization processing submodule.
  • the target exposure data is the adjusted exposure data determined according to the first exposure data. For example, if the first exposure data is 100, and the second preprocessing sub-module determines that the exposure parameter of the image sensor needs to be adjusted to 200, then the target exposure data is 200.
  • the second preprocessing sub-module since the second preprocessing sub-module adjusts the exposure parameters of the image sensor, the gain of the video data obtained through the image sensor subsequently changes.
  • the YUV image received once is subjected to reasonable noise reduction processing. While adjusting the exposure parameters of the image sensor, the second preprocessing sub-module sends the target exposure data to the optimization processing sub-module, which is convenient for the optimization processing sub-module to determine the noise reduction parameters. , so as to perform reasonable noise reduction processing on the next received YUV image according to the noise reduction parameters.
  • the external ISP node includes multiple second target models, each of the multiple second target models corresponds to an exposure value range, and the exposure value corresponding to each second target model
  • the number of ranges can be one or more.
  • the second target model can be used for noise reduction processing.
  • the second preprocessing sub-module can also send the target exposure data to external Set the ISP node so that the external ISP node can determine the exposure value range to which the target exposure data fed back by the second preprocessing sub-module belongs, so that according to the determined exposure value range, the corresponding second target model can be selected from multiple second target models.
  • the target model, the selected second target model is used for the next noise reduction process.
  • the optimization processing submodule performs image optimization processing based on the received data.
  • the optimization processing sub-module optimizes the YUV image according to the target exposure data, such as performing noise reduction processing on the YUV image, so as to obtain a clear and bright video frame.
  • the optimization processing submodule sends the obtained video frames to display.
  • the optimization processing sub-module sends the video frames obtained after the image optimization processing to the display screen for display.
  • the second preprocessing sub-module also counts the AWB and AF based on the second RAW data, and sends the AWB to the optimization processing submodule, so that the optimization processing submodule can perform white balance adjustment during image optimization processing.
  • the second preprocessing submodule sends the AF to the camera, so that the camera can perform adjustment processing according to the AF.
  • the video data in the super night video mode, is fused and denoised through the external ISP, the processed video data is sent to the built-in ISP, and the original video data is provided for the built-in ISP, so that Because the built-in ISP can determine the target exposure data based on the original video data. In this way, the built-in ISP can generate clear video frames based on the processed video data provided by the external ISP and the determined target exposure data.
  • FIG. 7 is a schematic structural diagram of hardware according to an exemplary embodiment.
  • the hardware involved in this embodiment of the present application mainly includes a camera, an external ISP, and a built-in ISP.
  • the external ISP includes multiple interfaces, a routing unit, a front-end processing unit, and a back-end processing unit.
  • the routing unit is connected to the front-end processing unit, and the front-end processing unit is connected to the back-end processing unit.
  • the routing unit is used to execute the implementation shown in Figure 6.
  • the front-end processing unit is used to perform the function of the first preprocessing submodule in the embodiment of FIG.
  • the processing unit includes an IFE, and the back-end processing unit includes an NPU; the built-in ISP includes a first ISP front-end unit, a second ISP front-end unit, and an ISP back-end unit, the first ISP front-end unit is connected to the ISP back-end unit, and the second ISP front-end unit The unit is connected with the ISP back-end unit, and the first ISP front-end unit and the second ISP front-end unit are used to perform the function of the second preprocessing submodule in Figure 6, and the ISP back-end unit is used to perform the optimization processing submodule in Figure 6 Function.
  • the first ISP front-end unit includes IFE0
  • the second ISP front-end unit includes IFE1
  • the ISP back-end unit includes IPE.
  • the foregoing is only an exemplary illustration of the multiple units included in the external ISP and the internal ISP, but does not constitute a limitation to the structural components thereof.
  • the external ISP or the internal ISP may further include other units, which is not limited in this embodiment of the present application.
  • the external ISP receives the first RAW data from the camera.
  • the first RAW data may be from the main camera of the electronic device.
  • the first RAW data may also be from an auxiliary camera of the electronic device.
  • the first RAW data comes from the main camera of the electronic device, and in the case of 3x focus, the first RAW data comes from the auxiliary camera of the electronic device, which is not limited in this embodiment of the present application .
  • the external ISP receives the first RAW data from the camera through a mobile industry processor interface (mobile industry processor interface, Mipi)0.
  • a mobile industry processor interface mobile industry processor interface, Mipi0.
  • the external ISP copies and routes the first RAW data through the routing unit.
  • the external ISP first copies the first RAW data through the routing unit, so as to obtain another copy of the RAW data, which is referred to as the second RAW data here.
  • the routing unit performs routing processing on the two sets of RAW data. For example, one of the RAW data (such as the first RAW data) is transmitted to the front-end processing unit in the external ISP, and the front-end processing unit is based on this one.
  • the first RAW data is preprocessed, and the first RAW data obtained after preprocessing is sent to the back-end processing unit in the external ISP, and the back-end processing unit performs fusion and noise reduction processing; another RAW data (such as 2nd RAW data) for direct output to the built-in ISP.
  • another RAW data such as 2nd RAW data
  • the preprocessing of the RAW data by the front-end processing unit may refer to the embodiment shown in FIG. 6 above, and the fusion and noise reduction processing of the preprocessed RAW data by the back-end processing unit may also refer to the embodiment shown in FIG. 6 above.
  • the above is only described by taking the example of transmitting the first RAW data to the front-end processing unit in the external ISP and directly outputting the second RAW data to the built-in ISP.
  • the second RAW data may also be transmitted to the front-end processing unit in the external ISP, and the first RAW data may be directly output to the built-in ISP, which is not limited in this embodiment of the present application.
  • the back-end processing unit outputs the video data after the noise reduction processing, and the routing unit outputs the second RAW data.
  • the back-end processing unit sends the noise-reduced video data to the built-in ISP through the Mipi0 interface of the external ISP, and the routing unit sends another 60-frame second RAW data through the Mipi1 interface of the external ISP to the built-in ISP.
  • the built-in ISP receives the video data output by the back-end processing unit and the second RAW data output by the routing unit.
  • the built-in ISP receives the video data output by the back-end processing unit through the first ISP front-end unit, and then the first ISP front-end unit processes the video data output by the back-end processing unit, such as RGB conversion, and converts The final RGB image is compressed to obtain a YUV image.
  • the first ISP front-end unit may also perform preprocessing on the received video data before performing data format conversion. For example, preprocessing includes but is not limited to color correction, downsampling, demosaicing, etc.
  • the first ISP front-end unit transmits the YUV image to the ISP back-end unit for processing.
  • the ISP back-end unit is mainly used for image processing tasks such as hardware noise reduction, image cropping, noise reduction, color processing, detail enhancement, etc.
  • hardware noise reduction includes multi-frame noise reduction (multi-frame noise reduction, MFNR), multi-frame super resolution (multi-frame super resolution, MFSR).
  • the built-in ISP receives the second RAW data output by the routing unit through the second ISP front-end unit. After receiving the second RAW data, the second ISP front-end unit determines the first exposure data based on the second RAW data, and determines whether the current exposure degree is reasonable according to the first exposure data, and if not, determines the target exposure data, and according to The target exposure data is used to adjust the exposure parameters of the camera.
  • the second ISP front-end unit sends the target exposure data to the camera through the I2C interface, so as to control the camera. For its specific implementation, reference may be made to the embodiment shown in FIG. 6 .
  • the second ISP front-end unit also counts the AF based on the second RAW data, and sends the AF to the camera through the I2C interface, so that the camera can perform adjustment processing according to the AF.
  • the second ISP front-end unit also counts AWB, color and other data based on the second RAW data.
  • the second ISP front-end unit transmits data such as 3A and color to the ISP back-end unit, so that the ISP back-end unit can optimize the YUV image according to the data transmitted by the second ISP front-end unit, such as performing noise reduction processing on the YUV image, thereby obtaining Clear, crisp video frames.
  • the second ISP front-end unit can also send the target exposure data to the external ISP through the peripheral interface, so that the external ISP can select one of multiple second target models for noise reduction according to the target exposure data.
  • the second target model so that the next video data is subjected to noise reduction processing according to the selected second target model.
  • the peripheral interface may be a secure digital input and output (secure digital input and output, SDIO) interface.
  • the built-in ISP outputs the video frame through the ISP back-end unit, so as to display the video frame on the display screen.
  • FIG. 8 is a schematic diagram showing a flow of a method for generating video frames according to an exemplary embodiment.
  • the method may be applied to the above-mentioned electronic device, and the electronic device includes an image sensor, a first image signal processing module, and a second image signal processing module.
  • the second image signal processing module is a module with a video data processing function
  • the first image signal processing module is a module capable of acquiring video data and having a video data processing function.
  • the second image signal processing module includes an ISP integrated in the SOC, that is, includes the above-mentioned built-in ISP
  • the first image signal processing module includes an ISP outside the SOC, that is, includes the above-mentioned external ISP.
  • the electronic device implements a method for generating a video frame through an image sensor, a first image signal processing module, and a second image signal processing module, and the method may include the following:
  • Step 801 The image sensor outputs first RAW data.
  • the first RAW data is the original RAW data acquired by the image sensor, that is, the first RAW data is the original video data.
  • step 801 may include: detecting a night scene video shooting instruction through a camera application in the electronic device, where the night scene video shooting instruction is used to instruct video recording in a night scene mode.
  • the image sensor outputs first RAW data.
  • Step 802 the first image signal processing module acquires first RAW data.
  • Step 803 the first image signal processing module copies the first RAW data to obtain the second RAW data.
  • the second RAW data may be another copy of RAW data obtained by copying the RAW data by the routing submodule.
  • Step 804 The first image signal processing module performs image enhancement processing on the first RAW data to obtain video enhancement data.
  • the first RAW data includes long exposure data and short exposure data collected in the same time period.
  • the specific implementation of step 804 may include: the first image signal processing module fuses the long-exposure data and the short-exposure data to obtain fused RAW data.
  • the first image signal processing module performs noise reduction processing on the fused RAW data to obtain video enhancement data.
  • the specific implementation of the fusion processing of the long exposure data and the short exposure data by the first image signal processing module may include: the first image signal processing module inputs the long exposure data and the short exposure data into the first object model In the method, fusion processing is performed by the first object model, and the first object model can perform fusion processing on arbitrary long exposure and short exposure data.
  • the first image signal processing module fuses the long exposure data and the short exposure data, and before obtaining the fused RAW data, it also includes: the first image signal processing module performs preprocessing Processing, the preprocessing includes at least one of LSC processing, BLC processing, BPC processing, and color interpolation processing.
  • the specific implementation of the fusion processing of the long exposure data and the short exposure data by the first image signal processing module includes: the first image signal processing module performs fusion processing on the preprocessed long exposure data and short exposure data to obtain Fusion of RAW data.
  • the specific implementation of the first image signal processing module to perform noise reduction processing on the fused RAW data may include: the first image signal processing module inputs the fused RAW data into the second target model, and the second target model Perform noise reduction processing, and the second target model can perform noise reduction processing on any RAW data.
  • the first image signal processing module includes a plurality of second target models, and each second target model in the plurality of second target models corresponds to an exposure value range.
  • the first image signal processing module before the first image signal processing module inputs the fused RAW data into the second target model, it further includes: the first image signal processing module receives target exposure data, and the target exposure data is generated by the second image signal processing module Determined based on the first exposure data, the first exposure data is obtained by the second image signal processing module based on the second RAW data to perform statistics on the exposure data, and the target exposure data is used to adjust the exposure parameters of the image sensor of the electronic device.
  • the first image signal processing module selects a second target model from multiple second target models according to the target exposure data and the exposure value range corresponding to each second target model, and the selected second target model is used for noise reduction processing .
  • the above preprocessing process is performed by the first preprocessing submodule, and the fusion and noise reduction processing processes are performed by the enhancement submodule.
  • the video enhancement data refers to the video data output by the enhancement sub-module
  • the second RAW data refers to the video data output by the routing sub-module.
  • Step 805 The first image signal processing module sends the video enhancement data and the second RAW data to the second image signal processing module.
  • the first image signal processing module sends the video enhancement data and the second RAW data to the second preprocessing submodule respectively.
  • Step 806 The second image signal processing module generates a video frame based on the video enhancement data and the second RAW data.
  • step 806 includes: the second image signal processing module performs format conversion processing on the video enhancement data to obtain a YUV image.
  • the second image signal processing module determines target data based on the second RAW data, and the target data is used to adjust the image quality of the YUV image.
  • the second image signal processing module adjusts the YUV image based on the target data, and uses the adjusted YUV image as a video frame.
  • target data includes but not limited to 3A data, color.
  • the first RAW data is acquired by the first image signal processing module, and the first RAW data is copied to obtain the second RAW data.
  • Image enhancement processing is performed based on the first RAW data to obtain video enhancement data.
  • the second image signal processing module generates video frames based on the video enhancement data and the second RAW data. That is, image enhancement processing is performed by the first image signal processing module, and the first image signal processing module also provides the second image signal processing module with second RAW data that can be used to adjust the exposure parameters, so as to ensure that clear video frame.
  • the disclosed devices and methods may be implemented in other ways.
  • the system embodiments described above are only illustrative.
  • the division of the modules or units is only a logical function division.
  • multiple units or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the procedures in the methods of the above embodiments in the present application can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a computer-readable storage medium.
  • the computer program When executed by a processor, the steps in the above-mentioned various method embodiments can be realized.
  • the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form.
  • the computer-readable medium may at least include: any entity or device capable of carrying computer program codes to electronic equipment, recording media, computer memory, read-only memory (Read-Only Memory, ROM), RAM, electrical carrier signal, telecommunications signals and software distribution media.
  • entity or device capable of carrying computer program codes to electronic equipment, recording media, computer memory, read-only memory (Read-Only Memory, ROM), RAM, electrical carrier signal, telecommunications signals and software distribution media.
  • ROM read-only memory
  • RAM random access memory
  • electrical carrier signal telecommunications signals and software distribution media.
  • U disk mobile hard disk
  • magnetic disk or optical disk etc.
  • computer readable media may not be electrical carrier signals and telecommunication signals under legislation and patent practice.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Studio Devices (AREA)

Abstract

La présente demande se rapporte au domaine technique des terminaux, et divulgue un procédé et un appareil de génération de trame vidéo, un dispositif électronique et un support de stockage. Le procédé comprend les étapes suivantes : un capteur d'image délivre des premières données brutes ; un premier module de traitement de signal d'image obtient les premières données brutes ; le premier module de traitement de signal d'image copie les premières données brutes pour obtenir des secondes données brutes ; le premier module de traitement de signal d'image réalise un traitement d'amélioration d'image sur les premières données brutes pour obtenir des données d'amélioration vidéo ; le premier module de traitement de signal d'image envoie les données d'amélioration vidéo et les secondes données brutes à un second module de traitement de signal d'image ; et le second module de traitement de signal d'image génère une trame vidéo sur la base des données d'amélioration vidéo et des secondes données brutes. Selon la présente demande, le traitement d'amélioration d'image est réalisé au moyen du premier module de traitement de signal d'image, et le premier module de traitement de signal d'image fournit en outre, pour le second module de traitement de signal d'image, les secondes données brutes qui peuvent être utilisées pour ajuster un paramètre d'exposition, de sorte à pouvoir garantir qu'une trame vidéo transparente puisse être obtenue.
PCT/CN2022/116757 2021-11-05 2022-09-02 Procédé et appareil de génération de trame vidéo, dispositif électronique et support de stockage WO2023077938A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202111316825 2021-11-05
CN202111316825.0 2021-11-05
CN202210254250.2 2022-03-10
CN202210254250.2A CN116095509B (zh) 2021-11-05 2022-03-10 生成视频帧的方法、装置、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2023077938A1 true WO2023077938A1 (fr) 2023-05-11

Family

ID=86197906

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/116757 WO2023077938A1 (fr) 2021-11-05 2022-09-02 Procédé et appareil de génération de trame vidéo, dispositif électronique et support de stockage

Country Status (2)

Country Link
CN (1) CN116095509B (fr)
WO (1) WO2023077938A1 (fr)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040196390A1 (en) * 2003-04-02 2004-10-07 Matsushita Electric Industrial Co., Ltd. Image processing device and camera
CN109167915A (zh) * 2018-09-29 2019-01-08 南昌黑鲨科技有限公司 图像处理方法、系统及计算机可读存储介质
US20190087198A1 (en) * 2017-09-21 2019-03-21 Qualcomm Incorporated Fully extensible camera processing pipeline interface
CN110351489A (zh) * 2018-04-04 2019-10-18 展讯通信(天津)有限公司 生成hdr图像的方法、装置和移动终端
WO2020103503A1 (fr) * 2018-11-22 2020-05-28 Oppo广东移动通信有限公司 Procédé et appareil de traitement d'images de scène nocturne, dispositif électronique, et support de stockage
CN111601019A (zh) * 2020-02-28 2020-08-28 北京爱芯科技有限公司 图像数据处理模组及电子设备
JP2020137078A (ja) * 2019-02-26 2020-08-31 株式会社ユピテル システムおよびプログラム等
CN113448623A (zh) * 2021-06-29 2021-09-28 北京紫光展锐通信技术有限公司 图像帧的处理方法、电子设备、芯片和存储介质

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102780889B (zh) * 2011-05-13 2015-04-01 中兴通讯股份有限公司 视频图像处理方法、装置及设备
CN109146814B (zh) * 2018-08-20 2021-02-23 Oppo广东移动通信有限公司 图像处理方法、装置、存储介质及电子设备
CN110633755A (zh) * 2019-09-19 2019-12-31 北京市商汤科技开发有限公司 网络训练方法、图像处理方法及装置、电子设备
CN112819699A (zh) * 2019-11-15 2021-05-18 北京金山云网络技术有限公司 视频处理方法、装置及电子设备
CN112866802B (zh) * 2019-11-27 2022-03-15 深圳市万普拉斯科技有限公司 视频处理方法、装置、存储介质及计算机设备
CN113066020B (zh) * 2021-03-11 2024-07-16 Oppo广东移动通信有限公司 图像处理方法及装置、计算机可读介质和电子设备
CN113507598B (zh) * 2021-07-09 2023-07-11 Oppo广东移动通信有限公司 视频画面的显示方法、装置、终端及存储介质

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040196390A1 (en) * 2003-04-02 2004-10-07 Matsushita Electric Industrial Co., Ltd. Image processing device and camera
US20190087198A1 (en) * 2017-09-21 2019-03-21 Qualcomm Incorporated Fully extensible camera processing pipeline interface
CN110351489A (zh) * 2018-04-04 2019-10-18 展讯通信(天津)有限公司 生成hdr图像的方法、装置和移动终端
CN109167915A (zh) * 2018-09-29 2019-01-08 南昌黑鲨科技有限公司 图像处理方法、系统及计算机可读存储介质
WO2020103503A1 (fr) * 2018-11-22 2020-05-28 Oppo广东移动通信有限公司 Procédé et appareil de traitement d'images de scène nocturne, dispositif électronique, et support de stockage
JP2020137078A (ja) * 2019-02-26 2020-08-31 株式会社ユピテル システムおよびプログラム等
CN111601019A (zh) * 2020-02-28 2020-08-28 北京爱芯科技有限公司 图像数据处理模组及电子设备
CN113448623A (zh) * 2021-06-29 2021-09-28 北京紫光展锐通信技术有限公司 图像帧的处理方法、电子设备、芯片和存储介质

Also Published As

Publication number Publication date
CN116095509A (zh) 2023-05-09
CN116095509B (zh) 2024-04-12

Similar Documents

Publication Publication Date Title
WO2022262260A1 (fr) Procédé de photographie et dispositif électronique
WO2021052232A1 (fr) Procédé et dispositif de photographie à intervalle de temps
CN113810600B (zh) 终端的图像处理方法、装置和终端设备
WO2019183813A1 (fr) Dispositif et procédé de capture d'image
WO2021057277A1 (fr) Procédé de photographie dans une lumière sombre et dispositif électronique
WO2023015981A1 (fr) Procédé de traitement d'images et son dispositif associé
WO2023077939A1 (fr) Procédé et appareil de commutation de caméra, dispositif électronique et support de stockage
CN113810601B (zh) 终端的图像处理方法、装置和终端设备
CN111770282B (zh) 图像处理方法及装置、计算机可读介质及终端设备
WO2023160285A1 (fr) Procédé et appareil de traitement vidéo
CN118175436A (zh) 图像处理方法及其相关设备
WO2024174625A1 (fr) Procédé de traitement d'images et dispositif électronique
WO2024078275A1 (fr) Appareil et procédé de traitement d'image, dispositif électronique et support de stockage
WO2024032033A1 (fr) Procédé de traitement vidéo et dispositif électronique
WO2023160221A1 (fr) Procédé de traitement des images et dispositif électronique
CN115426449B (zh) 一种拍照方法和终端
CN116048323B (zh) 图像处理方法及电子设备
US20240155236A1 (en) Image processing method and electronic device
CN115705663B (zh) 图像处理方法与电子设备
WO2023077938A1 (fr) Procédé et appareil de génération de trame vidéo, dispositif électronique et support de stockage
CN115706869A (zh) 终端的图像处理方法、装置和终端设备
WO2023160220A1 (fr) Procédé de traitement d'image et dispositif électronique
CN117714835B (zh) 一种图像处理方法、电子设备及可读存储介质
CN116051368B (zh) 图像处理方法及其相关设备
CN115696067A (zh) 终端的图像处理方法、装置和终端设备

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE