WO2023005678A1 - 图像信号处理器、图像信号处理装置及方法、芯片、终端设备 - Google Patents

图像信号处理器、图像信号处理装置及方法、芯片、终端设备 Download PDF

Info

Publication number
WO2023005678A1
WO2023005678A1 PCT/CN2022/105651 CN2022105651W WO2023005678A1 WO 2023005678 A1 WO2023005678 A1 WO 2023005678A1 CN 2022105651 W CN2022105651 W CN 2022105651W WO 2023005678 A1 WO2023005678 A1 WO 2023005678A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
exposure
image signal
hdr
processor
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/105651
Other languages
English (en)
French (fr)
Inventor
王晓哲
杜建国
倪阳
关锐
杨红星
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Ziguang Zhanrui Communication Technology Co Ltd
Original Assignee
Beijing Ziguang Zhanrui Communication Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Ziguang Zhanrui Communication Technology Co Ltd filed Critical Beijing Ziguang Zhanrui Communication Technology Co Ltd
Publication of WO2023005678A1 publication Critical patent/WO2023005678A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H04N25/58—Control of the dynamic range involving two or more exposures
    • H04N25/587—Control of the dynamic range involving two or more exposures acquired sequentially, e.g. using the combination of odd and even image fields
    • H04N25/589—Control of the dynamic range involving two or more exposures acquired sequentially, e.g. using the combination of odd and even image fields with different integration times, e.g. short and long exposures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/79—Arrangements of circuitry being divided between different or multiple substrates, chips or circuit boards, e.g. stacked image sensors

Definitions

  • the present invention relates to the technical field of image signal processing, in particular to an image signal processor, an image signal processing device and method, a chip and a terminal device.
  • the software processing scheme controls the exposure parameters frame by frame to generate three frames of overexposure, normal and underexposure images, and then performs high dynamic fusion processing to generate high dynamic images.
  • This software processing method is to control the exposure at the frame level. Due to the user's hand shaking or moving objects within the shooting range, ghost images are prone to appear, and the user experience is not good. It is only used for taking pictures, and cannot be used for preview and video recording.
  • the hardware processing solution requires the support of HDR (High-Dynamic Range, high dynamic range) sensor.
  • the HDR sensor supports multiple exposures at line level and frame level, namely long exposure, medium exposure and short exposure.
  • the HDR sensor collects image data with different exposures and fuses them to generate a high-dynamic image.
  • the places with high brightness are not exposed, and the details of places with low brightness are still clearly visible.
  • Using the data collected by the HDR sensor and subsequent processing can perfectly solve the problem of taking pictures of high dynamic scenes and recording videos.
  • the online processing method is usually adopted in the prior art, and all long-exposure and medium-exposure image data before the arrival of the short-exposure image data need to be cached in the on-chip memory, which occupies a large amount of on-chip memory. This will lead to a sharp increase in the chip area, greatly increasing the hardware cost of the ISP (Image Signal Processor, Image Signal Processor).
  • ISP Image Signal Processor, Image Signal Processor
  • embodiments of the present invention provide an image signal processor and a chip, which can process images collected by an HDR sensor, save the demand for on-chip memory, and reduce the area of the chip.
  • the embodiments of the present invention also provide an image signal processing device and method, and a terminal device, which can effectively improve the degree of hardware miniaturization and reduce the space occupied by the hardware.
  • An image signal processor includes: a separation module, a RAW domain processing module, an HDR fusion module; the RAW domain processing module includes a first RAW domain processor corresponding to short-exposure image data, and other corresponding RAW domain processor for exposure duration image data;
  • the separation module is configured to receive the image data output by the HDR sensor, decompose the image data, and transmit the decomposed image data with different exposure durations to the corresponding RAW domain processor;
  • Each RAW domain processor in the RAW domain processing module processes the received image data in parallel, and the first RAW domain processor transmits the processed short-exposure image data to the HDR fusion module, and the other The RAW domain processor writes the processed exposure image data into the external memory;
  • the HDR fusion module is configured to, after receiving the short-exposure image data transmitted by the first RAW domain processor, read images of other exposure durations corresponding to the short-exposure image data from the external memory data, and perform fusion processing on the short-exposure image data and image data of other exposure durations corresponding to the short-exposure image data to obtain an HDR image.
  • the image data output by the HDR sensor is row-level exposure image data, and each exposure window covers one or more rows of images, and the long, medium and short exposure image data are interleaved and output in row units; or the output of the HDR sensor
  • the image data is frame-level exposure image data, and each exposure window covers one or more lines of images, and the long, medium and short exposure image data are interleaved and output in units of frames.
  • the exposure windows corresponding to different exposure times are different.
  • the image data output by the HDR sensor is the image data of each exposure duration output in sequence from long to short exposure duration.
  • the image data output by the HDR sensor is fusion data including image data with different exposure time lengths; the separation module is further configured to parse the fusion data into different The image data of the exposure time.
  • the RAW domain processor corresponding to other exposure duration image data includes: a second RAW domain processor corresponding to medium exposure image data, and/or a third RAW domain processor corresponding to long exposure image data.
  • a chip includes the aforementioned image signal processor.
  • An image signal processing device comprising the aforementioned image signal processor, and an external memory located on a different chip from the image signal processor;
  • the external memory is used for storing long exposure image data and medium exposure image data that need to be fused by the image signal processor.
  • the external memory is any one or more of the following: DDR SDRAM.
  • a terminal device comprising: an HDR sensor, a transmission interface, and the image signal processing device according to any one of claims 8 to 9;
  • the HDR sensor is used to collect image data with a high dynamic range
  • the transmission interface is used to transmit the high dynamic range image data collected by the HDR sensor to the image signal processing device;
  • the image signal processing device is configured to perform fusion processing on the high dynamic range image data to obtain an HDR image.
  • the transmission interface includes: a MIPI interface and a serial interface.
  • An image signal processing method comprising:
  • Decomposing the image data performing parallel processing on the decomposed image data of different exposure durations, and writing image data of exposure durations other than the processed short-exposure image data into an external memory;
  • the image data output by the HDR sensor is the image data of each exposure duration output in sequence from long to short exposure duration.
  • the image data output by the HDR sensor is fusion data including image data of different exposure durations; the method further includes: before decomposing the image data, parsing the fusion data into fusion data of different exposure durations image data.
  • An embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium is a non-volatile storage medium or a non-transitory storage medium, and a computer program is stored thereon, and the computer program is run by a processor When performing the steps of the above method.
  • An embodiment of the present invention also provides an electronic device, including a memory and a processor, the memory stores a computer program that can run on the processor, and the processor executes the steps of the above method when running the computer program .
  • the image signal processor and the chip provided by the embodiment of the present invention adopt an online + offline hardware architecture for the image data collected by the HDR sensor.
  • the image signal processor does not need an on-chip memory, but uses an external memory to buffer the image data to be processed , thus greatly saving the chip memory and reducing the hardware cost.
  • the image signal processing device and method based on the above image signal device can not only meet the requirements of HDR image processing, but also can effectively save hardware cost.
  • the terminal equipment using the above-mentioned image signal processing device can not only meet the performance requirements of previewing, photographing and video recording scenes, improve the camera performance of the terminal equipment, but also greatly reduce the equipment cost.
  • FIG. 1 is a functional block diagram of an image signal processor provided by an embodiment of the present invention
  • FIG. 2 is a flowchart of an image signal processing method provided by an embodiment of the present invention.
  • FIG. 3 is a schematic structural diagram of a terminal device provided by an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of the time sequence of processing image data of different exposures by the image signal processor in the embodiment of the present invention.
  • HDR sensors support multiple exposures at the line-level and frame-level.
  • Row-level exposure refers to different exposures for each row according to the needs of the current scene.
  • N1 rows are in long exposure
  • N2 rows are in medium exposure
  • N3 rows are in short exposure.
  • Frame-level exposure means that, taking frames as units, one frame exposure is completed according to certain exposure parameters, and then the exposure parameters are updated for the next frame exposure.
  • an online mode is adopted in the prior art.
  • the sensor is activated and the sensor starts to collect images.
  • the sensor outputs long-exposure image data in row units, and the ISP needs to cache the long-exposure image data into the on-chip memory.
  • the medium exposure starts.
  • the sensor outputs the medium exposure image data; the ISP still needs to cache the medium exposure image data into the on-chip memory.
  • the short exposure starts, and after the short exposure is completed, the sensor outputs short exposure image data.
  • the ISP will online fuse the previously cached long-exposure and medium-exposure images with the short-exposure images to generate the data of the line in the HDR image.
  • the entire HDR image is finally generated for subsequent photo taking or video recording. Due to the need for on-chip memory, the chip area has increased sharply, and the application environment of integration and small hardware space is not used.
  • the embodiment of the present invention provides an image signal processor and a chip, and also provides an image signal processing device and method, and terminal equipment for HDR sensor
  • the collected image data adopts an online + offline hardware architecture, and the image signal processor does not need on-chip memory, but uses an external memory to cache the image data to be processed, thereby saving chip memory and reducing hardware costs.
  • FIG. 1 it is a functional block diagram of an image signal processor according to an embodiment of the present invention.
  • the image signal processor 100 includes: a separation module 110 , a RAW domain processing module, and an HDR fusion module 130 .
  • the RAW domain processing module includes a first RAW domain processor 121 corresponding to short-exposure image data, and a RAW domain processor corresponding to other exposure duration image data; the RAW domain processor corresponding to other exposure duration image data may be It includes: a second RAW domain processor 122 corresponding to medium exposure image data, and/or a third RAW domain processor 123 corresponding to long exposure image data.
  • Figure 1 shows the RAW domain processors corresponding to three different exposure duration image data, namely: the first RAW domain processor 121 for processing short-exposure image data, and the second RAW domain processor 121 for processing medium-exposure image data processor 122, and a third RAW domain processor 123 for processing long exposure image data.
  • RAW domain processor included in the RAW domain processing module in FIG. 1 is just an example, and should not be construed as a limitation to the RAW domain processing module. In practical applications, there are several types of exposed image data, and there are several RAW domain processors, and each RAW domain processor corresponds to one type of exposed image data.
  • the processing of the received image data by the RAW domain processor mainly includes but not limited to: BPC (Bad Pixel Correct, bad pixel correction), BLC (Black Level Correct, black level correction), LSC (Lens Shading Correct, lens shading correction), etc.
  • BPC Bit Pixel Correct, bad pixel correction
  • BLC Black Level Correct, black level correction
  • LSC Lins Shading Correct, lens shading correction
  • the separation module 110 is configured to receive the image data output by the HDR sensor, decompose the image data, and transmit the decomposed image data with different exposure durations to the corresponding RAW domain processor. For example, in FIG. 1 , the separation module 110 transmits the decomposed long-exposure image data to the third RAW domain processor 123, transmits the decomposed medium-exposure image data to the second RAW domain processor 122, and decomposes the obtained The short-exposure image data of is transmitted to the first RAW domain processor 121.
  • the external memory 200 refers to a memory that exists independently of the physical entity of the image signal processor 100, for example, SDRAM (Synchronous Dynamic Random Access Memory, Synchronous Dynamic Random Access Memory), DDR SDRAM (Double Data Rate SDRAM, dual Double rate SDRAM, referred to as DDR), DDRII (second generation DDR) and so on.
  • SDRAM Synchronous Dynamic Random Access Memory
  • DDR SDRAM Double Data Rate SDRAM, dual Double rate SDRAM, referred to as DDR
  • DDRII second generation DDR
  • the HDR fusion module 130 is configured to read other exposure durations corresponding to the short exposure image data from the external memory 200 after receiving the short exposure image data transmitted by the first RAW domain processor 121 image data, and perform fusion processing on the short-exposure image data and image data of other exposure durations corresponding to the short-exposure image data to obtain an HDR image.
  • the image signal processor 100 can process image data output by various HDR sensors.
  • the HDR sensor may support row-level or frame-level exposure modes, that is, the image data output by the HDR sensor may be row-level exposure image data or frame-level exposure image data.
  • the row-level exposure refers to performing different exposures on each row according to the current scene requirements. For example, at the same time, there are N1 rows in long exposure, N2 rows in medium exposure, and N3 rows in short exposure. Within one frame time, image data of long, medium and short exposures are simultaneously acquired.
  • the frame-level exposure refers to completing the exposure of one frame according to certain exposure parameters in units of frames, and then updating the exposure parameters to perform the exposure of the next frame.
  • each exposure window can cover one or more rows of images, or one or more frames of images. For exposures of different durations, the corresponding exposure windows will also be different.
  • the following takes the row-level exposure mode as an example, assuming that the HDR sensor supports long, medium, and short exposures. Depending on the current environment, different exposure values can be set for long, medium, and short exposures.
  • the medium and short exposure wait. Until the exposure of the long exposure window ends, the Nth row starts the medium exposure; and so on, after the medium exposure ends, the Nth row starts the short exposure.
  • the HDR fusion module 130 in the image signal processor 100 will fuse the image data of the long, medium and short exposures only after the short exposure ends.
  • the HDR sensor has already output the long exposure data of (L2+L3) duration exposure and the medium exposure data of L3 duration exposure to the external memory 200 .
  • the HDR fusion module 130 After the HDR fusion module 130 receives the short-exposure image data of the N-th line, it needs to read the long- and medium-exposure image data corresponding to the N-th line from the external memory 200, and then three kinds of long-, medium-, and short-exposure Row images are fused.
  • the specific fusion processing manner may adopt the existing technology and some fusion technologies developed in the future, which are not limited in this embodiment of the present invention.
  • a 3A statistical module (not shown) corresponding to each RAW domain processor may also be included, and the 3A refers to AE (Automatic Exposure, automatic exposure) , AWB (Automatic white balance, automatic white balance), AF (Automatic Focus, automatic focus), the described 3A statistics module mainly extracts the above-mentioned parameters from the image data processed by the RAW domain processor.
  • a processing module for further processing the HDR image output by the HDR fusion module 130 such as a RAW domain processing module, an RGB domain processing module, and a YUV processing module, may also be included.
  • the processing modules of these different domains can be integrated into one processor, or they can be independent processors.
  • the embodiment of the present invention does not limit the specific processing content and processing methods of the processing modules in each domain, and can be based on specific applications. need to design.
  • YUV image data After being processed by the above-mentioned different domain processing modules, YUV image data can be finally generated, and the YUV image data can also be written into the external memory 200 or other memories for various subsequent applications.
  • HDR sensors can transmit image data in different ways, such as:
  • each pixel bit is the HDR sensor acquisition bit width, such as 10 bits.
  • the HDR sensor collects image data of three exposures, and sequentially transmits long, medium, and short exposure data.
  • each pixel is 20bit, which contains long, medium, and short exposure information.
  • the separation module 110 in FIG. 1 can separate and process the two forms of image data output by the above HDR sensor in different ways. details as follows:
  • the separation module 110 only needs to send the image data of different exposures to the corresponding RAW domain processor according to the agreement.
  • the separation module 110 needs to parse the high-bit fused data into separate RAW data of different exposures, and transmit them to respective corresponding RAW domain processors for processing.
  • This approach has two advantages: First, the RAW domain processor only needs to maintain the original processing bit width, such as 10bit. It does not need to be upgraded according to the bit width of the fusion data output by the HDR sensor, such as upgrading to 20bit. This not only maintains the consistency of the processing bit width on the pipeline, but also saves the upgrade cost of the algorithm module on the RAW domain processor. The second is that the 3A statistical module can be unified with the first transmission method, avoiding the design of two sets of mechanisms.
  • the above-mentioned image signal processor can be used as an independent chip, or as a part of the chip.
  • the image signal processor provided by the embodiment of the present invention adopts an online (Online) + offline (offline) architecture, and compared with the existing online (Online) architecture, its bandwidth will be increased.
  • the image data with long, medium and short exposures, in the online+offline architecture of the present invention has more long and medium exposure data writing and reading DDR.
  • bandwidth is calculated for the online+offline architecture of the present invention and the existing online architecture, so as to evaluate its impact on the operation of the image signal processor.
  • the recording frame rate is set to 30fps.
  • the HDR sensor captures three 4K RAW10bit images of long, medium and short exposures.
  • the image signal processor needs to write two 4K images of long exposure and medium exposure into DDR and read them out.
  • the data read in one frame is as follows:
  • the overall bandwidth of the image processor is:
  • the image signal processor and the chip provided by the embodiment of the present invention adopt an online + offline hardware architecture for the image data collected by the HDR sensor.
  • the image signal processor does not need an on-chip memory, but uses an external memory to buffer the image data to be processed , thus greatly saving the chip memory and reducing the hardware cost.
  • an embodiment of the present invention also provides an image signal processing device, which includes the image signal processor 100 described in FIG. 1 and an external memory 200 located on a different chip from the image signal processor 100 .
  • the external memory 200 is used to store the long-exposure image data and the medium-exposure image data that need to be fused by the image signal processor.
  • the external memory 200 may be, for example, SDRAM, DDR, DDRII (second generation DDR) and the like.
  • an embodiment of the present invention also provides an image signal processing method, as shown in FIG. 2 , which is a flowchart of the method, including the following steps:
  • Step S21 receiving image data output by the HDR sensor.
  • Step S22 decomposing the image data, performing parallel processing on the decomposed image data with different exposure durations, and writing image data with exposure durations other than the processed short-exposure image data into an external memory.
  • Step S23 after decomposing and obtaining the short-exposure image data, read out corresponding other exposure-duration image data from the external memory, perform fusion processing on the short-exposure image data and the other exposure-duration image data, and obtain HDR image.
  • the HDR sensor may support row-level or frame-level exposure modes, that is, the image data output by the HDR sensor may be row-level exposure image data or frame-level exposure image data.
  • the HDR sensor can transmit image data in different ways, one is to transmit different exposure data separately, and the other is to fuse and transmit multiple different exposure data inside the HDR sensor.
  • image data in different ways, one is to transmit different exposure data separately, and the other is to fuse and transmit multiple different exposure data inside the HDR sensor.
  • the method may further include: parsing the fusion data into image data of different exposure time lengths between the above step S21 and step S22.
  • the image signal processing device and method provided by the embodiments of the present invention can not only meet the requirements of HDR image processing, but also effectively save hardware costs.
  • an embodiment of the present invention also provides a terminal device, as shown in FIG.
  • the transmission interface includes: MIPI (Mobile Industry Processor Interface, mobile industry processor interface) interface 501 and serial interface 502.
  • MIPI interface 501 is a sensor standard interface for receiving parallel data
  • serial interface 502 is used for converting the parallel data into serial data for output.
  • the HDR sensor 400 is used to collect high dynamic range image data; the transmission interface transmits the high dynamic range image data collected by the HDR sensor 400 to the image signal processor 100, and the image
  • the signal processor 100 adopts an online+offline hardware architecture, and writes other exposure image data except the short-exposure image data into the external memory 200 without on-chip memory.
  • the image signal processor 100 outputs YUV image data to the memory 300 .
  • the memory 300 and the external memory 200 may be the same physical entity, or may be different physical entities.
  • the image signal processor 100 transmits the long-exposure image data of (L2+L3) lines collected by the HDR sensor 400 to the third RAW domain processor for processing, obtains 3A statistical information, and then writes it into the external memory 200.
  • the middle-exposure image data of line L3 is similarly processed by the second RAW domain processor.
  • the image processor 100 receives the short-exposure N-th line of image data, and the second RAW domain processor completes the online processing, the image processor 100 reads back the long-exposure N-th line of image data and the middle Expose the Nth row of image data, and perform fusion processing with the online short-exposure Nth row of data to generate an HDR image for subsequent algorithm module processing.
  • the first RAW domain processor processes the first row of short-exposure image data, it has completed the exposure time of the L3 row.
  • the second RAW domain processor processes the exposure image data in row L3+1, and writes it into the external memory 200
  • the third RAW domain processor processes the exposure image data in row L2+L3+1, and writes it into External memory 200.
  • the HDR fusion module receives the first row of short-exposure image data
  • the image signal processor 100 reads the first row of long-exposure image data and the first row of middle-exposure image data in parallel in the external memory 200, and the three kinds of exposure data are fused in real time. The fused data is transmitted to subsequent modules for processing.
  • the external memory 200 can be set for the image signal processor 100 higher priority.
  • the image signal processor 100 can support multiple exposures at the row level, and can also support multiple exposures at the frame level.
  • the terminal device based on the above image signal processor 100 can meet the performance requirements of preview, photo taking and video recording scenes. Requirements, improve the performance of the terminal equipment camera, and can also greatly reduce equipment costs.
  • the terminal device may be a device with a camera function, such as a mobile phone and a PAD.
  • the real-time processing of the collected high dynamic range image data on the mobile phone can meet the needs of ZSL (Zero Shutter Lag, zero shutter lag) photography and video recording.
  • ZSL Zero Shutter Lag, zero shutter lag
  • the above-mentioned image signal processor may correspond to a chip in a network device, such as a SoC (System-On-a-Chip, system on a chip), a baseband chip, a chip module, and the like.
  • SoC System-On-a-Chip, system on a chip
  • baseband chip a baseband chip
  • chip module a chip module
  • each module/unit contained in the product may be a software module/unit, or a hardware module/unit, or may be partly a software module/unit, partly is a hardware module/unit.
  • each module/unit contained therein may be realized by hardware such as a circuit, or at least some modules/units may be realized by a software program, and the software program Running on the integrated processor inside the chip, the remaining (if any) modules/units can be realized by means of hardware such as circuits; They are all realized by means of hardware such as circuits, and different modules/units can be located in the same component (such as chips, circuit modules, etc.) or different components of the chip module, or at least some modules/units can be realized by means of software programs,
  • the software program runs on the processor integrated in the chip module, and the remaining (if any) modules/units can be realized by hardware such as circuits; /Units can be realized by means of hardware such as circuits, and different modules/units can be located in the same component (such as chips, circuit modules, etc.) or different components in the terminal, or at least some modules/units can be implemented in the form of software programs Realization, the software program runs on
  • An embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium is a non-volatile storage medium or a non-transitory storage medium, and a computer program is stored thereon, and the computer program is executed by a processor During operation, the steps of the method provided by the embodiment corresponding to the above-mentioned FIG. 2 are executed.
  • An embodiment of the present invention also provides another image signal processing device, including a memory and a processor, the memory stores a computer program that can run on the processor, and the processor executes the computer program when running the computer program.
  • FIG. 2 corresponds to the steps of the method provided by the embodiment.
  • An embodiment of the present invention also provides an electronic device, including a memory and a processor, the memory stores a computer program that can run on the processor, and the processor executes the above-mentioned Figure 2 when running the computer program. Corresponding to the steps of the method provided in the embodiment.
  • the disclosed methods, devices and systems can be implemented in other ways.
  • the device embodiments described above are only illustrative; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may be physically included separately, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)

Abstract

一种图像信号处理器、图像信号处理装置及方法、芯片、终端设备，图像信号处理器包括：分离模块、RAW域处理模块、HDR融合模块；RAW域处理模块包括对应短曝光图像数据的第一RAW域处理器、以及对应其他曝光时长图像数据的RAW域处理器；分离模块接收HDR传感器输出的图像数据，并对图像数据进行分解，将分解得到的不同曝光时长的图像数据传输到对应的RAW域处理器；RAW域处理模块中的各RAW域处理器并行处理各自接收到的图像数据，并且第一RAW域处理器将处理后的短曝光图像数据传输给HDR融合模块，其他RAW域处理器将处理后的曝光图像数据写入外部存储器；HDR融合模块在收到第一RAW域处理器传输的短曝光图像数据后，从外部存储器中读取出与短曝光图像数据对应的其他曝光时长的图像数据，并对短曝光图像数据、以及与短曝光图像数据对应的其他曝光时长的图像数据进行融合处理，得到HDR图像。本发明方案可以有效减少芯片面积。

Description

图像信号处理器、图像信号处理装置及方法、芯片、终端设备
本申请要求2021年7月29日提交中国专利局、申请号为202110864724.0、发明名称为“图像信号处理器、图像信号处理装置及方法、芯片、终端设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及图像信号处理技术领域，具体地涉及一种图像信号处理器、还涉及一种图像信号处理装置及方法，还涉及一种芯片及一种终端设备。
背景技术
在日常生活中，高动态场景较为常见。比如室内场景拍摄窗户旁的人或物体，因为窗外和室内亮度不同，其亮度动态范围有较大差异。此时使用手机进行拍照，很容易出现窗外的物体过曝，而室内的人或物体却因光线不足导致某些细节不可见。在室外场景，逆光环境下也会出现这种问题。
对于高动态场景处理，一般有软件和硬件两种方案。软件处理方案通过逐帧控制曝光参数，产生过曝、正常和欠曝三帧图像，然后进行高动态融合处理，产生高动态图像。这种软件处理方式是按照帧级控制曝光，由于用户手抖或者拍摄范围内的运动物体，很容易出现鬼影，用户体验不佳，仅用于拍照，预览和录像都不能使用。硬件处理方案需要有HDR(High-Dynamic Range，高动态范围)传感器的支持。HDR传感器支持行级和帧级多种曝光，即长曝光、中曝光和短曝光。针对高动态场景，HDR传感器采集不同曝光的图像数据，进行融合，产生高动态图像，亮度高的地方不过曝，亮度低的地方细节依然清晰 可见。使用HDR传感器采集的数据及后续处理，可以完美地解决高动态场景拍照及视频录制问题。
针对HDR传感器的图像处理，现有技术中通常采用在线处理方式，需要在片上内存中缓存短曝光图像数据到来之前的所有长曝光和中曝光图像数据，其所占用的片上内存很大。这将导致芯片面积急剧增大，大大增加了ISP(Image Signal Prosessor，图像信号处理器)的硬件成本。
发明内容
本发明实施例一方面提供一种图像信号处理器及一种芯片，实现对HDR传感器采集的图像的处理，节省对片上内存的需求，减小芯片面积。
本发明实施例另一方面还提供一种图像信号处理装置及方法、终端设备，可以有效提升硬件小型化程度，减小硬件占用空间。
为此，本发明实施例提供如下技术方案：
一种图像信号处理器，所述图像信号处理器包括：分离模块、RAW域处理模块、HDR融合模块；所述RAW域处理模块包括对应短曝光图像数据的第一RAW域处理器、以及对应其他曝光时长图像数据的RAW域处理器；
所述分离模块，用于接收HDR传感器输出的图像数据，并对所述图像数据进行分解，将分解得到的不同曝光时长的图像数据传输到对应的RAW域处理器；
所述RAW域处理模块中的各RAW域处理器并行处理各自接收到的图像数据，并且所述第一RAW域处理器将处理后的短曝光图像数据传输给所述HDR融合模块，所述其他RAW域处理器将处理后的曝光图像数据写入外部存储器；
所述HDR融合模块，用于在收到所述第一RAW域处理器传输 的短曝光图像数据后，从所述外部存储器中读取出与所述短曝光图像数据对应的其他曝光时长的图像数据，并对所述短曝光图像数据、以及与所述短曝光图像数据对应的其他曝光时长的图像数据进行融合处理，得到HDR图像。
可选地，所述HDR传感器输出的图像数据为行级曝光图像数据，并且每个曝光窗口覆盖一行或多行图像，长中短曝光图像数据以行为单位交织输出；或者所述HDR传感器输出的图像数据为帧级曝光图像数据，并且每个曝光窗口覆盖一行或多行图像,长中短曝光图像数据以帧为单位交织输出。
可选地，不同曝光时间对应的曝光窗口不同。
可选地，所述HDR传感器输出的图像数据为按照曝光时长由长到短依次输出的各曝光时长的图像数据。
可选地，所述HDR传感器输出的图像数据为包含不同曝光时长的图像数据的融合数据；所述分离模块，还用于在对所述图像数据进行分解之前，将所述融合数据解析为不同曝光时长的图像数据。
可选地，所述对应其他曝光时长图像数据的RAW域处理器包括：对应中曝光图像数据的第二RAW域处理器、和/或对应长曝光图像数据的第三RAW域处理器。
一种芯片，包括前面所述的图像信号处理器。
一种图像信号处理装置，包括前面所述的图像信号处理器、以及与所述图像信号处理器位于不同芯片的外部存储器；
所述外部存储器用于存储所述图像信号处理器需要融合的长曝光图像数据和中曝光图像数据。
可选地，所述外部存储器为以下任意一种或多种：DDR SDRAM。
一种终端设备，包括：HDR传感器、传输接口、以及权利要求8至9任一项所述的图像信号处理装置；
所述HDR传感器，用于采集高动态范围的图像数据；
所述传输接口，用于将所述HDR传感器采集的高动态范围的图像数据传送给所述图像信号处理装置；
所述图像信号处理装置，用于对所述高动态范围的图像数据进行融合处理，得到HDR图像。
可选地，所述传输接口包括：MIPI接口、串行接口。
一种图像信号处理方法，所述方法包括：
接收HDR传感器输出的图像数据；
对所述图像数据进行分解，对分解得到的不同曝光时长的图像数据进行并行处理，并将处理后的短曝光图像数据之外的其他曝光时长图像数据写入外部存储器；
在分解得到短曝光图像数据后，从所述外部存储器中读取出对应的其他曝光时长图像数据，对所述短曝光图像数据和所述其他曝光时长图像数据进行融合处理，得到HDR图像。
可选地，所述HDR传感器输出的图像数据为按照曝光时长由长到短依次输出的各曝光时长的图像数据。
可选地，所述HDR传感器输出的图像数据为包含不同曝光时长的图像数据的融合数据；所述方法还包括：对所述图像数据进行分解之前，将所述融合数据解析为不同曝光时长的图像数据。
本发明实施例还提供一种计算机可读存储介质，所述计算机可读存储介质为非易失性存储介质或非瞬态存储介质，其上存储有计算机程序，所述计算机程序被处理器运行时执行上述方法的步骤。
本发明实施例还提供一种电子设备，包括存储器和处理器，所述存储器上存储有可在所述处理器上运行的计算机程序，所述处理器运行所述计算机程序时执行上述方法的步骤。
本发明实施例提供的图像信号处理器及芯片，对HDR传感器采集的图像数据采用在线+离线的硬件架构，所述图像信号处理器无需片上内存，而是采用外部存储器来缓存待处理的图像数据，从而大大节省了芯片内存，降低了硬件成本。
基于上述图像信号器的图像信号处理装置及方法，既可以满足HDR图像处理的需求，也可以有效节省硬件成本。
采用上述图像信号处理装置的终端设备，既可以满足预览、拍照和视频录制场景的性能要求，提升终端设备相机性能，而且还可以大大降低设备成本。
附图说明
图1是本发明实施例提供的图像信号处理器的一种原理框图；
图2是本发明实施例提供的图像信号处理方法的一种流程图；
图3是本发明实施例提供的终端设备的一种结构示意图；
图4是本发明实施例中图像信号处理器对不同曝光图像数据处理的时序示意图。
具体实施方式
为使本发明的上述目的、特征和有益效果能够更为明显易懂，下面结合附图对本发明的具体实施例做详细的说明。
HDR传感器支持行级和帧级的多种曝光。行级曝光是指，根据当前场景需求，对每行进行不同的曝光，同一时刻有N1行处于长曝，有N2行处于中曝，有N3行处于短曝。在一帧时间内，同时获取了长、中、短三种曝光的图像数据，整体缩短了采集数据时间，大大减弱了鬼影出现的可能性。帧级曝光是指，以帧为单位，按照某种曝光参数完成一帧曝光，然后再更新曝光参数进行下一帧曝光。虽然帧级曝光去鬼影效果弱于行级曝光，但它比软件方式效果也要好很多，也可以用于预览和视频录制。
对于HDR传感器多种曝光数据的处理，现有技术中采用在线模式。以行级曝光为例，当软件配置好三种曝光参数后，启动传感器，传感器开始采集图像。首先，传感器会以行为单位输出长曝光的图像数据，ISP中需要将长曝光图像数据缓存到片上内存中。当完成长曝光后，开始中曝光，中曝光完毕后传感器输出中曝光图像数据；ISP依然需要将中曝光图像数据缓存到片上内存中。紧接着开始短曝光，短曝光完成后，传感器输出短曝光图像数据。此时ISP会将之前缓存的对应行数的长曝光和中曝光图像与短曝光图像进行在线融合，产生HDR图像中该行数据。依次类推，最终产生整幅HDR图像，用于后续的拍照或者视频录制。由于需要片上内存，导致芯片面积急剧增大，不利用集成化及硬件空间较小的应用环境。
针对这种采用片上内存进行数据缓存的硬件结构所存在的问题，本发明实施例提供一种图像信号处理器及一种芯片，还提供一种图像信号处理装置及方法、终端设备，对HDR传感器采集的图像数据采用在线+离线的硬件架构，所述图像信号处理器无需片上内存，而是采用外部存储器来缓存待处理的图像数据，从而节省芯片内存，降低硬件成本。
如图1所示，是本发明实施例图像信号处理器的一种原理框图。
在该实施例中，所述图像信号处理器100包括：分离模块110、RAW域处理模块、HDR融合模块130。其中，所述RAW域处理模块包括对应短曝光图像数据的第一RAW域处理器121、以及对应其他曝光时长图像数据的RAW域处理器；所述对应其他曝光时长图像数据的RAW域处理器可以包括：对应中曝光图像数据的第二RAW域处理器122、和/或对应长曝光图像数据的第三RAW域处理器123。图1中示出了对应三种不同曝光时长图像数据的RAW域处理器，即：用于处理短曝光图像数据的第一RAW域处理器121、用于处理中曝光图像数据的第二RAW域处理器122，以及用于处理长曝光图像数据的第三RAW域处理器123。
需要说明的是，图1中对于所述RAW域处理模块包括的RAW域处理器只是一种举例，不应理解为对所述RAW域处理模块的限定。在实际应用中，有几类曝光的图像数据，就有几个RAW域处理器，每个RAW域处理器对应一类曝光的图像数据。
所述RAW域处理器对接收到的图像数据的处理主要包括但不限于：BPC(Bad Pixel Correct，坏像素校正)、BLC(Black Level Correct，黑电平校正)、LSC(Lens Shading Correct，镜头阴影校正)等。当然，在不同的应用中，所述RAW域处理器的功能可以有所不同，对此本发明实施例不做限定。
所述分离模块110用于接收HDR传感器输出的图像数据，并对所述图像数据进行分解，将分解得到的不同曝光时长的图像数据传输到对应的RAW域处理器。比如，图1中，所述分离模块110将分解得到的长曝光图像数据传输给第三RAW域处理器123，将分解得到的中曝光图像数据传输给第二RAW域处理器122，将分解得到的短曝光图像数据传输给第一RAW域处理器121。
所述RAW域处理模块中的各RAW域处理器并行处理各自接收到的图像数据，并且所述第一RAW域处理器121将处理后的短曝光图像数据传输给所述HDR融合模块130，所述其他RAW域处理器将处理后的曝光图像数据写入外部存储器200。所述外部存储器200是指独立于所述图像信号处理器100物理实体而存在的存储器，具体比如可以是SDRAM(Synchronous Dynamic Random Access Memory，同步动态随机存储器)、DDR SDRAM(Double Data Rate SDRAM，双倍速率SDRAM，简称为DDR)、DDRII(第二代DDR)等。
所述HDR融合模块130用于在收到所述第一RAW域处理器121传输的短曝光图像数据后，从所述外部存储器200中读取出与所述短曝光图像数据对应的其他曝光时长的图像数据，并对所述短曝光图像数据、以及与所述短曝光图像数据对应的其他曝光时长的图像数据进行融合处理，得到HDR图像。
需要说明的是，本发明实施例提供的图像信号处理器100可以处理各种HDR传感器输出的图像数据。比如，所述HDR传感器可以支持行级或帧级两种曝光模式，也就是说，所述HDR传感器输出的图像数据可以为行级曝光图像数据或者帧级曝光图像数据。所述行级曝光是指，根据当前场景需求，对每行进行不同的曝光，比如，同一时刻有N1行处于长曝，有N2行处于中曝，有N3行处于短曝。在一帧时间内，同时获取了长、中、短三种曝光的图像数据。所述帧级曝光是指，以帧为单位，按照某种曝光参数完成一帧曝光，然后再更新曝光参数进行下一帧曝光。另外，每个曝光窗口可以覆盖一行或多行图像、或者覆盖一帧或多帧图像。针对不同时长的暴光，对应的曝光窗口也会不同。
参照图1，下面以行级曝光模式为例，假设HDR传感器支持长、中、短三种曝光。根据当前环境，可以为长、中、短三种曝光设置不同的曝光值。当第N行数据处于长曝光窗口时，中、短曝光等待。直到长曝光窗口曝光结束，第N行开始中曝光；依次类推，中曝光结束后，第N行开始短曝光。
假设长曝光配置曝光时长为L1，中曝光时长为L2，短曝光时长为L3。对于第N行图像，需要等待短曝光结束后，图像信号处理器100中的HDR融合模块130才会对长、中、短三种曝光的图像数据进行融合。当第N行图像完成短曝光时，HDR传感器已经向外部存储器200输出(L2+L3)时长内曝光的长曝光数据和L3时长内曝光的中曝光数据。在HDR融合模块130接收到第N行的短曝光图像数据后，需要从外部存储器200读取出对应第N行的长、中曝光图像数据，然后将长、中、短曝光的三种第N行图像进行融合。具体的融合处理方式可以采用现有技术及未来发展的一些融合技术，对此本发明实施例不做限定。
进一步地，在本发明图像信号处理器的其他具体实施例中，还可包括对应每个RAW域处理器的3A统计模块(未图示)，所述3A是 指AE(Automatic Exposure，自动曝光)、AWB(Automatic white balance，自动白平衡)、AF(Automatic Focus，自动对焦)，所述3A统计模块主是从RAW域处理器处理后的图像数据中提取上述这些参数。
另外，在本发明图像信号处理器的其他具体实施例中，还可包括对HDR融合模块130输出的HDR图像做进一步处理的处理模块，比如RAW域处理模块、RGB域处理模块，YUV处理模块，这些不同域的处理模块可以集成为一个处理器，也可以为各自独立的处理器，而且，对于各域处理模块的具体处理内容及处理方式等，本发明实施例不做限定，可以根据具体应用需要来设计。
经过上述不同域处理模块的处理后，最终可产生YUV图像数据，所述YUV图像数据也可以写入所述外部存储器200中，也可以写入其他存储器中，以供后续各种不同应用使用。
在实际应用中，HDR传感器可以采用不同方式传输图像数据，比如：
一种是对不同曝光数据分别传输，每个像素bit位为HDR传感器采集位宽，比如10bit。比如，HDR传感器采集三种曝光的图像数据，并依次传递长、中、短曝光数据。
另一种是HDR传感器内部对多种不同曝光数据进行融合，产生高bit位宽的数据，比如融合后每个像素点为20bit，其中包含长、中、短曝光信息。
相应地，图1中的分离模块110可以针对上面HDR传感器输出的两种形式的图像数据，采用不同方式进行分离处理。具体如下：
对于上述第一种传输方式，分离模块110只需将不同曝光的图像数据按照约定发送到对应的RAW域处理器中即可。
对于上述第二种传输方式，分离模块110需要将高bit位的融合数据解析为分离的不种曝光的RAW数据，分别传输给各自对应的 RAW域处理器进行处理。这种做法有两个优点：一是RAW域处理器只需要保持原有的处理位宽即可，比如10bit。它不需要根据HDR传感器输出的融合数据的位宽进行升级，比如升级到20bit。这既保持了流水线上处理位宽的一致性，也节省了RAW域处理器上算法模块的升级成本。二是3A统计模块可以与第一种传输方式保持统一，避免设计两套机制。
在实际应用中，上述图像信号处理器可以作为独立的芯片，也可以作为芯片的一部分。
需要说明的是，本发明实施例提供的图像信号处理器，采用在线(Online)+离线(offline)的架构，与现有的在线(Online)架构相比较，它的带宽会有所增加。
以长、中、短三种曝光的图像数据，在本发明online+offline架构中，多了长、中曝光数据的写入和读出DDR。
下面分别针对本发明online+offline架构及现有online架构，对带宽进行计算，以评估其对所述图像信号处理器工作的影响。
以三曝光的4K视频录制为例，录制帧率设置为30fps。在一帧时间内，即33ms，HDR传感器采集长、中、短三种曝光的3幅4K RAW10bit图像。所述图像信号处理器需要将长、中两种曝光的2幅4K图像写入DDR，并将其读出。
其带宽情况如下：
33ms中，假设10％的vblank(场消隐)，采集数据有效时间为33*90％＝29.7ms。
一帧内写出数据如下：
长曝光4K RAW10+中曝光4K RAW10+HDR 4K YUV10，具体数据为：
3840*2160*10/8+3840*2160*10/8+3840*2160*1.5*10/8＝34.61MB ；
一帧内读入数据如下：
长曝光4K RAW10+中曝光4K RAW10，具体数据为：
3840*2160*10/8+3840*2160*10/8＝19.78MB；
因此，所述图像信号处理器的整体带宽为：
(3840*2160*10/8+3840*2160*10/8+3840*2160*1.5*10/8+3840*2160*10/8+3840*2160*10/8)/29.7*1000＝1831MB/s；
在现有的Online架构中，图像处理器的整体带宽为：
(3840*2160*1.5*10/8)/29.7*1000＝499MB/s。
由上述计算结果可以看出，本发明实施例提供的Online+offline硬件架构的图像信号处理器带宽虽然比现有online架构的图像信号处理器大了很多，但整体带宽并不大，这在手机芯片中是可以接受的。
本发明实施例提供的图像信号处理器及芯片，对HDR传感器采集的图像数据采用在线+离线的硬件架构，所述图像信号处理器无需片上内存，而是采用外部存储器来缓存待处理的图像数据，从而大大节省了芯片内存，降低了硬件成本。
相应地，本发明实施例还提供一种图像信号处理装置，该图像信号处理装置包括图1所述的图像信号处理器100、以及与所述图像信号处理器100位于不同芯片的外部存储器200。其中，所述外部存储器200用于存储所述图像信号处理器需要融合的长曝光图像数据和中曝光图像数据。所述外部存储器200比如可以是SDRAM、DDR、DDRII(第二代DDR)等。
相应地，本发明实施例还提供一种图像信号处理方法，如图2所示，是该方法的一种流程图，包括以下步骤：
步骤S21，接收HDR传感器输出的图像数据。
步骤S22，对所述图像数据进行分解，对分解得到的不同曝光时长的图像数据进行并行处理，并将处理后的短曝光图像数据之外的其他曝光时长图像数据写入外部存储器。
步骤S23，在分解得到短曝光图像数据后，从所述外部存储器中读取出对应的其他曝光时长图像数据，对所述短曝光图像数据和所述其他曝光时长图像数据进行融合处理，得到HDR图像。
在本发明实施例中，所述HDR传感器可以支持行级或帧级两种曝光模式，也就是说，所述HDR传感器输出的图像数据可以为行级曝光图像数据或者帧级曝光图像数据。
另外，所述HDR传感器可以采用不同方式传输图像数据，一种是对不同曝光数据分别传输，另一种是HDR传感器内部对多种不同曝光数据进行融合后传输，具体可参照前面的相应描述。
相应地，对于第二种方式，所述方法在上述步骤S21和步骤S22之间还可包括：将所述融合数据解析为不同曝光时长的图像数据。
本发明实施例提供的图像信号处理装置及方法，既可以满足HDR图像处理的需求，也可以有效节省硬件成本。
基于上述图像信号处理装置，本发明实施例还提供一种终端设备，如图3所示，该终端设备包括：HDR传感器400、传输接口、以及图像信号处理器100、外部存储器200、内存300。其中，所述传输接口包括：MIPI(Mobile Industry Processor Interface，移动产业处理器接口)接口501和串行接口502。其中，MIPI接口501为传感器标准接口，接收并行数据；串行接口502用于将并行数据转换为串行数据输出。
在该实施例中，所述HDR传感器400用于采集高动态范围的图像数据；所述传输接口将所述HDR传感器400采集的高动态范围的图像数据传送给所述图像信号处理器100，图像信号处理器100采用Online+offline的硬件架构，将短曝光图像数据之外的其他曝光图像数 据写入外部存储器200，无需片上内存。图像信号处理器100输出YUV图像数据到内存300。需要说明的是，所述内存300与所述外部存储器200可以是同一物理实体，也可以是不同的物理实体。
参照图4，假设图像信号处理器100将HDR传感器400采集的(L2+L3)行的长曝光图像数据，传输到第三RAW域处理器上进行处理，获取3A统计信息，然后写入外部存储器200。同样，L3行的中曝光图像数据，由第二RAW域处理器进行类似处理。当图像处理器100接收到短曝光的第N行图像数据，并由第二RAW域处理器在线完成处理后，图像处理器100从外部存储器200中分别读回长曝光第N行图像数据和中曝光第N行图像数据，与在线处理的短曝光第N行数据进行融合处理，产生HDR图像，供后续算法模块处理。
如图4所示，当第一RAW域处理器处理第一行短曝光图像数据时，其已完成L3行的曝光时间。此时第二RAW域处理器处理L3+1行中曝光图像数据，并将其写入外部存储器200；第三RAW域处理器处理L2+L3+1行长曝光图像数据，也将其写入外部存储器200。HDR融合模块在接收到第一行短曝光图像数据时，图像信号处理器100并行读取外部存储器200中第一行长曝光图像数据和第一行中曝光图像数据，三种曝光数据实时融合，融合后的数据传输到后续模块处理。
需要说明的是，在实际应用中，所述终端设备中可能还有其他芯片需要读取外部存储器200中的数据，为了满足实时处理的需求，外部存储器200可以为所述图像信号处理器100设置较高的优先级。
所述图像信号处理器100既可以支持行级别的多种曝光，也可以支持帧级别的多种曝光，基于上述图像信号处理器100的终端设备，既可以满足预览、拍照和视频录制场景的性能要求，提升终端设备相机性能，而且还可以大大降低设备成本。
需要说明的是，所述终端设备可以是手机、PAD等具有拍照功能的设备。比如，在手机上实现对采集的高动态范围图像数据的实时处 理，可以满足ZSL(Zero Shutter Lag，零快门滞后)拍照和视频录制需求。
在具体实施中，上述图像信号处理器可以对应于网络设备中的芯片，例如SoC(System-On-a-Chip，片上系统)、基带芯片、芯片模组等。
在具体实施中，关于上述实施例中描述的各个装置、产品包含的各个模块/单元，其可以是软件模块/单元，也可以是硬件模块/单元，或者也可以部分是软件模块/单元，部分是硬件模块/单元。
例如，对于应用于或集成于芯片的各个装置、产品，其包含的各个模块/单元可以都采用电路等硬件的方式实现，或者，至少部分模块/单元可以采用软件程序的方式实现，该软件程序运行于芯片内部集成的处理器，剩余的(如果有)部分模块/单元可以采用电路等硬件方式实现；对于应用于或集成于芯片模组的各个装置、产品，其包含的各个模块/单元可以都采用电路等硬件的方式实现，不同的模块/单元可以位于芯片模组的同一组件(例如芯片、电路模块等)或者不同组件中，或者，至少部分模块/单元可以采用软件程序的方式实现，该软件程序运行于芯片模组内部集成的处理器，剩余的(如果有)部分模块/单元可以采用电路等硬件方式实现；对于应用于或集成于终端的各个装置、产品，其包含的各个模块/单元可以都采用电路等硬件的方式实现，不同的模块/单元可以位于终端内同一组件(例如，芯片、电路模块等)或者不同组件中，或者，至少部分模块/单元可以采用软件程序的方式实现，该软件程序运行于终端内部集成的处理器，剩余的(如果有)部分模块/单元可以采用电路等硬件方式实现。
本发明实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质为非易失性存储介质或非瞬态存储介质，其上存储有计算机程序，所述计算机程序被处理器运行时执行上述图2对应实施例提供的方法的步骤。
本发明实施例还提供了另一种图像信号处理装置，包括存储器和 处理器，所述存储器上存储有可在所述处理器上运行的计算机程序，所述处理器运行所述计算机程序时执行上述图2对应实施例所提供的方法的步骤。
本发明实施例还提供了一种电子设备，包括存储器和处理器，所述存储器上存储有可在所述处理器上运行的计算机程序，所述处理器运行所述计算机程序时执行上述图2对应实施例所提供的方法的步骤。
需要指出的是，本发明实施例中涉及的“第一”、“第二”等词汇，仅用于区分描述的目的，而不能理解为指示或暗示相对重要性，也不能理解为指示或暗示顺序。
在本发明所提供的几个实施例中，应该理解到，所揭露的方法、装置和系统，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的；例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式；例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理包括，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
虽然本发明披露如上，但本发明并非限定于此。任何本领域技术 人员，在不脱离本发明的精神和范围内，均可作各种更动与修改，因此本发明的保护范围应当以权利要求所限定的范围为准。

Claims (16)

  1. 一种图像信号处理器，其特征在于，所述图像信号处理器包括：分离模块、RAW域处理模块、HDR融合模块；所述RAW域处理模块包括对应短曝光图像数据的第一RAW域处理器、以及对应其他曝光时长图像数据的RAW域处理器；
    所述分离模块，用于接收HDR传感器输出的图像数据，并对所述图像数据进行分解，将分解得到的不同曝光时长的图像数据传输到对应的RAW域处理器；
    所述RAW域处理模块中的各RAW域处理器并行处理各自接收到的图像数据，并且所述第一RAW域处理器将处理后的短曝光图像数据传输给所述HDR融合模块，所述其他RAW域处理器将处理后的曝光图像数据写入外部存储器；
    所述HDR融合模块，用于在收到所述第一RAW域处理器传输的短曝光图像数据后，从所述外部存储器中读取出与所述短曝光图像数据对应的其他曝光时长的图像数据，并对所述短曝光图像数据、以及与所述短曝光图像数据对应的其他曝光时长的图像数据进行融合处理，得到HDR图像。
  2. 根据权利要求1所述的图像信号处理器，其特征在于，
    所述HDR传感器输出的图像数据为行级曝光图像数据，并且每个曝光窗口覆盖一行或多行图像，长中短曝光图像数据以行为单位交织输出；或者
    所述HDR传感器输出的图像数据为帧级曝光图像数据，并且每个曝光窗口覆盖一行或多行图像,长中短曝光图像数据以帧为单位交织输出。
  3. 根据权利要求2所述的图像信号处理器，其特征在于，不同曝光时间对应的曝光窗口不同。
  4. 根据权利要求1所述的图像信号处理器，其特征在于，所述HDR传感器输出的图像数据为按照曝光时长由长到短依次输出的各曝光时长的图像数据。
  5. 根据权利要求1所述的图像信号处理器，其特征在于，所述HDR传感器输出的图像数据为包含不同曝光时长的图像数据的融合数据；
    所述分离模块，还用于在对所述图像数据进行分解之前，将所述融合数据解析为不同曝光时长的图像数据。
  6. 根据权利要求1至5任一项所述的图像信号处理器，其特征在于，所述对应其他曝光时长图像数据的RAW域处理器包括：
    对应中曝光图像数据的第二RAW域处理器、和/或对应长曝光图像数据的第三RAW域处理器。
  7. 一种芯片，其特征在于，包括如权利要求1至6任一项所述的图像信号处理器。
  8. 一种图像信号处理装置，其特征在于，包括如权利要求1至6任一项图像信号处理器、以及与所述图像信号处理器位于不同芯片的外部存储器；
    所述外部存储器用于存储所述图像信号处理器需要融合的长曝光图像数据和中曝光图像数据。
  9. 根据权利要求8所述的图像信号处理装置，其特征在于，所述外部存储器为以下任意一种或多种：DDR SDRAM。
  10. 一种终端设备，其特征在于，包括：HDR传感器、传输接口、以及权利要求8至9任一项所述的图像信号处理装置；
    所述HDR传感器，用于采集高动态范围的图像数据；
    所述传输接口，用于将所述HDR传感器采集的高动态范围的图 像数据传送给所述图像信号处理装置；
    所述图像信号处理装置，用于对所述高动态范围的图像数据进行融合处理，得到HDR图像。
  11. 根据权利要求10所述的终端设备，其特征在于，所述传输接口包括：MIPI接口、串行接口。
  12. 一种图像信号处理方法，其特征在于，所述方法包括：
    接收HDR传感器输出的图像数据；
    对所述图像数据进行分解，对分解得到的不同曝光时长的图像数据进行并行处理，并将处理后的短曝光图像数据之外的其他曝光时长图像数据写入外部存储器；
    在分解得到短曝光图像数据后，从所述外部存储器中读取出对应的其他曝光时长图像数据，对所述短曝光图像数据和所述其他曝光时长图像数据进行融合处理，得到HDR图像。
  13. 根据权利要求12所述的图像信号处理方法，其特征在于，所述HDR传感器输出的图像数据为按照曝光时长由长到短依次输出的各曝光时长的图像数据。
  14. 根据权利要求12所述的图像信号处理方法，其特征在于，所述HDR传感器输出的图像数据为包含不同曝光时长的图像数据的融合数据；所述方法还包括：
    对所述图像数据进行分解之前，将所述融合数据解析为不同曝光时长的图像数据。
  15. 一种计算机可读存储介质，所述计算机可读存储介质为非易失性存储介质或非瞬态存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器运行时执行权利要求12至14中任一项所述方法的步骤。
  16. 一种电子设备，包括存储器和处理器，所述存储器上存储有可在所述处理器上运行的计算机程序，其特征在于，所述处理器运行所述计算机程序时执行权利要求12至14中任一项所述方法的步骤。
PCT/CN2022/105651 2021-07-29 2022-07-14 图像信号处理器、图像信号处理装置及方法、芯片、终端设备 Ceased WO2023005678A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110864724.0 2021-07-29
CN202110864724.0A CN113596357B (zh) 2021-07-29 2021-07-29 图像信号处理器、图像信号处理装置及方法、芯片、终端设备

Publications (1)

Publication Number Publication Date
WO2023005678A1 true WO2023005678A1 (zh) 2023-02-02

Family

ID=78251890

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/105651 Ceased WO2023005678A1 (zh) 2021-07-29 2022-07-14 图像信号处理器、图像信号处理装置及方法、芯片、终端设备

Country Status (2)

Country Link
CN (1) CN113596357B (zh)
WO (1) WO2023005678A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025138813A1 (zh) * 2023-12-26 2025-07-03 荣耀终端股份有限公司 录像中抓拍图像的方法、电子设备及可读介质

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113596357B (zh) * 2021-07-29 2023-04-18 北京紫光展锐通信技术有限公司 图像信号处理器、图像信号处理装置及方法、芯片、终端设备
CN115942137B (zh) * 2022-12-22 2025-01-03 上海集成电路研发中心有限公司 增益调用系统及方法
CN116033274B (zh) * 2023-02-27 2023-06-09 江苏游隼微电子有限公司 一种兼容3d降噪的图像宽动态方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012084983A (ja) * 2010-10-07 2012-04-26 Mitsubishi Electric Corp 撮像装置及び画像撮像方法
US20160119575A1 (en) * 2014-10-24 2016-04-28 Texas Instruments Incorporated Image data processing for digital overlap wide dynamic range sensors
CN108055484A (zh) * 2017-12-14 2018-05-18 豪威科技(上海)有限公司 序列转换模块、高动态范围图像序列转换模块及转换方法
CN109104584A (zh) * 2017-06-21 2018-12-28 比亚迪股份有限公司 图像传感器及其高动态范围图像的获取方法
CN113596357A (zh) * 2021-07-29 2021-11-02 北京紫光展锐通信技术有限公司 图像信号处理器、图像信号处理装置及方法、芯片、终端设备

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107220956A (zh) * 2017-04-18 2017-09-29 天津大学 一种基于多幅具有不同曝光度的ldr图像的hdr图像融合方法
US20180376087A1 (en) * 2017-06-23 2018-12-27 Qualcomm Incorporated Using the same pixels to capture both short and long exposure data for hdr image and video
CN112788249B (zh) * 2017-12-20 2022-12-06 杭州海康威视数字技术股份有限公司 图像融合方法、装置、电子设备及计算机可读存储介质
US11637963B2 (en) * 2019-07-16 2023-04-25 Samsung Electronics Co., Ltd. Image sensor and image processing system comprising the same
CN113179374A (zh) * 2021-04-22 2021-07-27 深圳传音控股股份有限公司 图像处理方法、移动终端及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2012084983A (ja) * 2010-10-07 2012-04-26 Mitsubishi Electric Corp 撮像装置及び画像撮像方法
US20160119575A1 (en) * 2014-10-24 2016-04-28 Texas Instruments Incorporated Image data processing for digital overlap wide dynamic range sensors
CN109104584A (zh) * 2017-06-21 2018-12-28 比亚迪股份有限公司 图像传感器及其高动态范围图像的获取方法
CN108055484A (zh) * 2017-12-14 2018-05-18 豪威科技(上海)有限公司 序列转换模块、高动态范围图像序列转换模块及转换方法
CN113596357A (zh) * 2021-07-29 2021-11-02 北京紫光展锐通信技术有限公司 图像信号处理器、图像信号处理装置及方法、芯片、终端设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025138813A1 (zh) * 2023-12-26 2025-07-03 荣耀终端股份有限公司 录像中抓拍图像的方法、电子设备及可读介质

Also Published As

Publication number Publication date
CN113596357A (zh) 2021-11-02
CN113596357B (zh) 2023-04-18

Similar Documents

Publication Publication Date Title
CN113596357B (zh) 图像信号处理器、图像信号处理装置及方法、芯片、终端设备
KR102149187B1 (ko) 전자 장치와, 그의 제어 방법
WO2020057198A1 (zh) 图像处理方法、装置、电子设备及存储介质
CN110213502B (zh) 图像处理方法、装置、存储介质及电子设备
CN109005366A (zh) 摄像模组夜景摄像处理方法、装置、电子设备及存储介质
CN113744120B (zh) 多媒体处理芯片、电子设备和图像处理方法
CN113744117B (zh) 多媒体处理芯片、电子设备及动态图像处理方法
CN110012227B (zh) 图像处理方法、装置、存储介质及电子设备
CN105100631A (zh) 一种自动连续间隔拍照、摄像的方法及移动终端
CN117692761A (zh) 运动抓拍方法及电子设备
CN113873141B (zh) 电子设备
CN113744119B (zh) 多媒体处理芯片和电子设备
CN106911907A (zh) 一种伴飞光电跟踪测量吊舱高清压缩与存储装置
CN103402055A (zh) 移动终端用摄像头芯片、带摄像功能的移动终端及其拍照方法和装置
CN115665342A (zh) 图像处理方法、图像处理电路、电子设备和可读存储介质
WO2023010755A1 (zh) 一种hdr视频转换方法、装置、设备及计算机存储介质
CN112135068A (zh) 一种多输入视频融合处理的方法和装置
US10506161B2 (en) Image signal processor data traffic management
CN114945019B (zh) 数据传输方法、装置及存储介质
CN113873142B (zh) 多媒体处理芯片、电子设备和动态图像处理方法
CN108874343B (zh) 预览视窗共享方法及装置
CN108282600A (zh) 冰箱内部图像捕获装置以及冰箱
CN114157791A (zh) 一种hdr传感器和非hdr传感器的切换方法与电子设备
CN115118886A (zh) 图像处理芯片、应用处理芯片、电子设备及图像处理方法
CN113873143B (zh) 多媒体处理芯片和电子设备

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22848283

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 22848283

Country of ref document: EP

Kind code of ref document: A1