CN114666515A - Real-time acquisition device and method for original image data - Google Patents

Real-time acquisition device and method for original image data Download PDF

Info

Publication number
CN114666515A
CN114666515A CN202210323674.XA CN202210323674A CN114666515A CN 114666515 A CN114666515 A CN 114666515A CN 202210323674 A CN202210323674 A CN 202210323674A CN 114666515 A CN114666515 A CN 114666515A
Authority
CN
China
Prior art keywords
image data
original image
signal processing
processing unit
deserializer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210323674.XA
Other languages
Chinese (zh)
Inventor
赵国涛
饶文俊
潘曦明
王磊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Fullhan Microelectronics Co ltd
Original Assignee
Shanghai Fullhan Microelectronics Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Fullhan Microelectronics Co ltd filed Critical Shanghai Fullhan Microelectronics Co ltd
Priority to CN202210323674.XA priority Critical patent/CN114666515A/en
Publication of CN114666515A publication Critical patent/CN114666515A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/262Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50Constructional details
    • H04N23/54Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70Circuitry for compensating brightness variation in the scene
    • H04N23/741Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80Camera processing pipelines; Components thereof

Abstract

The invention provides a real-time acquisition device and a method of original image data, wherein the real-time acquisition device comprises an image sensor, a GMSL module, an FPGA and an image signal processing unit, wherein the image sensor is used for exposure to obtain the original image data; the GMSL module is used for transmitting and copying the original image data to obtain two identical original image data, and transmitting the two original image data to the FPGA and the image signal processing unit respectively; the image signal processing unit is used for configuring exposure parameters of the image sensor according to the original image data; the FPGA is used for acquiring and storing the original image data in real time so as to solve the problem that the ISP cannot capture and store the original data from the image sensor in real time.

Description

Real-time acquisition device and method for original image data
Technical Field
The invention relates to an image signal processing technology, in particular to a real-time acquisition device and a real-time acquisition method for original image data.
Background
In the current ISP (Image Signal Processing) field such as security protection, developers often need to acquire raw Image data of an Image sensor to perform lens shading correction, CCM correction, and the like, sometimes need to locate problems encountered in development through the raw Image data, and also need to acquire raw Image data to perform algorithm simulation in a chip design company, which all need the raw Image data of the Image sensor.
At present, due to reasons such as cost and the like, an ISP does not have enough resources to support a function of capturing original image data in real time, or only captures a frame of original image data, and then stores the original image data in a computer through protocols such as a serial port and the like, and the efficiency is particularly low and the stability is not good enough.
To support real-time capture of raw image data, one approach is for the ISP to create enough buffers at design time and support a USB-like high-speed transmission interface, which is not acceptable at cost-sensitive ISPs. In another method, an ISP (internet service provider) can support to capture a frame of original image data during design and then store the original image data in a serial port mode, the method cannot capture the original image data in real time, and the application scene is very limited.
In addition, general image sensor suppliers have FPGAs to acquire raw image data of the image sensor, but do not support automatic exposure, are single in adaptive scene, and cannot meet many situations.
Disclosure of Invention
The invention aims to provide a device and a method for acquiring original image data in real time, which solve the problem that an ISP (internet service provider) cannot capture and store the original data from an image sensor in real time.
In order to solve the above problems, the present invention provides a real-time acquisition apparatus for raw image data, which is characterized by comprising an image sensor, a GMSL module, an FPGA, and an image signal processing unit, wherein the image sensor is configured to obtain raw image data by exposure; the GMSL module is used for transmitting and copying the original image data to obtain two identical original image data, and respectively transmitting the two original image data to the FPGA and the image signal processing unit; the image signal processing unit is used for configuring exposure parameters of the image sensor according to the original image data; and the FPGA is used for acquiring and storing the original image data in real time.
Optionally, the image sensor transmits the raw image data to the GMSL module through an MIPI interface.
Optionally, the GMSL module includes a serializer, a deserializer, and a controller connected to each other,
the serializer is used for receiving the original image data transmitted by the image sensor and serializing the original image data;
the deserializer receives the serialized original image data transmitted by the deserializer, deserializes the original image data, copies the deserialized original image data to obtain two identical original image data, and transmits the two identical original image data to the FPGA and the image signal processing unit respectively; and
the controller is used for configuring the serializer and the deserializer so that the serializer is matched with the image sensor and the deserializer is respectively matched with the FPGA and the image signal processing unit.
Further, the serializer transmits the raw image data after the serialization processing to the deserializer through a GMSL link.
Further, the deserializer transmits the two same original image data to the FPGA and the image signal processing unit through the MIPI interface.
Further, the controller is connected with the serializer and the deserializer through an I2C interface.
Optionally, the image signal processing unit is configured to obtain a register parameter and an exposure parameter required by the image sensor to maintain an exposure state according to the original image data transmitted by the deserializer, configure the register parameter to the image sensor, and automatically control the exposure parameter of the image sensor according to the exposure parameter.
Optionally, the image signal processing unit is connected to the image sensor through an I2C interface.
Optionally, the image processing system further comprises an external storage device, and the FPGA is connected to the external storage device, so that the FPGA is configured to collect the original image data transmitted by the deserializer in real time and store the original image data in the external storage device.
The invention also provides a real-time acquisition method of the original image data, which adopts the real-time acquisition device of the original image data and comprises the following steps:
step S1: exposing the image sensor according to the initial exposure parameters to obtain original image data, and transmitting the original image data to the GMSL module;
step S2: the GMSL module transmits and copies the original image data to obtain two identical original image data;
step S3: the GMSL module transmits two identical original image data to the FPGA3 and an image signal processing unit respectively;
step S4: the image signal processing unit obtains new exposure parameters of the image sensor according to the original image data, the image sensor performs exposure according to the new exposure parameters to obtain original image data, and the original image data are transmitted to the GMSL module; meanwhile, the FPGA3 collects and stores the original image data in real time; and
step S5: steps S2 to S4 are executed in a loop.
Compared with the prior art, the invention has the following beneficial effects:
the invention provides a real-time acquisition device and a method of original image data, wherein the real-time acquisition device comprises an image sensor, a GMSL module, an FPGA and an image signal processing unit, wherein the image sensor is used for exposure to obtain the original image data; the GMSL module is used for transmitting and copying the original image data to obtain two identical original image data, and respectively transmitting the two original image data to the FPGA and the image signal processing unit; the image signal processing unit is used for configuring exposure parameters of the image sensor according to the original image data; the FPGA is used for acquiring and storing the original image data in real time so as to solve the problem that the ISP cannot capture and store the original data from the image sensor in real time.
Drawings
Fig. 1 is a block diagram of a real-time raw image data acquiring apparatus according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to an apparatus for acquiring raw image data in real time and a method thereof. The present invention will now be described in more detail with reference to the accompanying drawings, in which preferred embodiments of the invention are shown, it being understood that one skilled in the art may modify the invention herein described while still achieving the advantageous effects of the invention. Accordingly, the following description should be construed as broadly as possible to those skilled in the art and not as limiting the invention.
In the interest of clarity, not all features of an actual implementation are described. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. It will of course be appreciated that in the development of any such actual embodiment, numerous implementation-specific details must be set forth in order to achieve the developer's specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art.
In order to make the objects and features of the present invention more comprehensible, embodiments of the present invention are described in detail below with reference to the accompanying drawings. It is to be noted that the drawings are in a very simplified form and are all used in a non-precise ratio for the purpose of facilitating and distinctly aiding in the description of the embodiments of the invention.
Fig. 1 is a block diagram of a real-time raw image data acquiring apparatus according to this embodiment. As shown in fig. 1, the present embodiment provides a real-time obtaining apparatus of original image data, including an image sensor 1, a GMSL (Gigabit Multimedia Serial Links) module 2, an FPGA3, and an image signal processing unit 4, where the image sensor 1 is configured to obtain original image data through exposure; the GMSL module 2 is configured to receive the original image data provided by the image sensor 1, transmit and copy the original image data to obtain two identical pieces of the original image data, and transmit the two pieces of the original image data to the FPGA3 and the image signal processing unit 4, respectively; the image signal processing unit 4 configures parameters of the image sensor 1 according to the raw image data, so that the raw image data obtained by the image sensor 1 can be adaptive to the current scene; the FPGA3 is used to collect and store the raw image data in real time.
The image sensor 1 is, for example, a cmos image sensor, which is used to obtain raw image data through exposure. The original image data includes image data information, transmission rate, len number, and the like, wherein the data format of the original image data is an original image data raw format. In this embodiment, in order to transmit the raw image data simply and quickly, the image sensor 1 transmits the raw image data to the GMSL module 2 through an MIPI interface.
The GMSL module 2 includes a serializer 21, a deserializer 22 and a controller 23 that are connected to each other, where the serializer 21 is configured to receive the raw image data transmitted by the image sensor 1, serialize the raw image data, arrange the raw image data into packets, and transmit the packets to the deserializer 22 in a serial manner. The image sensor 1 transmits the raw image data to the serializer 21 through the MIPI interface, and the serializer 21 transmits a packet to the deserializer 22 through the GMSL link.
The deserializer 22 receives the original image data serialized by the serializer 21, deserializes the original image data again, restores the original image data to an original pattern, and copies the original image data restored to the original pattern to obtain two identical original image data. The deserializer 22 transmits two identical sets of the original image data to the FPGA3 and the image signal processing unit 4, respectively. In detail, the deserializer 22 transmits two identical sets of the original image data to the FPGA3 and the image signal processing unit 4 through the MIPI interface, respectively.
The controller 23 is, for example, an MCU (micro controller Unit), and the controller 23 is configured to configure the serializer 21 and the deserializer 22 so that the serializer 21 can be matched with the image sensor 1, and so that the deserializer 21 can be matched with the FPGA3 and the image signal processing Unit 4, respectively, so that the GMSL module 2 can normally access the corresponding image sensor 1. In detail, the controller 23 configures the input mode and the number of lens of the serializer 21 such that the input mode of the serializer 21 matches the output mode of the image sensor 1, and the number of lens of the serializer 21 matches the number of lens of the image sensor 1. The output mode of the deserializer 22 is matched with the input mode of the FPGA3 and also matched with the input mode of the image signal processing unit 4; the number of lens of said deserializer 22 matches the number of lens of said FPGA3, and also matches the number of lens of said image signal processing unit 4; and ensures that the transmission rate of the deserializer 22 is greater than the transmission rate of the image sensor 1 so that transmission can be performed normally. In the present embodiment, the controller 23 connects the serializer 21 and the deserializer 22 through an I2C interface.
The image signal processing unit 4 is configured to obtain register parameters and exposure parameters required by the image sensor 1 to maintain a normal exposure state according to the raw image data transmitted by the deserializer 22, configure the register parameters to the image sensor 1, and automatically control the exposure parameters of the image sensor 1 according to the exposure parameters, so that the raw image data obtained by the image sensor 1 can be adaptive to a current scene. In the present embodiment, the image signal processing unit 4 is connected to the image sensor 1 through an I2C interface.
The FPGA3 is connected to an external storage device, such as a computer, and is configured to collect the raw image data transmitted by the deserializer 22 in real time and store the raw image data in the external storage device.
The embodiment also provides a real-time acquisition method of original image data, which comprises the following steps:
step S1, the image sensor 1 exposes according to the initial exposure parameters to obtain the original image data, and transmits the original image data to the GMSL module 2;
step S2, the GMSL module 2 transmits and copies the original image data to obtain two identical original image data;
step S3, the GMSL module 2 transmits the two identical sets of original image data to the FPGA3 and the image signal processing unit 4, respectively;
step S4, the image signal processing unit 4 obtains a new exposure parameter of the image sensor 1 according to the original image data, the image sensor performs exposure according to the new exposure parameter to obtain original image data, and transmits the original image data to the GMSL module 2; meanwhile, the FPGA3 collects and stores the original image data in real time;
step S5, executing steps S2 to S4 in a loop until the exposure of the image sensor 1 is finished.
Step S2 includes the following steps:
first, the image sensor 1 transmits the raw image data to the serializer 21 through an MIPI interface, and the serializer 21 transmits a packet to a deserializer through a GMSL link.
Next, the deserializer 22 receives the original image data serialized by the serializer 21, deserializes the original image data again, restores the original image data to an original pattern, and copies the original image data restored to the original pattern, so as to obtain two identical original image data.
Then, the deserializer 22 sends two identical copies of the original image data to the FPGA3 and the image signal processing unit 4, respectively, through the MIPI interface.
Next, the controller 23 configures the serializer 21 and the deserializer 22 so that the serializer 21 can be matched with the image sensor 1, so that the deserializer 22 can be matched with the FPGA3 and the image signal processing unit 4, respectively, so that the GMSL module 2 can normally access the corresponding image sensor 1.
In summary, the present invention provides a device and a method for acquiring raw image data in real time, in which a GMSL link and an FPGA are used to acquire raw image data of an image sensor, wherein the FPGA can capture the raw image data in real time, so that the present invention can capture the raw image data and simultaneously support continuous acquisition of raw image data of the image sensor for image algorithm simulation.
In addition, unless otherwise specified or indicated, the description of the terms "first" and "second" in the specification are only used for distinguishing various components, elements, steps and the like in the specification, and are not used for representing logical relationships, sequence relationships or the like between the various components, elements, steps.
It is to be understood that while the present invention has been described in conjunction with the preferred embodiments thereof, it is not intended to limit the invention to those embodiments. It will be apparent to those skilled in the art from this disclosure that many changes and modifications can be made, or equivalents modified, in the embodiments of the invention without departing from the scope of the invention. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention are within the scope of the technical solution of the present invention, unless the technical essence of the present invention is not departed from the content of the technical solution of the present invention.

Claims (10)

1. The device for acquiring the original image data in real time is characterized by comprising an image sensor, a GMSL module, an FPGA and an image signal processing unit, wherein the image sensor is used for exposure to obtain the original image data; the GMSL module is used for transmitting and copying the original image data to obtain two identical original image data, and respectively transmitting the two original image data to the FPGA and the image signal processing unit; the image signal processing unit is used for configuring exposure parameters of the image sensor according to the original image data; and the FPGA is used for acquiring and storing the original image data in real time.
2. The apparatus for real-time acquisition of raw image data according to claim 1, wherein the image sensor transmits the raw image data to the GMSL module through an MIPI interface.
3. The apparatus for real-time acquisition of raw image data according to claim 1, wherein said GMSL module comprises a serializer, a deserializer and a controller connected to each other,
the serializer is used for receiving the original image data transmitted by the image sensor and serializing the original image data;
the deserializer receives the serialized original image data transmitted by the deserializer, deserializes the original image data, copies the deserialized original image data to obtain two identical original image data, and transmits the two identical original image data to the FPGA and the image signal processing unit respectively; and
the controller is used for configuring the serializer and the deserializer so that the serializer is matched with the image sensor and the deserializer is respectively matched with the FPGA and the image signal processing unit.
4. The apparatus for real-time acquisition of raw image data according to claim 3, wherein said serializer transmits said raw image data after serialization processing to said deserializer through a GMSL link.
5. The apparatus according to claim 3, wherein the deserializer transmits two identical copies of the raw image data to the FPGA and the image signal processing unit, respectively, via an MIPI interface.
6. The apparatus for real-time acquisition of raw image data as set forth in claim 3, wherein said controller is connected to said serializer and deserializer through an I2C interface.
7. The apparatus according to claim 1, wherein the image signal processing unit is configured to obtain a register parameter and an exposure parameter required by the image sensor to maintain an exposure state according to the raw image data transmitted by the deserializer, configure the register parameter to the image sensor, and automatically control the exposure parameter of the image sensor according to the exposure parameter.
8. The apparatus for real-time acquisition of raw image data as claimed in claim 1, wherein said image signal processing unit is connected to said image sensor through an I2C interface.
9. The apparatus for acquiring raw image data in real time according to claim 1, further comprising an external storage device, wherein the FPGA is connected to the external storage device, so that the FPGA is configured to acquire the raw image data transmitted by the deserializer in real time and store the raw image data in the external storage device.
10. A real-time acquisition method of raw image data, using the real-time acquisition apparatus of raw image data according to any one of claims 1 to 9, comprising the steps of:
step S1: exposing the image sensor according to the initial exposure parameters to obtain original image data, and transmitting the original image data to the GMSL module;
step S2: the GMSL module transmits and copies the original image data to obtain two identical original image data;
step S3: the GMSL module transmits two identical original image data to the FPGA3 and an image signal processing unit respectively;
step S4: the image signal processing unit obtains new exposure parameters of the image sensor according to the original image data, the image sensor performs exposure according to the new exposure parameters to obtain original image data, and the original image data are transmitted to the GMSL module; meanwhile, the FPGA3 collects and stores the original image data in real time; and
step S5: steps S2 to S4 are executed in a loop.
CN202210323674.XA 2022-03-29 2022-03-29 Real-time acquisition device and method for original image data Pending CN114666515A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210323674.XA CN114666515A (en) 2022-03-29 2022-03-29 Real-time acquisition device and method for original image data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210323674.XA CN114666515A (en) 2022-03-29 2022-03-29 Real-time acquisition device and method for original image data

Publications (1)

Publication Number Publication Date
CN114666515A true CN114666515A (en) 2022-06-24

Family

ID=82033497

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210323674.XA Pending CN114666515A (en) 2022-03-29 2022-03-29 Real-time acquisition device and method for original image data

Country Status (1)

Country Link
CN (1) CN114666515A (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070040035A1 (en) * 2003-11-13 2007-02-22 Metrologic Instruments, Inc. Method of and apparatus for dynamically and adaptively controlling system control parameters in a multi-mode image capture and processing system
US20160381338A1 (en) * 2015-04-07 2016-12-29 SZ DJI Technology Co., Ltd. System and method for storing image data in parallel in a camera system
KR20170142708A (en) * 2016-06-20 2017-12-28 주식회사 디엠씨시스 Camera system for adas
CN108924477A (en) * 2018-06-01 2018-11-30 北京图森未来科技有限公司 A kind of long-distance video processing method and system, video processing equipment
US20190132555A1 (en) * 2017-10-30 2019-05-02 Qualcomm Incorporated Methods and systems to broadcast sensor outputs in an automotive environment
CN110620903A (en) * 2019-08-28 2019-12-27 华为技术有限公司 Camera module, image acquisition system and method for outputting image data
CN111601019A (en) * 2020-02-28 2020-08-28 北京爱芯科技有限公司 Image data processing module and electronic equipment
CN111641765A (en) * 2020-06-03 2020-09-08 广东远峰汽车电子有限公司 Camera module, vehicle-mounted camera system and remote photographing method
CN213846868U (en) * 2020-12-31 2021-07-30 北京都是科技有限公司 Signal acquisition system

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070040035A1 (en) * 2003-11-13 2007-02-22 Metrologic Instruments, Inc. Method of and apparatus for dynamically and adaptively controlling system control parameters in a multi-mode image capture and processing system
US20160381338A1 (en) * 2015-04-07 2016-12-29 SZ DJI Technology Co., Ltd. System and method for storing image data in parallel in a camera system
KR20170142708A (en) * 2016-06-20 2017-12-28 주식회사 디엠씨시스 Camera system for adas
US20190132555A1 (en) * 2017-10-30 2019-05-02 Qualcomm Incorporated Methods and systems to broadcast sensor outputs in an automotive environment
CN108924477A (en) * 2018-06-01 2018-11-30 北京图森未来科技有限公司 A kind of long-distance video processing method and system, video processing equipment
CN110620903A (en) * 2019-08-28 2019-12-27 华为技术有限公司 Camera module, image acquisition system and method for outputting image data
CN111601019A (en) * 2020-02-28 2020-08-28 北京爱芯科技有限公司 Image data processing module and electronic equipment
CN111641765A (en) * 2020-06-03 2020-09-08 广东远峰汽车电子有限公司 Camera module, vehicle-mounted camera system and remote photographing method
CN213846868U (en) * 2020-12-31 2021-07-30 北京都是科技有限公司 Signal acquisition system

Similar Documents

Publication Publication Date Title
US9778315B2 (en) Testbench builder, system, device and method having agent loopback functionality
JPH01233634A (en) Program controller
CN110049294A (en) Based on the aloof from politics and material pursuits image frame grabber of Zynq high and processing system
WO2020215182A1 (en) Image data processing method, transmission method, transmission apparatus and data interface switching apparatus
Khalilzad et al. FPGA implementation of real-time Ethernet communication using RMII interface
US20150156099A1 (en) Technique for monitoring component processing
CN114666515A (en) Real-time acquisition device and method for original image data
US7765317B1 (en) System and methods for locating FPDU headers when markers are disabled
US20210367814A1 (en) Communication method and device based on parallel system, and terminal
CN109815181B (en) Method and device for converting any bit width based on AXI protocol interface
WO2022213883A1 (en) Synchronous exposure circuit, exposure method and exposure device for multiple visual sensors
CN110636219B (en) Video data stream transmission method and device
JP2008236217A (en) Flexray network connecting method, flexray network and electronic control unit for flexray network
CN111757051B (en) Data transmission method, intelligent terminal and radar system
Mielczarek et al. Framework for high-performance video acquisition and processing in mtca. 4 form factor
CN112449137A (en) Data transmission system and data transmission method based on mobile industry processor interface
Wang et al. A PCIe-based Hardware Acceleration Architecture of the Communication Protocol Stack
CN116954809B (en) Remote control method, device and system based on android device and electronic device
Augustine et al. FPGA based image readout from Camera using Ethernet protocol and Python
CN112422229B (en) Physical layer reconstruction system
CN216673171U (en) Image acquisition system and image acquisition device
CN117156244A (en) Video data processing system, method and vehicle
CN112422835B (en) High-speed image acquisition method, system, equipment and storage medium
CN112422771B (en) Resource resetting system
US20240135497A1 (en) Image enhancement method, chip and image acquisition device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination