TW202118276A - Image correction system with line buffer and implementation method thereof solving problem including making the read efficiency of DRAM poorer and producing a situation of repetitive pixel reading - Google Patents

Abstract

Disclosed are an image correction system with a line buffer and an implementation method thereof, which generally disclose a main memory capable of temporarily storing an input image; N line buffers stored in a cache memory and capable of being defined to include multiple N*M cache block groups, wherein each cache block group can temporarily store multiple pixels and an input image address of the most recently retrieved input image in the main memory; and an image corrector capable of generating an output image address of an output image, and to look up a input image address corresponding to the output image address in, for example, a LUT (look up table) in the memory, wherein if the cache hits, the image corrector reads relevant pixels of the input image from one of the line buffers, so as to subsequently perform a distortion correction algorithm. Accordingly, the disclosed is able to solve problem encountered in prior art, that is, storing image read from DRAM in line buffers which makes the read efficiency of DRAM poorer and produces a situation of repetitive pixel reading, moreover, the disclosed achieves advantages such as reducing the number of line buffers as well as storing cost of memory.

Description

Image correction system with line buffer and its implementation method

The present invention relates to memory access technology, in particular to a kind of cache memory by combining line buffer and cache block (Cache Block), which can be accessed by an image corrector (or image interpolator). "Image correction system with line buffer and its implementation method" of the input image recently captured by the main memory.

In order to calibrate an input image stored in DRAM, such as for distortion correction/dewarp input image, multiple pixels in the input image need to be read and temporarily stored in a line buffer of SRAM. It is called a line buffer), but in order to increase the clock rate of the DRAM, the same pixel in the input image will be repeatedly read (multi-read), which will cause the output frequency of the image ( The frame rate will be reduced. Furthermore, since the input image may be warping due to the image capturing lens being a fisheye lens, a scan line of the output image is at the corresponding position of the input image. The scan line may be curved, which means that the pixel data of the input image corresponding to the output image is not stored in consecutive addresses in the DRAM, resulting in a situation where the DRAM reading efficiency is not high (because compared to SRAM, DRAM access Slower).

For this reason, it has been proposed to store the images read from DRAM in multiple line buffers, which means that the number of line buffers can be increased to improve the poor reading efficiency of DRAM and the repeated reading of pixels. As shown in "Figure 1", but in this way, the cost of the line buffer will increase accordingly. For example, if you want to temporarily store the two data lines (L_1', L_2') shown in Figure 1, you need at least 7 A line buffer can only be achieved. Therefore, how to propose a better image correction system that can maintain the output frequency under the premise of balancing the access bandwidth of the line buffer and the cost of the line buffer. It is a problem to be solved.

To achieve the above objective, the present invention provides an image correction system with a line buffer, which mainly includes a main memory, an image corrector, and a cache memory respectively coupled to the main memory and the image corrector; wherein, The main memory can temporarily store an input image; the image corrector can generate an output image line data, and at least one line data of the output image can be obtained by performing a distortion correction algorithm on a plurality of line data of the input image; cache memory Can include N line buffers, where N line buffers can be defined as including multiple N*M-sized cache block groups, and each cache block group can temporarily store the most recently captured input image from the main memory The input image address of multiple pixels and multiple pixels, and the input image address stored in each cache block group can be the row number and address label of the input image on the same or different scan lines; image corrector It can also generate an output image address of the line data of the output image, and the output image address can be the row number and address label of one of the line data of the output image, and search for the input image address corresponding to the output image address. To determine whether the input image address stored in one of the line buffers (or temporarily stored in one of the cache block groups) is a cache hit, if the cache hits, the image corrector can retrieve one of the caches from one of the line buffers. Take the block group to read the relevant pixels of the input image to continue the deformation correction algorithm such as interpolation.

Therefore, after the present invention is implemented, it can at least solve the problem of storing images read from the DRAM in the line buffer in the conventional method, and the DRAM reading efficiency is poor and the pixels are repeatedly read. At the same time, it can reduce The number of line buffers has the beneficial effect of reducing the storage cost of the memory.

In order for your reviewer to have a clear understanding of the purpose, technical features and effects of the present invention after implementation, the following descriptions and illustrations are used for illustration, please refer to it.

大小的快取區塊組，各快取區塊組用於暫存主記憶體10最近擷取之輸入影像的多個像素(the most-used pixel)及前物多個像素所對應的的一輸入影像位址，且各快取區塊組儲存的輸入影像位址，可經組態為輸入影像於相同或不同掃描線的列編號及位址標籤； (4)  影像校正器20亦用於產生輸出影像之線資料的一輸出影像位址，且輸出影像位址可被定義為輸出影像之其中一線資料的列編號及位址標籤，並查找輸出影像位址所對應的輸入影像位址，以決定線緩衝器(LB_1~LB_N)所儲存的輸入影像位址(或可稱決定其中一快取區塊組所暫存的輸入影像位址)是否快取命中(Cache Hit)； (5)  承上，若快取命中，則影像校正器20可被組態為從線緩衝器(LB_1~LB_N)的其中一快取區塊組讀取輸入影像的相關像素，以續行變形校正演算法； (6)  承上，若並未快取命中，意即快取失誤(Cache Miss)，則影像校正器20可被組態為從主記憶體10對應的輸入影像位址讀取輸入影像的相關像素，並令主記憶體10最近擷取之輸入影像的輸入影像位址，同步更新至線緩衝器(LB_1~LB_N)，以利後續快取像素資料。 (7)  另，本發明在一較佳實施例中，前述的變形校正演算法可為一雙線性內插(bilinear interpolation)或一最臨近內插法(nearest neighbor interpolation)的內插演算法(interpolation)，但並不以此為限。Please refer to "Figure 2", which is a system architecture diagram of the present invention. The present invention proposes an image correction system 1 with a line buffer, which mainly includes a main memory 10, an image corrector 20, and are respectively coupled to the main The memory 10 and a cache memory 30 of the image corrector 20. The aforementioned coupling can be, for example, electrical connection or information connection, but is not limited to this. Among them: (1) The main memory 10 is provided for temporary An input image is stored (for example, an image is captured by an image capturing device with a fisheye lens and input), and the main memory 10 can be, for example, a dynamic random access memory (DRAM); (2) For the image corrector 20 To generate line data of an output image, and at least one line data of the output image can be obtained by performing a distortion correction algorithm on a plurality of line data of the input image; (3) The cache memory 30 can be a static random access memory (SRAM), which can include N line buffers (LB_1~LB_N). The aforementioned N line buffers (LB_1~LB_N) can be defined as including multiple

Cache block groups of different sizes. Each cache block group is used to temporarily store the most-used pixels of the most-used pixel of the input image captured by the main memory 10 and one of the pixels corresponding to the predecessor. Input image address, and the input image address stored in each cache block group can be configured as the row number and address label of the input image in the same or different scan lines; (4) The image corrector 20 is also used Generate an output image address of the line data of the output image, and the output image address can be defined as the row number and address label of one of the line data of the output image, and search for the input image address corresponding to the output image address, To determine whether the input image address stored in the line buffers (LB_1~LB_N) (or can be said to determine the input image address temporarily stored in one of the cache block groups) whether the cache hit (Cache Hit); (5) In addition, if the cache hits, the image corrector 20 can be configured to read the relevant pixels of the input image from one of the cache block groups of the line buffers (LB_1~LB_N) to continue the line distortion correction algorithm (6) Continuing, if there is no cache hit, which means a cache miss (Cache Miss), the image corrector 20 can be configured to read the input image from the input image address corresponding to the main memory 10 Relevant pixels, and the input image address of the input image recently captured by the main memory 10 is synchronized to the line buffers (LB_1~LB_N) to facilitate subsequent fast-fetching of pixel data. (7) In addition, in a preferred embodiment of the present invention, the aforementioned deformation correction algorithm can be a bilinear interpolation or a nearest neighbor interpolation interpolation algorithm (interpolation), but not limited to this.

大小的快取區塊組，各快取區塊組可暫存主記憶體10最近擷取之一輸入影像的多個像素及前述多個像素的一輸入影像位址，且各快取區塊組儲存的輸入影像位址，可經組態為輸入影像於相同或不同掃描線的列編號及位址標籤，舉例而言，輸入影像位址可被表示為例如(line6, 位址200)，而各快取區塊組可儲存例如64x64像素(pixel)的影像資料，但並不以此為限； (2)  產生輸出影像位址(步驟S20)：影像校正器20產生一輸出影像之一線資料的一輸出影像位址，輸出影像的至少一線資料可由暫存於主記憶體10的輸入影像的複數個線資料執行一變形校正演算法而得，且輸出影像位址可被定義為輸出影像之其中一線資料的列編號及位址標籤，舉例而言，輸出影像位址可被表示為(line5, 位址100)； (3)  承上，有關步驟S10之更具體說明，還請參閱「第4圖」~「第6圖」之實施示意圖，首先，假設影像校正器20係以漸進式掃描方式(progressive-scan)，由左至右(x方向)且由上至下(y方向)的方向作輸出，則影像校正器20所輸出的第1條資料線，可能要在完成輸入影像中的第1條資料線L_1(由於輸入影像有變形情形，故資料線L_1可能呈彎曲狀)的讀取後，方能進行影像校正而產生，同理，影像校正器20所輸出的第2條資料線，可能要在完成輸入影像中的第2條資料線L_2(其亦可能呈彎曲狀)的讀取後，方能進行影像校正而產生； (4)  承上，依此可知，在步驟S10執行時，可令被定義出的多個快取區塊組，組態為由線緩衝器LB_4的第1個快取區塊CB_41開始暫存資料線L_1的部分像素資料及其輸入影像位址，接著，由線緩衝器LB_3的第1個快取區塊CB_31暫存，至此，即如「第4圖」所示，而圖中所示的快取區塊(CB_41、CB_31)相當於定義其為線緩衝器(LB_1~ LB_4)的第一快取區塊組CBG_1，但並不以此圖所示限制第一快取區塊組CBG_1的大小； (5)  其後，可接續由線緩衝器LB_3的第2個快取區塊CB_32暫存資料線L_1的部分像素資料及其輸入影像位址，接著，由線緩衝器LB_2的第2個快取區塊CB_22暫存，至此，即如「第5圖」所示，而圖中所示的快取區塊(CB_32、CB_22)相當於定義其為線緩衝器(LB_1~ LB_4)的第二快取區塊組CBG_2，但並不以此圖所示限制第二快取區塊組CBG_2的大小； (6)  其後，可接續由線緩衝器LB_2的第2個快取區塊CB_23暫存，以此類推，當快取記憶體30完成輸入影像中的資料線L_1及資料線L_2的暫存後，即如「第6圖」所示，相當於將線緩衝器(LB_1~ LB_5)定義出可沿著Y軸暫存輸入影像的多個快取區塊組，例如「第4圖」所示為驅動線緩衝器(LB_1~ LB_4)的第一快取區塊組CBG_1，「第5圖」所示為驅動線緩衝器(LB_1~ LB_4)的第二快取區塊組CBG_2，「第6圖」所示則為依序驅動線緩衝器(LB_1~ LB_5)至其最後一個快取區塊組，但圖中所示僅為舉例，並不以圖中所示限制各快取區塊組的大小； (7)  承上，至此本實施例僅驅動不多於5個線緩衝器(LB_1~ LB_5)暫存輸入影像，相比於習知技術需依據各列掃瞄線的方向，以至少7個線緩衝器逐一儲存資料線L_1及資料線L_2(甚至包含其它不在資料線上的像素)之像素資料及其輸入影像位址的作法，本發明至少可達成減少線緩衝器之使用及減輕像素重複讀取之情形的有利功效； (8)  快取命中檢查(步驟S30)：影像校正器20查找輸出影像位址所對應的一輸入影像位址，以決定其中一線緩衝器(LB_1~LB_N)所儲存的輸入影像位址（或可稱決定其中一快取區塊組所暫存的輸入影像位址）是否快取命中； (9)  承步驟S30，若快取命中，則接續執行快取像素及校正影像(步驟S40)：影像校正器20可被組態為從其中一線緩衝器(LB_1~LB_N)的其中一快取區塊組讀取輸入影像的相關像素，以續行變形校正演算法； (10)   承上，若步驟S30的結果為快取失誤，則接續執行讀取主記憶體及更新線緩衝器(步驟S50)：影像校正器20可被組態為從主記憶體10對應的輸入影像位址讀取輸入影像的相關像素，並令主記憶體10最近擷取之輸入影像的輸入影像位址，同步更新至線緩衝器(LB_1~LB_N)，以利後續快取像素資料。 (11)   另，本發明在一較佳實施例中，前述的變形校正演算法可為一雙線性內插或一最臨近內插法的內插演算法(interpolation)，但並不以此為限。Please refer to "Figure 3", which is the system flow chart of the present invention. Please also refer to "Figure 2". The present invention proposes an implementation method S of an image correction system with a line buffer for including a main memory 10. Implementation of an image correction system of an image corrector 20 and a cache memory 30, including: (1) Setting the cache memory (step S10): The cache memory 30 is configured to be coupled to the main memory respectively The body 10 and the image corrector 20. The cache memory 30 may include N line buffers (LB_1~LB_N), of which N line buffers (LB_1~LB_N) can be defined as including multiple

A cache block group of a large size, each cache block group can temporarily store a plurality of pixels of an input image recently captured by the main memory 10 and an input image address of the aforementioned plurality of pixels, and each cache block The input image address stored in the group can be configured as the row number and address label of the input image on the same or different scan lines. For example, the input image address can be expressed as (line6, address 200), Each cache block group can store, for example, 64x64 pixel (pixel) image data, but it is not limited to this; (2) Generate an output image address (step S20): The image corrector 20 generates an output image line An output image address of the data, at least one line of data of the output image can be obtained by performing a distortion correction algorithm on a plurality of line data of the input image temporarily stored in the main memory 10, and the output image address can be defined as the output image The row number and address label of one of the first-line data. For example, the output image address can be expressed as (line5, address 100); (3) Continuing from the above, for more specific instructions on step S10, please refer to " Figure 4" ~ "Figure 6" are schematic diagrams of implementation. First, assume that the image corrector 20 adopts a progressive-scan method, from left to right (x direction) and from top to bottom (y direction) The first data line output by the image corrector 20 may be the first data line L_1 in the input image (due to the deformation of the input image, the data line L_1 may be curved) After reading, the image can be corrected and generated. Similarly, the second data line output by the image corrector 20 may be completed after the second data line L_2 in the input image (which may also be curved ) Can only be generated by image correction after reading; (4) From the above, it can be seen from this that when step S10 is executed, multiple defined cache block groups can be configured as line buffers The first cache block CB_41 of the LB_4 starts to temporarily store part of the pixel data of the data line L_1 and its input image address, and then the first cache block CB_31 of the line buffer LB_3 is temporarily stored, so far, that is As shown in "Figure 4", the cache block (CB_41, CB_31) shown in the figure is equivalent to the first cache block group CBG_1 defined as the line buffer (LB_1~LB_4), but not Limit the size of the first cache block group CBG_1 as shown in this figure; (5) After that, the second cache block CB_32 of the line buffer LB_3 can be used to temporarily store part of the pixel data of the data line L_1 and its Enter the image address, and then temporarily store it in the second cache block CB_22 of the line buffer LB_2. So far, as shown in "Figure 5", the cache blocks (CB_32, CB_22) ) Is equivalent to defining it as the second cache block group CBG_2 of the line buffer (LB_1~LB_4), but does not limit the size of the second cache block group CBG_2 as shown in this figure; (6) After that, the second cache block CB_23 of the line buffer LB_2 can be temporarily stored, and so on. When the cache memory 30 completes the temporary storage of the data line L_1 and the data line L_2 in the input image, that is As shown in "Figure 6", it is equivalent to defining the line buffers (LB_1~LB_5) to define multiple cache block groups that can temporarily store input images along the Y axis. For example, "Figure 4" shows the drive The first cache block group CBG_1 of the line buffers (LB_1~LB_4), "Figure 5" shows the second cache block group CBG_2 of the drive line buffers (LB_1~LB_4), "Figure 6" The figure shown is to drive the line buffers (LB_1~LB_5) in sequence to the last cache block group, but the figure shown is only an example, and the size of each cache block group is not limited as shown in the figure (7) To continue, so far this embodiment only drives no more than 5 line buffers (LB_1~LB_5) to temporarily store the input image. Compared with the prior art, it needs at least 7 lines according to the direction of each row of scanning lines. A line buffer stores the pixel data of the data line L_1 and the data line L_2 (even other pixels not on the data line) and their input image addresses one by one. The present invention can at least reduce the use of line buffers and reduce pixel duplication. (8) Cache hit check (step S30): The image corrector 20 searches for an input image address corresponding to the output image address to determine the location of one of the line buffers (LB_1~LB_N) The stored input image address (or the input image address that determines one of the cache block groups temporarily stored) is cache hit; (9) Step S30, if the cache hits, continue to execute the cache pixel And correct the image (step S40): the image corrector 20 can be configured to read the relevant pixels of the input image from one of the cache block groups of one of the line buffers (LB_1~LB_N) to continue the distortion correction algorithm (10) Continuing, if the result of step S30 is a cache error, continue reading the main memory and updating the line buffer (step S50): the image corrector 20 can be configured to correspond to the main memory 10 The input image address of reads the relevant pixels of the input image, and the input image address of the input image captured by the main memory 10 is synchronously updated to the line buffers (LB_1~LB_N) to facilitate subsequent cache pixel data . (11) In addition, in a preferred embodiment of the present invention, the aforementioned deformation correction algorithm can be a bilinear interpolation or a nearest-nearest interpolation interpolation algorithm (interpolation), but this is not the case. Is limited.

Please refer to "Figure 7", which is another preferred embodiment of the present invention. This embodiment proposes an image correction system 1 with a line buffer. Its technology is mainly similar to that of "Figure 2" to "Figure 6". The main difference is that the image correction system of this embodiment further includes separate couplings. A LUT (Look Up Table) memory 40 connected to the main memory 10 and the image corrector 20 can store the bit address conversion relationship between the input image and the output image for the image corrector 20 to find The input image address corresponding to the output image address. Preferably, the LUT memory 40 of this embodiment can also be a static random access memory (SRAM), but it is not limited to this.

The above are only the preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Anyone who is familiar with this technique can make equal changes and modifications without departing from the spirit and scope of the present invention. Should be covered within the scope of the patent of the present invention.

In summary, the present invention has patent requirements such as "industrial applicability", "novelty", and "advancedness"; the applicant filed an application for a patent for invention with the Bureau of Patent in accordance with the provisions of the Patent Law.

1: Image correction system with line buffer 10: Main memory 20: Image corrector 30: Cache memory 40: LUT memory LB_1: Line buffer LB_2: Line buffer CB_22: Cache block CB_23: Cache block LB_3: Line buffer CB_31: Cache block CB_32: Cache block LB_4: Line buffer CB_41: Cache block CBG_1: The first cache block group CBG_2: The second cache block group LB_5: Line buffer LB_N: Line buffer L_1: The first data line of the input image L_2: The second data line of the input image S: Implementation method of image correction system with line buffer S10: Set cache memory S20: Generate output image address S30: Cache hit check S40: Cache pixels and correct images S50: Read main memory and update line buffer LB_1’: Conventional Line Buffer LB_2’: Line buffer LB_3’: Line buffer L_1’: Learn the data line of the input image L_2’: Learn the data line of the input image

Figure 1 is a schematic diagram of a conventional line buffer reading input images. Figure 2 is a system architecture diagram of the present invention. Figure 3 is a system flow chart of the present invention. Figure 4 is a schematic diagram (1) of the implementation of the present invention. Figure 5 is a schematic diagram (2) of the implementation of the present invention. Figure 6 is a schematic diagram of the implementation of the present invention (3). Figure 7 is another preferred embodiment of the present invention.

1: Image correction system with line buffer

10: Main memory

20: Image corrector

30: Cache memory

LB_1: Line buffer

LB_2: Line buffer

LB_N: Line buffer

Claims (8)

An image correction system with a line buffer includes: a main memory for temporarily storing an input image; an image corrector for generating line data of an output image, and at least one line data of the output image It is obtained by executing a distortion correction algorithm on a plurality of line data of the input image; a cache memory is respectively coupled to the main memory and the image corrector, the cache memory includes N line buffers , Where the N line buffers are defined as containing multiple

A cache block group of a size, each cache block group is used to temporarily store a plurality of pixels of the input image recently captured by the main memory and an input image address of the plurality of pixels, and each of the cache blocks Take the input image address stored in the block group and configure it as the row number and address label of the input image on the same or different scan line; the image corrector is also used to generate one of the line data of the output image The output image address, and the output image address is defined as the row number and address label of one of the line data of the output image, and the input image address corresponding to the output image address is searched to determine which Whether the input image address stored in the line buffer is a cache hit; and if the cache hits, the image corrector reads the relevant pixels of the input image from one of the cache block groups of the line buffer To continue the deformation correction algorithm. For example, the image correction system with line buffer of the first item of the patent application, wherein, if the input image address stored in the line buffer is not cached, the image corrector will be from the main memory corresponding to the The input image address reads the relevant pixels of the input image, and causes the input image address of the input image recently captured by the main memory to be synchronously updated to the line buffer. For example, the image correction system with a line buffer of the first item of the patent application further includes a LUT memory respectively coupled to the main memory and the image corrector for storing the input image and the output image. Address conversion relationship for the image corrector to find the input image address corresponding to the output image address. For example, the image correction system with a line buffer in the first patent application, wherein the distortion correction algorithm is an interpolation algorithm using a bilinear interpolation or a nearest interpolation method. An implementation method of an image correction system with a line buffer includes: a step of setting a cache memory: a cache memory is configured to be respectively coupled to a main memory and an image corrector, the cache memory The body contains N line buffers, where the N line buffers are defined as containing multiple

Cache block group of the size, each cache block group temporarily stores a plurality of pixels of an input image captured by the main memory and an input image address of the plurality of pixels, and each cache area The input image address stored in the block group is configured as the row number and address label of the input image on the same or different scan lines; a step of generating an output image address: the image corrector generates one line data of an output image An output image address of the output image, at least one line data of the output image can be obtained by performing a distortion correction algorithm on a plurality of line data of the input image temporarily stored in the main memory, and the output image address is defined Is the row number and address label of one of the line data of the output image; a cache hit check step: the image corrector searches for an input image address corresponding to the output image address to determine one of the lines Whether the input image address stored in the buffer is a cache hit; and a step of fetching pixels and correcting the image: if the cache hits, the image corrector reads from one of the cache block groups in the line buffer The relevant pixels of the input image are continued with the distortion correction algorithm. For example, the method for implementing an image correction system with a line buffer in the scope of the patent application, wherein the result of the cache hit check step is a cache error, and a step of reading the main memory and updating the line buffer is successively performed: The image corrector reads the relevant pixels of the input image from the input image address corresponding to the main memory, and causes the input image address of the input image captured by the main memory to be updated to the line synchronously buffer. For example, the implementation method of an image correction system with a line buffer in the scope of the patent application, wherein, when the cache hit check step is executed, the image corrector is based on the input image and the output stored in a LUT memory The image address conversion relationship is used to find the input image address corresponding to the output image address. For example, the method for implementing an image correction system with a line buffer in the scope of the patent application, wherein the distortion correction algorithm is an interpolation algorithm applying a bilinear interpolation or a nearest interpolation method.

Images