KR100924710B1 - Method for allocating address and method for writing/reading image to/from memory - Google Patents

Abstract

The present invention includes the steps of reading pixels of an image line by line; Receiving pixels of each line of the read image; And providing sequential addresses to the pixels of the image in column units.

Stereo image, depth map, SAD, matching window, memory controller

Description

METHOD FOR ALLOCATING ADDRESS AND METHOD FOR WRITING / READING IMAGE TO / FROM MEMORY}

TECHNICAL FIELD The present invention relates to imaging device technology, and more particularly, to a method of reading and writing an image into a memory for processing a stereo image obtained using binocular vision to generate a depth-map.

Based on stereo vision, for example, still images captured by two cameras, processing the acquired stereo images, and creating depth-maps is a method of collecting three-dimensional information about the outside world in the field of computer vision. It is a widely used method. Generally . In order to apply the depth-map to a real application, it is necessary to generate in real time more than 30 frames per second.

In the following, the general theory of generating depth-map using binocular vision is described.

1A to 1D are conceptual views of depth-map acquisition using binocular vision.

As shown in FIG. 1A, when a subject is photographed using two cameras, as shown in FIG. 1C, the camera photographs a subject according to the disparity of the two camera positions as shown in FIG. 1B. Left image and right image are different.

When two different images are represented as one image, a depth-map corresponding to the disparity may be generated as shown in FIG. 1D.

Such a depth-map is generated corresponding to the disparity between stereo images obtained from two cameras lying on the horizontal line based on the following equation (1).

(1)

(One)

In Equation (1), SAD stands for Sum of Absolute Difference, and the meaning of each symbol is as follows.

x: The horizontal coordinate in the image obtained from the camera sensor.

y: The vertical coordinate in the image obtained from the camera sensor.

For example, in the case of an image of 640 * 480, the range of x is 0 to 639, and the range of y is 0 to 479.

d: Disparity Range. As shown in FIG. 1B, it refers to a range in which a matching window moves along the horizontal axis. For example, if d is 64, the range that the matching window can move is 64 pixels.

n: Length of one side of the matching window. For example, n equals 11 means a matching window that is 11 x 11 pixels in size.

i: Pixel position in the horizontal direction within the matching window.

j: Pixel position in the horizontal direction within the matching window. For example, when n is 11, i and j range from 0 to 10.

I ^left : Left image from the binocular camera.

I ^right : Right image from the binocular camera. For example, I ^right (120, 110) means a pixel value located at a position shifted by 120 pixels in the right horizontal direction and 110 pixels in the bottom vertical direction with the left uppermost point as the origin in the right image. In general, a pixel value uses the brightness value of that pixel.

Equation (1) means that as shown in Figure 1, the difference in the position of the object appearing in the two images obtained by the binocular camera using the window-based difference in the horizontal direction, the area to compare the difference is as much as d This means performing a comparison on the entire image area.

The SAD value obtained at each pixel position is used as an index indicating the relative position of objects appearing in the image with respect to the camera. That is, the value of the position where the SAD value is the smallest within the disperity range d becomes a depth value representing the distance of the object. When the value is obtained for the entire image, a depth map is completed.

The following explains the window-based difference and matching window. In the binocular camera, the image obtained from the right side and the image obtained from the left side are different from each other due to the viewing point, and thus the position of the object appearing in the image is different. This difference is then used to find the difference between the right and left images using a window of a certain size. Here, a window of a specific size used to find a difference is called a matching window, and an image difference between the left and right sides obtained by using a matching window is called a window-based difference.

In addition, a difference generally refers to an absolute difference between brightness values of two pixels, and is a sum of absolute difference between all pixels in a window in a right and left image (SAD: Sum of Absolute Difference). Determine the similarity between the images and estimate the difference in object position.

Briefly summarizing the above for the purpose of understanding, the process of generating the depth-map sets one of the two images as the reference image, and the image patch of the reference image (for example, the left image) is the target image. This is achieved by moving in a specific interval (eg, Max Disparity Range) of the right image (eg, the right image) and obtaining the SAD value and the lowest value among them.

The process of generating such a depth-map is mainly performed through a computing device such as a PC. To this end, the image must first be input into a memory. When the image is input into the memory, a read operation is performed on pixels of each image from the memory, and the SAD operation is performed through the read pixels.

At this time, for example, when the SAD is performed in the R section based on the matching window of NxN size, (R + 1) N ^two memory read operations must be performed from both images. do. That is, the N ² memory read operation should be performed on the reference image (eg, the left image), and the RN ² memory read operation should be performed on the target image (eg, the right image).

Finally, for stereo matching of one image frame (eg, H (horizontal) x V (vertical) size) using a matching window of NxN size, HxV (R + 1) N ² times memory read operation should be performed.

However, performing such very frequent memory read operations, i.e., high memory references, makes it difficult to implement a real-time stereo matching system.

In addition, the process of reading the pixel of the image from the memory is read in the order of the horizontal direction of the image, while the matching window is made by two-dimensional NxN, that is, the horizontal direction and the vertical direction must be referred to at the same time.

In addition, since the process of inputting the image into the memory and the process of reading the pixel of the image from the memory occur at about the same time, the memory control structure becomes very complicated in the conventional method.

Such problems have a bad effect on various aspects such as heat generation, power consumption surge, operating temperature range limitation, manufacturing unit price increase, product yield drop.

Accordingly, an object of the present invention is to solve the above problems, and to reduce the memory reference amount, that is, the memory read operation to a minimum, and to simplify the memory control structure.

In addition, another object of the present invention is to reduce the memory reference amount as described above, and to simplify the memory control structure, it is possible to easily develop a real-time stereo matching device.

In order to achieve the above object, the present invention includes the steps of reading the pixels of the image line by line; Receiving pixels of each line of the read image; And providing sequential addresses to the pixels of the image in column units.

In addition, to achieve the above object, the present invention comprises the steps of receiving the pixels of each line of the image; Assigning sequential addresses to the pixels of the image in columns; And storing the input pixels in a vertical direction according to the address.

In addition, in order to achieve the above object, the present invention provides a method comprising the steps of: assigning sequential addresses in columns for pixels of an image; Storing pixels of a first line of the image according to the sequential addresses; And storing the first pixel of the second line while reading the first pixel of the first line by using the first of the addresses.

In addition, in order to achieve the above object, the present invention assigns sequential addresses to the pixels of the image in column units, and then assigns pixels of the first line of the image to the memory according to the sequential addresses. A memory controller performs a write operation and performs a read operation on the first pixel of the first line by using a first address among the addresses, and performs a write operation on the first pixel of the second line. .

In addition, in order to achieve the above object, the present invention provides a memory including the memory controller.

The present invention reduces the memory reference amount during the stereo matching operation to a minimum, and the memory control structure also enables the matching operation in real time at a low cost by reducing several operations into one operation.

In addition, the present invention allows the stereo matching system to be manufactured more stably and at a low cost, and has an effect of reducing power consumption.

This study was conducted as part of the IT new growth engine core technology development project of the Ministry of Information and Communication and the Ministry of Information and Communication Research and Development. [2005-S-111-02, Intelligent Robot Sensor]

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a diagram illustrating a procedure of storing an image according to a general scheme into a memory, FIG. 3 is a diagram illustrating a memory storage scheme according to a general scheme, and FIG. 4 is a diagram illustrating a SAD processing method of stereo matching.

As can be seen with reference to Figure 2, the image is made of pixels, in order to store the image into the memory, the process of reading the image by the sensor in a horizontal order (1). That is, reading is performed in units of lines. For example, in order to store the left image of the two images into the memory, the process of reading the image by the image sensor reads the pixels constituting the horizontal first line (Line) in the image frame, and then the horizontal direction The pixels constituting the second line of the circuit are read in order.

As can be seen with reference to Figure 3, in order to store the image read by the sensor in the memory should be given an address, in this case, an address is given to each line (Line) in the horizontal direction. That is, addresses are given one by one in a line unit. For example, the lowest address, for example, A0, is assigned to the first line of the image, and the second lowest address, for example, A1, is assigned to the second line.

According to the addresses given in this order, the pixels of each line of the image are stored horizontally in the memory. That is, as shown in FIG. 3, after the first line having the A0 address is stored, the second line having the A1 address is stored. The pixels in each line are similarly stored in the horizontal direction. For example, in the first line having the A0 address, the first pixel P0, the second pixel P1, and the third pixel P2 are in order. It is stored in the horizontal direction.

However, as can be seen with reference to FIG. 4, the process of performing the SAD is performed through a matching window 2 having an N × N size, so that not only a series of horizontal pixels but also a series of vertical directions in the image. Pixels are needed at the same time. That is, in order to perform the SAD, a series of vertical pixels 5 are required, but the memory storage method shown in FIG. 3 cannot read the series of vertical pixels at once. This is because a plurality of addresses are required to read the pixels in the vertical direction. The read operation of the memory is performed by providing one address, and outputting the next address.

5A and 5B illustrate a concept of a memory storage method according to the present invention.

As can be seen with reference to Figure 5a, the memory storage method according to the present invention gives an address in the column unit of the vertical direction of the image. That is, the present invention does not give an address to one line in the horizontal direction, but gives an address in units of vertical columns.

In other words, address A0 is assigned to the first column of the image in the vertical direction, and A1 address is assigned to the second column of the image.

At this time, if the image is composed of, for example, 320x240 pixels, as shown in FIG. 5B, an address A0 is assigned to the pixels P0, P320, P640, ... constituting the first row, Address 1 is assigned to the pixels P1, P321, P641, ... that make up the second row.

However, as shown in FIG. 2, the sensor reads the image in a horizontal direction (1), that is, a series of pixels P0, P1, P2, ... in a line unit, and outputs the image to the memory. Since the address is assigned to a series of pixels P0, P320, P640, ... in the vertical direction, that is, in a column unit, the storage method of the memory needs to be changed. That is, it is necessary to change the memory control scheme in order to give an address in the column direction in the vertical direction and to store it in the memory. This will be described with reference to FIG. 6.

6A to 6D are exemplary views illustrating a memory storing method according to the present invention.

As can be seen with reference to Figures 6a to 6d, the memory storage method according to the present invention stores the image of the line unit in the horizontal direction in the vertical direction according to the address.

For example, as in FIG. 6A, the pixel P0 (address A0), pixel P1 (address A1), pixel P2 (address A2), pixel P3 (address A3), and pixel P4 (address A4) of the first column When input to the memory, the memory stores the pixels in the vertical direction as described above according to the addresses A0, A1, A2, A3, and A4. In other words,

Then, as shown in Fig. 6B, the pixels P320 (address A0), pixel P321 (address A1), pixel P322 (address A2), pixel P323 (address A3), ... of the second column are input into the memory. If the address of the pixel P320 is A0, the memory is stored next to the pixel P0. Since the address of the pixel P321 is A1, the memory is stored vertically.

When the pixels P640 (address A0), pixel P641 (address A1), pixel P642 (address A2),... Of the third column are input as shown in FIG. 6C, the memory stores the pixels vertically. .

The memory in which the pixels are stored vertically is as shown in FIG. 6D.

7 is an exemplary view illustrating a memory read / write method according to the present invention.

As can be seen with reference to Fig. 7, the memory according to the invention can, for example, write one pixel at a time, but can read N pixels at a time. For example, assuming that an image consists of 320x240 pixels and each pixel consists of 8 bits, only one pixel of 8 bits can be written at a time, but the read can read as many as N pixels of the matching window. Alternatively, the column of pixels, that is, 240 pixels, may be read at a time. Therefore, the memory according to the present invention is very useful when the memory write operation is not frequent, while the memory read operation is frequent.

Therefore, according to the present invention, since the vertical size of the matching window is read from the memory at one time, the memory reference amount is changed from HW (R + 1) N ^ 2 to HW (R + 1) N * 2. Decreases.

On the other hand, as can be seen with reference to Figure 7, the memory control structure according to the present invention has a Read-Shift-Write structure for storing the associated data aligned. In general, in the memory control scheme, read / write are performed in separate operations, whereas in the present invention, read / write is performed in one operation. That is, in a general memory control structure, an address / control / data signal should be given every time pixel data is referenced, but in the present invention, if one address is provided, simultaneously Read-Shift- Write is executed at the same time. In other words, when the pixel P0, the pixel P2, the pixel P3, the pixel P4, the pixel P5 and the pixel P6 of the first line of the image are stored in the memory as shown in FIG. 6B, if the address A0 is provided, the pixel P0 can be read. In addition, at the same time, the pixel P320 of the second line can also perform a write operation. Similarly, while reading the first pixel P1 of the first line of the image from the memory, it is possible to write the second pixel P321 of the second line to the memory at the same time.

As a result, when the pixels of the image are addressed in a column unit in the vertical direction and stored in the memory, the SAD is provided through a matching window 2 having an NxN size shown in FIG. 4. Even though, the series of pixels in the vertical direction can be read from the memory at one time, the memory read operation is significantly reduced from HxV (R + 1) N ² to HxV (R + 1) N * 2. . For example, when the matching window has a size of 11 × 11, the memory read operation is reduced by 84.62% compared to the prior art, which is only 15.38%.

The method according to the invention described thus far can be implemented in software, hardware, or a combination thereof. For example, the method according to the present invention may be stored in a storage medium (eg, memory, flash memory, hard disk, etc.) and may be a software program executable by a processor (eg a microprocessor). It may be implemented with codes or instructions within.

In the above description of the preferred embodiments of the present invention by way of example, the scope of the present invention is not limited only to these specific embodiments, the present invention is in various forms within the scope of the spirit and claims of the present invention Can be modified, changed, or improved.

1A to 1D are conceptual diagrams of depth-map acquisition using binocular vision.

2 is a diagram illustrating a procedure of storing an image in a memory according to a general scheme.

3 is a diagram illustrating a memory storage method according to a general method.

4 is a diagram illustrating a SAD processing method of stereo matching.

5A and 5B are exemplary views illustrating the concept of a memory storage method according to the present invention.

6A to 6D are exemplary views illustrating a memory storing method according to the present invention.

7 is an exemplary view illustrating a memory read / write method according to the present invention.

Claims (6)

delete Receiving pixels of each line of the image; Assigning sequential addresses to the pixels of the image in columns; And storing the input pixels in a vertical direction according to the address. The method of claim 2, And reading the stored pixel through any one of the sequentially assigned addresses, and storing neighboring pixels in the line direction with the pixel. Assigning sequential addresses on a column-by-column basis to pixels of an image; Storing pixels of a first line of the image according to the sequential addresses; And storing the first pixel of the second line while reading the first pixel of the first line using a first one of the addresses. After assigning sequential addresses to columns of the image in column units, writes pixels of the first line of the image into memory according to the sequential addresses, and writes a first address among the addresses. And performing a read operation on the first pixel of the first line while performing a write operation on the first pixel of the second line. After assigning sequential addresses to columns of the image in column units, writes pixels of the first line of the image into memory according to the sequential addresses, and writes a first address among the addresses. And a memory controller configured to perform a write operation on the first pixel of the first line while performing a read operation on the first pixel of the first line.

Images