CN108550102B

CN108550102B - Hardware accelerator

Info

Publication number: CN108550102B
Application number: CN201810380604.1A
Authority: CN
Inventors: 刘劲松
Original assignee: Allwinner Technology Co Ltd
Current assignee: Allwinner Technology Co Ltd
Priority date: 2018-04-25
Filing date: 2018-04-25
Publication date: 2022-05-17
Anticipated expiration: 2038-04-25
Also published as: CN108550102A

Abstract

The invention discloses a hardware accelerator which is used for improving the running efficiency of a CPU. The hardware accelerator comprises a storage unit, a logic bank group, an information bin, a bilateral operation unit, a comparison unit and a control unit. On the basis of the prior art, the hardware accelerator protected by the invention replaces a software algorithm to give full play to the inherent processing speed advantage of hardware, a logic bank group consisting of block logic banks is arranged, rich cache space selection can be provided, the waste of cache resources is avoided, point-based operation in the traditional image processing process is improved into line-based operation by arranging an information bin and a bilateral operation unit, and the result of a subsequent position is iteratively calculated by using the calculation result of a previous position, so that the number of elements required in the hardware accelerator is greatly reduced, and the lightweight hardware structure is realized.

Description

Hardware accelerator

Technical Field

The present application relates to the field of circuits, and more particularly to a hardware accelerator.

Background

Computer vision, especially computer stereovision, has received much attention in recent years, and there is an increasing demand for obtaining target depth information. For example, the intelligent security detection object's location, unmanned aerial vehicle three-dimensional location, automatic driving barrier judge etc. robot navigation, 3d print etc..

As is known, in order to obtain depth information, a large amount of image data processing and calculation are performed, and an algorithm for processing image data is often complex in calculation and large in calculation amount. However, merely optimizing software algorithms does not improve algorithm execution efficiency well and does not keep up with the amount of data brought about by computer vision development. Therefore, more and more research is focused on the design of hardware accelerators. The existing hardware accelerator lacks good portability, cannot optimize a cache structure, occupies and wastes a large amount of system cache resources, and simultaneously needs a large amount of operation resources, so that the problems of high power consumption, low battery life and the like of a mobile device comprising the hardware accelerator are caused.

Disclosure of Invention

In view of this, embodiments of the present invention provide a hardware accelerator to solve the problems in the prior art that the hardware accelerator has poor portability, occupies and wastes a large amount of system caches, and has more hardware accelerator components, large power consumption, and short battery life.

The invention provides a hardware accelerator, which comprises an acquisition unit, a logic bank group, a comparison unit, a control unit, an information bin and an arithmetic unit, wherein the acquisition unit is used for acquiring a logic bank group;

the acquisition unit is connected with the logic bank group, the logic bank group is connected with the arithmetic unit, the arithmetic unit is connected with the information bin, the information bin is connected with the comparison unit, and the control unit is connected with the acquisition unit, the logic bank group, the comparison unit, the information bin and the arithmetic unit;

the acquisition unit is used for acquiring image data to be operated;

the logical bank group is composed of a plurality of rams and is used for storing the image data; the logical bank group is composed of a plurality of rams and is used for storing the image data; the logic bank group comprises a plurality of logic bank units, each logic bank unit is composed of a plurality of rams, the plurality of rams are divided and combined to form a plurality of logic banks, the plurality of rams are divided into a plurality of rams, the plurality of logic banks are composed of a plurality of rams, at least two rams in any one logic bank do not belong to the same ram, each logic bank unit is composed of a plurality of logic banks, and the plurality of logic banks can be combined according to a preset mode;

for example, the logical bank group may be composed of N (N × width) windows bit rams, where the windows are adapted to the system bit width, the N windows are numbered and arranged sequentially from left to right, each window is divided into N parts in a top-down sequence by taking the window as a unit, the logical bank group includes N parts of srams in total, consecutive m rams serve as a logical bank unit, m is greater than 1 and not greater than N, the logical bank group includes N/m (whole down) logical bank units in total, each logical bank unit is composed of a plurality of logical banks, each logical bank includes at least two rams in its corresponding logical bank unit, at least two rams in each logical bank do not belong to the same ram, and thus the setting manner that at least two rams do not belong to the same ram can maximally improve the efficiency of data reading and writing, but do not wait for reading and writing data while two rams belong to the same ram, and the robustness of the system can be improved, each block of logic bank in the logic bank unit can be combined, for example, two-to-two combination, three-to-three combination or four-to-four combination, so that logic banks with selectable memory capacity can be formed in the logic bank unit, and each logic bank unit can be freely combined, thereby providing rich storage space selection.

The operation unit is configured to perform operation on data in the logical bank group and output an operation result, where the operation includes an initialization operation, and the initialization operation specifically includes the following steps: determining an initialization position of a left side view and a searching range corresponding to a right side view, and when the center of a searching block is positioned at the initialization position, calculating the sum of absolute values of gray value differences of pixel points covered by each line of the searching block and pixel points of a corresponding line in the searching range of the right side view; the center of the search block horizontally moves by taking the initialization position as a starting point, and the search range correspondingly horizontally moves according to the movement of the center of the search block;

the information bin is used for storing the operation result output by the operation unit, the operation executed by the operation unit also comprises bilateral operation, and the bilateral operation calculates the sum of the absolute value of the gray value difference between the pixel points covered by each line of the search block after the center of the search block moves horizontally and the pixel points of the corresponding line in the right side view search range according to the operation result stored by the information bin in an iteration mode;

the control unit is used for sending a control signal to periodically control the operation unit to perform image data operation;

and the comparison unit reads the operation result stored in the information bin to obtain a comparison result.

Preferably, the accelerator further comprises a preprocessing unit that SOBEL filters the image.

Preferably, the comparison result is a feature matching point of the image;

preferably, the hardware accelerator further comprises a denoising unit for verifying the matching points to remove noise points.

Preferably, the image data is left and right view data.

Preferably, the left view and the right view are from a binocular vision system, the left view is obtained by shooting by a left camera in the binocular vision system, and the right view is obtained by shooting by a right camera in the binocular vision system.

Preferably, the left and right views are derived from a monocular vision system, and the left and right views are respectively captured by the monocular vision system at different positions.

Preferably, the control unit is further configured to set a search block in each of the left view and the right view, where the search block is a left view search block and a right view search block, and the left view search block and the right view search block slide horizontally in the corresponding view and cover a pixel point of the size of the search block.

Preferably, the left view search block and the right view search block are each a winsize square search block, the winsize is odd, and 5. ltoreq. winsize.ltoreq.21.

Preferably, the logical bank group is divided into N logical banks.

Preferably, each logical bank is configured to store the gray value of the pixel covered by the sliding of the search block in the view.

Preferably, the arithmetic unit includes a first arithmetic subunit composed of a subtractor and an adder, the first arithmetic subunit is configured to calculate an HSAD, the HSAD is a sum of absolute values of differences in gray values of corresponding pixels in the left view search block and the right view search block, and the first arithmetic subunit is connected to the logical bank group and the information bin and sends the HSAD to the information bin.

Preferably, when the center of the left side view search block slides to the a-th pixel row of the left side view interesting region for the first time, the control unit sends a control signal to the arithmetic unit, and the first arithmetic sub-unit performs an initialization operation to determine the HSAD of the current position n, where the current position n is the position of one pixel point in the a-th pixel row in the left side view interesting region, and a e [1, the total number of pixel rows in the interesting region ].

Preferably, the first arithmetic subunit calculates HSAD in the following way:

wherein, HSAD_nmjThe HSAD of the j row which represents the position m of the center of the corresponding right view search block when the center of the left view search block is positioned at the current position n;

l_sobel_irepresenting the gray value of the ith pixel point of the jth line in the left side view search block when the center of the left side view search block is at the position n in the preprocessed left side view l _ sobel;

r_sobel_irepresenting the gray value of the ith pixel point in the jth line in the right side view search block when the center of the right side view search block is at the position m in the preprocessed right side view r _ sobel;

wherein i, j is 0 … winze-1, M is more than or equal to 1 and less than or equal to M, and M is the preset maximum parallax.

Preferably, the arithmetic unit further includes:

the second operation subunit comprises a plurality of subtractors and is connected with the logic bank group;

the third operation subunit comprises a plurality of subtractors and is connected with the information bin, the first operation subunit and the second operation subunit;

the fourth operation subunit comprises a plurality of subtractors and is connected with the logic bank group;

a fifth arithmetic subunit including a plurality of adders, the fifth arithmetic subunit being connected to the third arithmetic subunit and the second arithmetic subunit.

Preferably, after the first operation subunit completes the initialization operation, the center of the left view search block slides in the horizontal direction until all pixel points in the row where the current position n is located in the left view region of interest are traversed, and the operation unit performs bilateral operation to obtain the HSAD of each pixel point and stores the HSAD in the information bin.

Preferably, the bilateral operation determines the HSAD of the next position n +1 based on the HSAD of the current position n, the next position n +1 being located within the left side view region of interest.

Preferably, the next position n +1 is a position where a pixel point is horizontally slid rightwards from the current position n in the center of the left view search block;

the second operationSub-usage for cell computation AD_lThe AD_lWhen the center of the left view searching block is at the position n, the absolute value of the gray value difference between the leftmost pixel point of the jth line of the right view searching block and the leftmost pixel point of the jth line of the left view searching block is obtained;

the third operation subunit is used for calculating the HSAD_nmjAnd AD corresponding thereto_lA difference of (d);

the fourth operation subunit is used for calculating AD_rThe AD_rWhen the center of the left side view searching block is at the position of n +1, the absolute value of the gray value difference between the rightmost pixel point of the jth line of the right side view searching block and the rightmost pixel point of the jth line of the left side view searching block;

the fifth operation subunit is used for calculating HSAD_n+1mjWherein HSAD_n+1mjFor the sum of the third arithmetic subunit calculation result and the fourth arithmetic subunit calculation result, i.e. HSAD_n+1mj＝HSAD_nmj-AD_l+AD_rWherein, HSAD_n+1mjIndicating the HSAD of the j-th line when the center of the left view search block is horizontally shifted to the right by one pixel when the corresponding right view search block center is at position m.

Preferably, the next position n +1 is a position where the center of the left view search block horizontally slides one pixel point leftwards from the current position n;

the second operation subunit is used for calculating AD_rThe AD_rWhen the center of the left side view searching block is at the position of n +1, the absolute value of the gray value difference between the rightmost pixel point of the jth line of the right side view searching block and the rightmost pixel point of the jth line of the left side view searching block;

the third operation subunit is used for calculating the HSAD_nmjAnd AD corresponding thereto_rA difference of (d);

the fourth operation subunit is used for calculating AD_lThe AD_lWhen the center of the left view searching block is at the position n, the absolute value of the gray value difference between the leftmost pixel point of the jth line of the right view searching block and the leftmost pixel point of the jth line of the left view searching block is obtained;

the fifth operation subunit is used for countingCalculation of HSAD_n+1mjWherein HSAD_n+1mjFor the sum of the calculation results of the third and fourth arithmetic subunits, i.e. HSAD_n+1mj＝HSAD_nmj-AD_r+AD_lWherein, HSAD_n+1mjIndicating the HSAD of the j-th row when the center of the left view search block is horizontally shifted to the left by one pixel, with the corresponding center of the right view search block at position m.

Preferably, when the center of the left-side-view search block slides from the current position n to the next position n +1, the search range of the right-side view is shifted to the right by one pixel, and the right-side-view search block horizontally slides from the right to the left by M positions within the search range.

Preferably, when the center of the left-side-view search block slides from the current position n to the next position n +1, the search range of the right-side view is shifted to the left by one pixel, and the right-side-view search block horizontally slides from the right to the left by M positions within the search range.

Preferably, the arithmetic unit further comprises a sixth arithmetic sub-unit connected to the bins, the sixth arithmetic sub-unit comprising an adder for calculating SAD.

Preferably, the sixth computing subunit calculates SAD in the following way:

wherein, HSAD_jWhen the center of the left side view search block is positioned at a certain position in the left side view interesting area, the HSAD of the jth row when the center of the right side view search block is positioned at a certain position in the right side view, j is 0 … winze-1, and M is more than or equal to 1 and less than or equal to M;

and when the center of the left side view searching block slides one pixel point in the left side view interesting region, the operation unit correspondingly calculates to obtain M SADs.

Preferably, the bins are used to store the HSAD and SAD, and the left view search block center position and the right view search block center position.

Preferably, when the center of the left side view search block traverses all pixel points in the left side view region of interest, the control unit sends a control signal to the operation unit to stop operation.

Preferably, when the center of the left side view search block slides from the current position n to the next position n +1 in the left side view interesting region, the control unit sends a control signal to the comparison unit to determine the center position of the right side view search block corresponding to the minimum value of the SAD corresponding to the current position n, and determine the pixel point corresponding to the center position of the right side view search block as the matching point.

On the other hand, the invention provides a binocular vision system parallax information extraction hardware accelerator which is characterized by comprising a matching point acquisition unit, a camera coordinate system generation unit and a parallax determination unit;

the matching point acquisition unit acquires a matching point obtained by using the hardware accelerator;

the camera coordinate system generation unit acquires internal and external parameters of a binocular vision system and establishes a camera coordinate system;

the parallax determining unit determines the parallax of the matching point according to the camera coordinate system and the matching point.

On the other hand, the invention provides a hardware accelerator for extracting parallax information of a monocular vision system, which is characterized by comprising a matching point acquisition unit, a camera coordinate system generation unit and a parallax determination unit;

the camera coordinate system generating unit acquires internal and external parameters of the monocular vision system and establishes a camera coordinate system;

A depth information extraction hardware accelerator using the hardware accelerator, wherein the depth information extraction hardware accelerator comprises:

the information acquisition unit is used for acquiring the parallax information obtained by the hardware accelerator according to the parallax information;

a depth information determination unit for determining depth information based on the disparity information.

The invention discloses a hardware accelerator, which has the following beneficial effects: the hardware accelerator disclosed by the invention can provide complete data storage and operation functions, fully exerts the speed advantage of a hardware circuit in addition to a software optimization algorithm, can be coupled in a general computer system or an embedded system or other processing systems, assists a CPU to execute complex operation, effectively improves the running efficiency of the CPU, and has higher portability; according to the hardware accelerator disclosed by the invention, the logic bank groups are set and effectively divided, so that on one hand, a rich cache structure is provided, and on the other hand, the shortage or excessive waste of cache resources is avoided; according to the hardware accelerator disclosed by the invention, the point-based operation in the traditional image processing process is improved into the line-based operation by arranging the information bin and the bilateral operation unit, and the result of the subsequent position is iteratively calculated by using the calculation result of the previous position, so that the number of elements required in the hardware accelerator is greatly reduced, and the lightweight hardware structure is realized.

Drawings

Fig. 1 is a block diagram of a hardware accelerator according to an embodiment of the present invention.

Fig. 2 is a schematic diagram of a logical bank group according to an embodiment of the present invention.

Fig. 3 is a structural diagram of an arithmetic unit according to an embodiment of the present invention.

Fig. 4 is a schematic diagram of a search block generated by a control unit according to a second embodiment of the present invention.

FIG. 5 is a schematic diagram of a first operation subunit according to a third embodiment of the present invention.

Fig. 6 is a schematic diagram of operations of the second to fifth operation subunits according to the fourth embodiment of the present invention.

Fig. 7 is a circuit diagram of a bin according to a fifth embodiment of the present invention.

Fig. 8 is a block diagram of a hardware accelerator according to a seventh embodiment of the present invention.

Fig. 9 is a block diagram of a hardware accelerator according to an eighth embodiment of the present invention.

Fig. 10 is a block diagram of a hardware accelerator according to a ninth embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in further detail below with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention.

It should be further noted that, for the convenience of description, only some but not all of the relevant aspects of the present invention are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.

Example one

Fig. 1 is a block diagram of a hardware accelerator according to an embodiment of the present invention. The hardware accelerator of this embodiment specifically includes:

an acquisition unit (110) acquires image data including a left side view and a right side view.

In this embodiment, the left side view and the right side view specifically refer to two images including the same target object obtained from two different positions, and specifically, the left side view and the right side view may be obtained by two same or different cameras at different positions, or the left side view and the right side view may be obtained by moving the same camera, which is not limited in this embodiment.

The target object specifically refers to an object included in the region of interest of the left side view. It can be understood that, in general, when two images are matched, the image area of the important object in one image is mainly matched with the whole or part of the image area of the other image to determine the relevance of the two images, therefore, in this embodiment, the left side view and the right side view should include the same object to complete the feature point matching of the left side view and the right side view.

A logical bank group (120) composed of a plurality of rams for storing the image data; the logical bank group is composed of a plurality of rams and is used for storing the image data; the logic bank group comprises a plurality of logic bank units, each logic bank unit is composed of a plurality of rams, the plurality of rams are divided and combined to form a plurality of logic banks, the plurality of rams are divided into a plurality of rams, the plurality of logic banks are composed of a plurality of rams, at least two rams in any one logic bank do not belong to the same ram, each logic bank unit is composed of a plurality of logic banks, and the plurality of logic banks can be combined according to a preset mode;

for example, the logical bank group may be composed of N (N) windows bit rams, where the windows are adapted to the system bit width, the N rams are numbered and arranged sequentially from left to right, each ram is divided into N copies from top to bottom in order by taking the windows as a unit, the logical bank group includes N windows, consecutive m rams are used as a logical bank unit, m is greater than 1 and not greater than N, the logical bank group includes N/m (rounded down) logical bank units, each logical bank unit is composed of several logical banks, each logical bank includes at least two rams in its corresponding logical bank unit, at least two rams in each logical bank do not belong to the same ram, and thus the arrangement of at least two rams do not belong to the same ram, and the efficiency of data reading can be maximally improved without waiting for reading and writing data while the two rams belong to the same ram, and the robustness of the system can be improved, each block of logic bank in the logic bank unit can be combined, for example, two-to-two combination, three-to-three combination or four-to-four combination, so that logic banks with selectable memory capacity can be formed in the logic bank unit, and each logic bank unit can be freely combined, thereby providing rich storage space selection. The arithmetic unit is used for carrying out arithmetic on the data in the logic bank group and outputting an arithmetic result. In this embodiment, as shown in fig. 2, assuming that the logical bank group is composed of 12 (4 × 21) × 64bit srams, each sram is divided according to a principle of dividing each sram into 21 rows, each sram may be divided into 4, for example, the logical bank group in the figure includes 48 srams, the consecutive 4 srams serve as a logical bank unit, there are 3 logical bank units in the figure, each logical bank unit is composed of 4 logical banks, the logical bank unit on the left side in the figure includes 4 logical banks, respectively, such as 0, 3, 6 and 9, each logical bank includes 4 srams, for example, 4 srams, the bank0 includes 4 srams, each logical bank only has 4 srams of the logical banks, and at least two srams of each logical bank do not belong to the same sram, the logical banks in the same logical bank group may be combined with each other, for example, the logical bank unit may be combined with each other, such as 3558, or 3658, and the logical bank groups may be combined with each other such as 3, or 3 Three combinations or four combinations, thus forming a logic bank with larger storage capacity and having rich storage space options.

And the arithmetic unit (130), wherein the arithmetic unit (130) reads the data in the logic bank group (120), carries out arithmetic and outputs an arithmetic result.

In this embodiment, as shown in fig. 3, the operation unit (130) includes a first operation subunit (1301), a second operation subunit (1302), a third operation subunit (1303), a fourth operation subunit (1304), a fifth operation subunit (1305), and a sixth operation subunit (1306).

The first arithmetic sub-unit (1301) includes a plurality of subtractors and adders, the second arithmetic sub-unit (1302) includes a plurality of subtractors, the third arithmetic sub-unit (1303) includes a plurality of subtractors, the fourth arithmetic sub-unit (1304) includes a plurality of subtractors, the fifth arithmetic sub-unit (1305) includes a plurality of adders, and the sixth arithmetic sub-unit (1306) includes a plurality of adders.

And the information bin (140) is used for storing the operation result output by the operation unit.

And a comparison unit (150) for reading the operation result stored in the information bin (140) to obtain a comparison result.

And the control unit (160), wherein the control unit (160) sends control signals to each unit to control the operation of each unit.

The first embodiment of the present invention provides a hardware accelerator, which enables block logic banks in a logic bank group to be freely combined by adaptively improving the logic bank group, so as to form a rich scale of storage resources, and solve the problems in the prior art that a storage structure is fixed, storage resources are wasted, or storage resources are insufficient.

Example two

The control unit (160) is provided with a search block on each of the left side view and the right side view, namely a left side view search block (1601) and a right side view search block (1602), the search blocks slide horizontally in the view, and each time the center of the search block slides to a position, a pixel point with the size of the search block can be covered, the left side view search block (1601) and the right side view search block (1602) are square search blocks with the size of a winsize, the winsize is an odd number, and the winsize is not less than 5 and not more than 21.

As shown in fig. 4, the center of the search block (1601) is slid horizontally to the right at the a-th pixel row in the left side view region of interest, the center of the search block (1602) is slid horizontally to the left at the corresponding pixel row in the right side view search region, a e [1, total number of rows of pixels in the region of interest ].

Those skilled in the art will appreciate that the center of the search block (1601) may also be slid horizontally to the left on the a-th pixel row in the left side view region of interest, and the center of the search block (1602) may also be slid horizontally to the right on the corresponding pixel row in the right side view search region. It should be noted here that the centers of the search blocks (1602) related to all embodiments subsequent to this embodiment are all horizontally slid from right to left in the corresponding pixel rows in the right side view search area.

Those skilled in the art will appreciate that the sliding of the center of the search block (1601) within the left view region of interest may stop when the center of the search block (1601) slides to the boundary of the region of interest, or may continue to slide to traverse all the pixels of the a-th pixel row.

When the center of the search block (1601) slides to a certain position, the center of the search block (1602) slides within the corresponding search range of the right side view, and the slide traverses M positions. It can be understood by those skilled in the art that, in the existing image matching technology, a pixel point search range in the right side view, which matches with a single pixel point in the left side view, can be determined by a calculation method such as epipolar geometry, and the pixel point search range must include a pixel point that matches with the single pixel point in the left side view, and likewise, in this embodiment, a search range in the right side view, which corresponds to a position where a center of the search block (1601) is located, can be determined by a calculation method in the existing technology, where it should be noted that, each search range in the right side view, which corresponds to a position of the center of the search block (1601) in the region of interest in this embodiment and all the following embodiments, must be M consecutive pixel points in a certain pixel row in the right side view, that is, a search range corresponding to a position of the center of the search block (1601) should not include pixel points of different pixel rows, where M is a preset maximum disparity value.

Illustratively, the search block (1601) and the search block (1602) are both 5 × 5 square search blocks, gray values of all pixel points included in the left view search block are shown in table 1, a center of the right view search block is located at a rightmost pixel point in a corresponding search range in the right view, and gray values of all pixel points included in the right view search block are shown in table 2.

100	105	112	134	150
					90	98	110	122	140
95	99	98	130	142
					99	95	98	125	139
97	96	98	128	130

TABLE 1

50	65	66	70	65
					70	85	89	92	80
75	90	95	95	82
					78	92	95	90	88
75	90	96	93	92

TABLE 2

EXAMPLE III

Fig. 5 is a schematic diagram of a first operation subunit (1301) according to a third embodiment of the present invention.

As shown in fig. 5, the first operation subunit (1301) includes a plurality of subtractors and adders, the first operation unit (1301) is connected to the logical bank group (120) for calculating an HSAD corresponding to an initialization operation, and the first operation subunit (1301) is further connected to the information bin (140) for sending a calculation result to the information bin (140);

the following describes a calculation method of HSAD corresponding to initialization operation:

in this embodiment, when the center of the left view search block slides to the a-th pixel row of the left view interesting area for the first time, the control unit sends a control signal to the arithmetic unit, the arithmetic unit performs initialization operation, the initialization operation determines the HSAD of the current position n, and the current position n is the position of one pixel point in the a-th pixel row in the left view interesting area, and a e [1, the total number of rows of pixels in the interesting area ].

Those skilled in the art can determine that the total number of rows of pixels in the region of interest corresponds to the number of initialization operations, and also corresponds to the number of current positions n corresponding to each initialization operation.

The center of the left view search block is located at the current position n, the center of the right view search block horizontally slides M positions from right to left within a search range corresponding to the current position n in the right view, and the search range corresponding to the current position n in the right view determined by an existing calculation method is a certain whole pixel row.

Specifically, the HSAD value is calculated once every time the center of the right side view search block slides by one position, the number of the calculated HSAD values is the number of lines of the left side view search block (the number of lines of the left side view search block is the same as that of the right side view search block), and it can be seen that the number of the HSAD values corresponding to each current position n is the product of M and the number of lines of the left side view search block. The HSAD value of the current position n specifically refers to the sum of absolute values of differences between gray values of corresponding pixels in the left view search block and the right view search block calculated in units of rows of the left view search block and the right view search block when the center of the left view search block is located at the current position n and the center of the right view search block is located at a pixel in the right side view corresponding to the current position n.

Illustratively, the left view search block and the right view search block are both 5 × 5 square search blocks, the center of the left view search block is located at the current position n, the grayscale values of all pixel points included in the left view search block are shown in table 1, the center of the right view search block is located at the rightmost pixel point in the right side view within the search range corresponding to the current position n, and the grayscale values of all pixel points included in the right view search block are shown in table 2.

Then, the 5 HSAD values of the current position n at this time are:

HSAD1＝|100-50|+|105-65|+|112-66|+|134-70|+|150-65|＝285

HSAD2＝|90-70|+|98-85|+|110-89|+|122-92|+|140-80|＝144

HSAD3＝|95-75|+|99-90|+|98-95|+|130-95|+|142-82|＝127

HSAD4＝|99-78|+|95-92|+|98-95|+|125-90|+|139-88|＝113

HSAD5＝|97-75|+|96-90|+|98-96|+|128-93|+|130-92|＝103

further, after the center of the right side view search block slides one pixel point to the left from the rightmost pixel point in the search range corresponding to the current position n in the right side view, the gray value in table 2 changes, and at this time, it is necessary to continue to calculate another 5 HSAD values of the current position n until the center of the right side view search block slides to the leftmost pixel point in the search range corresponding to the current position n in the right side view and calculates 5 HSAD values of the current position n corresponding to this position.

More generally, the HSAD of the initialization operation specifically adopts the following manner:

wherein, HSAD_nmjA HSAD representing a jth row of a corresponding right view search block center at position m when the left view search block center is at said current position n;

r_sobel_iand representing the gray value of the ith pixel point in the jth line in the right side view search block when the center of the right side view search block is at the position m in the preprocessed right side view r _ sobel.

Illustratively, the left side view search block and the right side search block are both 5 × 5 square search blocks, the center of the left side view search block is located at the current position n, the numbers of all pixel points included in the left side view search block are shown in table 3, the center of the right side view search block is located at the mth pixel point from the right in the search range corresponding to the current position n in the right side view, and the numbers of all pixel points included in the right side view search block are shown in table 4.

A1	A2	A3	A4	A5
					B1	B2	B3	B4	B5
C1	C2	C3	C4	C5
					D1	D2	D3	D4	D5
E1	E2	E3	E4	E5

TABLE 3

A’1	A’2	A’3	A’4	A’5
					B’1	B’2	B’3	B’4	B’5
C’1	C’2	C’3	C’4	C’5
					D’1	D’2	D’3	D’4	D’5
E’1	E’2	E’3	E’4	E’5

TABLE 4

HSAD_nm1＝|A1-A`1|+|A2-A`2|+|A3-A`3|+|A4-A`4|+|A5-A`5|

......

HSAD_nm5＝|E1-E`1|+|E2-E`2|+|E3-E`3|+|E4-E`4|+|E5-E`5|

Example four

As shown, the arithmetic unit comprises a second arithmetic subunit (1302), the second arithmetic subunit (1302) comprises a plurality of subtractors, and the subtractors are connected with the logic bank group (120);

the arithmetic unit further comprises a third arithmetic sub-unit (1303), wherein the third arithmetic sub-unit (1303) comprises a plurality of subtracters, and the third arithmetic sub-unit (1303) is connected with the information bin (140) and the second arithmetic sub-unit (1302);

the arithmetic unit also comprises a fourth arithmetic subunit (1304), wherein the fourth arithmetic subunit (1304) comprises a plurality of subtractors and is connected with the logic bank group (120);

the arithmetic unit further comprises a fifth arithmetic subunit (1305), the fifth arithmetic subunit (1305) comprises a plurality of adders, the fifth arithmetic subunit (1305) is connected with the third arithmetic subunit (1303) and the second arithmetic subunit (1302) to perform bilateral arithmetic, and arithmetic results are stored in the information bin (140).

The HSAD for the next position n +1 is determined by bilateral operation, the next position n +1 being located in the left side view region of interest.

Optionally, the next position n +1 is a position where the center of the left-side view search block horizontally slides one pixel point to the right from the current position n. Illustratively, when the center of the left view search block is located at the next position n +1, the numbers of all pixel points included in the left view search block are shown in table 5, and when the center of the right view search block is located at the m-th pixel point from the right in the search range corresponding to the next position n +1 in the right view, the numbers of all pixel points included in the right view search block are shown in table 6,

TABLE 5

A’2	A’3	A’4	A’5	A’6
					B’2	B’3	B’4	B’5	B’6
C’2	C’3	C’4	C’5	C’6
					D’2	D’3	D’4	D’5	D’6
E’2	E’3	E’4	E’5	E’6

TABLE 6

Therefore, when the center of the right side view search block is located at the mth pixel point from the right in the search range corresponding to the next position N +1 in the right side view, the calculation formula of the HSAD value at the N +1 th position is:

HSAD_n+1m1＝HSAD_nm1-|A1-A`1|+|A6-A`6|

......

HSAD_n+1m5＝HSAD_nm5-|E1-E`1|+|E6-E`6|

generally, when the center of the left-view search block slides from the current position n to the next position n +1, the search range of the right-view search block correspondingly shifts one pixel to the right, and the right-view search block horizontally slides M positions from the right to the left within the search range.

Optionally, the next position n +1 is a position where the center of the left view search block horizontally slides one pixel point to the left from the current position n. Illustratively, when the center of the left view search block is located at the next position n +1, the numbers of all pixel points included in the left view search block are as shown in table 7, and when the center of the right view search block is located at the m-th pixel point from the right in the search range corresponding to the next position n +1 in the right view, the numbers of all pixel points included in the right view search block are as shown in table 8,

TABLE 7

A’0	A’1	A’2	A’3	A’4
					B’0	B’1	B’2	B’3	B’4
C’0	C’1	C’2	C’3	C’4
					D’0	D’1	D’2	D’3	D’4
E’0	E’1	E’2	E’3	E’4

TABLE 8

HSAD_n+1m1＝HSAD_nm1-|A5-A`5|+|A0-A`0|

......

HSAD_n+1m5＝HSAD_nm5-|E5-E`5|+|E0-E`0|

generally, when the center of the left-view search block slides from the current position n to the next position n +1, the search range of the right-view search block correspondingly shifts one pixel to the right, and the right-view search block horizontally slides M positions from the right to the left within the search range. Those skilled in the art can clearly understand that the two selectable modes can be selected or combined according to the specific situation of the starting position (denoted as the current position n) of the center of the left-side view search block, that is, when the starting position of the center of the left-side view search block is the left end point of the a-th pixel row of the left-side view interesting region, the next position n +1 is the position where the center of the left-side view search block slides horizontally to the right by one pixel point from the current position n; when the starting position of the center of the left view searching block is the right end point of the A-th pixel row of the left view interesting area, the next position n +1 is the position of the center of the left view searching block, which is horizontally slid by one pixel point from the current position n to the left; when the initial position of the center of the left view searching block is the non-end position of the A-th pixel row of the left view interesting region, the left view searching block slides to the right and slides to the left so as to traverse all the pixel points in the interesting region of the A-th pixel row.

In general, the second arithmetic subunit calculates AD_lThe AD_lRepresenting the absolute value of the gray value difference between the leftmost pixel point of the jth line of the right view searching block and the leftmost pixel point of the jth line of the left view searching block when the center of the left view searching block is at the position n;

the third operation subunit calculates the j row HSAD corresponding to the current position and the AD corresponding to the j row HSAD_lA difference of (d);

the fourth operation subunit calculates AD_rThe AD_rAnd the absolute value of the gray value difference between the rightmost pixel point of the jth line of the right side view searching block and the rightmost pixel point of the jth line of the left side view searching block when the center of the left side view searching block is at the position of n +1 is shown.

Illustratively, when the next position n +1 is the position where the center of the left side view search block horizontally slides one pixel point to the right from the current position n, all pixel points contained in the left side view search blockIs shown in table 5 and when the center of the right side view search block is located in the search range corresponding to the next position n +1 in the right side view, the numbers of all pixel points included in the right side view search block of the m-th pixel point from the right are shown in table 6, AD_l＝|A1-A`1|，AD_r＝|A6-A`6|；

When the next position n +1 is a position where the center of the left side view search block horizontally slides one pixel point leftward from the current position n, the numbers of all pixel points included in the left side view search block are shown in table 7, and when the center of the right side view search block is located in the search range corresponding to the next position n +1 in the right side view, the numbers of all pixel points included in the m-th pixel point from the right in the right side view search block are shown in table 8, AD_l＝|A0-A`0|，AD_r＝|A5-A`5|

The fourth embodiment of the present invention provides a hardware accelerator, and specifically, an arithmetic unit in the hardware accelerator is used to execute an operation based on a pixel row, and a result of a subsequent position is iteratively calculated by using a calculation result of a previous position, so that the number of elements required in the hardware accelerator is greatly reduced, and the light weight of a hardware structure is realized.

EXAMPLE five

As shown in the figure, the bin (140) receives the operation results output by the first operation subunit (1301), the fifth operation subunit (1305) and the sixth operation subunit (1306), and can send the data to the sixth operation subunit (1306) for operation.

Wherein the first computing subunit (1301) calculates the HSAD at the time of the initialization operation, and performs 10 initialization operations, for example, if the region of interest includes 10 rows, to obtain 10 × width × M HSADs in total;

the fifth operation subunit (1305) calculates HSAD except for the initialization operation through bilateral operation, illustratively if the region of interest includes 10 rows, each row has 8 pixel points except for the position of the initialization operation, and 10 × 8 × winsize × M HSAD are obtained in total;

the sixth operational subunit (1306) calculates SAD in the following way:

wherein, HSAD_jWhen the center of the left side view search block is positioned at a certain position in the left side view interesting area, the HSAD of the jth row when the center of the right side view search block is positioned at a certain position in the right side view, j is 0 … winze-1, M is more than or equal to 1 and less than or equal to M, and M is the preset maximum parallax;

illustratively, if the region of interest includes 10 rows, each row including 9 pixels, 10 × 9 × M SAD's are obtained in total;

illustratively, one skilled in the art can determine that a bin holds 10 × 9 × M SADs, and that each SAD corresponds to a left view search block center position and a right view search block center position.

EXAMPLE six

In fig. 1, the comparison unit (150) is configured to read data stored in the bins (140) and output a comparison result.

In the information bin, for each pixel point in the region of interest, M SADs and M right side view search block center positions are correspondingly stored. The comparison unit (150) compares the M SADs, determines the center position of a right side view search block corresponding to the minimum value of the SADs, and determines pixel points corresponding to the center position of the right side view search block as matching points. Illustratively, if the region of interest includes 10 lines, each line includes 9 pixel points, the comparing unit (150) performs 90 comparisons, and for one pixel point in each comparison, determines a right-side view search block center position corresponding to a minimum value of SAD of the pixel point, and determines a pixel point corresponding to the right-side view search block center position as a matching point.

Optionally, the comparing unit may compare when each pixel point and its corresponding SAD and right search block center position are stored in the bin, or compare when all pixel points in the region of interest and their corresponding SAD and right search block center positions are stored in the bin.

EXAMPLE seven

In the figure, the hardware accelerator further includes a preprocessing unit (180) and a denoising unit (170), wherein the preprocessing unit (180) is connected to the obtaining unit (110) and the logic bank group (120) and is used for preprocessing the image, and optionally, the preprocessing mode is sobel filtering. The denoising unit (170) is connected with the comparison unit (140) and is used for denoising the comparison result output by the comparison unit, and optionally, the matching point is verified by adopting uniqueness verification and a parallax verification function, so that the noise point is removed.

Example eight

In the figure, the hardware accelerator further comprises a matching point acquisition unit (190), a camera coordinate system generation unit (200) and a parallax determination unit (210), wherein the matching point acquisition unit (190) acquires the matching points subjected to denoising processing, the camera coordinate system generation unit (200) establishes a camera coordinate system based on internal and external parameters of a camera, and the parallax determination unit (210) determines the parallax of the matching points.

Example nine

In the figure, the hardware accelerator further comprises an information acquisition unit (220) and a depth information acquisition unit (230), wherein the information acquisition unit (220) receives the parallax output by the parallax determination unit (210), and the depth information acquisition unit (230) determines the depth information of the point of the target corresponding to the matching point in the camera coordinate system by using a trigonometry method.

It should be noted that the foregoing is only a preferred embodiment of the invention and the technical principles employed. The technical content of the above embodiments can be selectively combined by those skilled in the art according to different technical requirements. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A hardware accelerator is characterized by comprising an acquisition unit, a logic bank group, a comparison unit, a control unit, an information bin and an arithmetic unit;

the acquisition unit is used for acquiring image data to be operated;

the logical bank group is composed of a plurality of rams and is used for storing the image data; the logic bank group comprises a plurality of logic bank units, each logic bank unit is composed of a plurality of rams, the plurality of rams are divided and combined to form a plurality of logic banks, the plurality of rams are divided into a plurality of rams, the plurality of logic banks are composed of a plurality of rams, at least two rams in any one logic bank do not belong to the same ram, each logic bank unit is composed of a plurality of logic banks, and the plurality of logic banks can be combined according to a preset mode;

the arithmetic unit is used for carrying out arithmetic on the data in the logic bank group and outputting an arithmetic result;

the information bin is used for storing the operation result output by the operation unit;

2. The hardware accelerator of claim 1 wherein the accelerator further comprises a pre-processing unit that SOBEL filters the image.

3. The hardware accelerator of claim 1 wherein the comparison result is a feature matching point of an image that satisfies a predetermined condition;

the hardware accelerator also includes a de-noising unit to validate the matching points to remove noise points.

4. The hardware accelerator of claim 1 wherein the image data acquired by the acquisition unit is left and right view data.

5. The hardware accelerator of claim 4 wherein the left and right views are derived from a binocular vision system, the left view captured by a left camera in the binocular vision system, and the right view captured by a right camera in the binocular vision system.

6. The hardware accelerator of claim 4 wherein the left and right views are derived from a monocular vision system, and the left and right views are each captured by the monocular vision system at different locations.

7. The hardware accelerator of claim 4 wherein the control unit is further configured to set a search block in each of the left view and the right view, the search block being a left view search block and a right view search block, respectively, and the left view search block and the right view search block slide horizontally in the corresponding view to cover a pixel point of a search block size.

8. The hardware accelerator of claim 7, wherein the left view search block and the right view search block are each a winsize square search block, the winsize being an odd number and 5% or less winsize or less 21.

9. The hardware accelerator of claim 8, wherein each logical bank is configured to save grayscale values of pixels covered by the sliding of the search block in the view.

10. The hardware accelerator according to claim 9, wherein the arithmetic unit comprises a first arithmetic subunit composed of a subtractor and an adder, the first arithmetic subunit is configured to calculate an HSAD, the HSAD is a sum of absolute values of gray level differences of pixels in a corresponding row in the left view search block and the right view search block, and the first arithmetic subunit is connected to the logical bank group and the information bin and sends the HSAD to the information bin;

when the center of the left side view searching block slides to the A-th pixel row of the left side view interesting area for the first time, the control unit sends a control signal to the arithmetic unit, the first arithmetic subunit performs initialization operation to determine the HSAD of the current position n, the current position n is the position of one pixel point in the A-th pixel row in the left side view interesting area, and A belongs to [1, the total pixel row number of the interesting area ];

the first operation subunit calculates the HSAD by:

r_sobel_ishown in the preprocessed right side view r _ sobelWhen the center of the right view searching block is at the position m, the gray value of the ith pixel point in the jth line in the right view searching block;

11. The hardware accelerator of claim 10, wherein the arithmetic unit further comprises:

12. The hardware accelerator according to claim 11, wherein after the first computing subunit completes the initialization operation, the center of the left view search block slides in a horizontal direction until all pixel points in a row of a current position n in the left view interesting region are traversed, and the computing unit performs bilateral operation to obtain an HSAD corresponding to each pixel point and stores the HSAD in the information bin.

13. The hardware accelerator of claim 12 wherein the arithmetic unit further comprises a sixth arithmetic sub-unit coupled to the bins, the sixth arithmetic sub-unit comprising an adder for calculating SAD.

14. The hardware accelerator of claim 13 wherein the sixth computational subunit calculates SAD by:

wherein, HSAD_jWhen the center of the left view search block is located at a certain position in the left view region of interest, the HSAD of the j-th row when the center of the right view search block is located at a certain position in the right view is shown, wherein j is 0 …

winsize-1；

15. The hardware accelerator of claim 14 wherein the bins are used to store HSAD and SAD, and left view search block center position and right view search block center position.

16. The hardware accelerator of claim 15 wherein the control unit sends a control signal to the arithmetic unit to stop the operation when the left view search block center traverses all pixels in the left view region of interest.

17. The hardware accelerator of claim 16, wherein when the center of the left side view search block slides from the current position n to the next position n +1 within the left side view region of interest, the control unit sends a control signal to the comparison unit to determine the center position of the right side view search block corresponding to the minimum SAD value corresponding to the current position n, and determine the pixel point corresponding to the center position of the right side view search block as the matching point.

18. A binocular vision system parallax information extraction hardware accelerator is characterized by comprising a matching point acquisition unit, a camera coordinate system generation unit and a parallax determination unit;

the matching point acquisition unit acquires a matching point obtained by using the hardware accelerator according to any one of claims 1 to 5 and 7 to 17;

19. A hardware accelerator for extracting parallax information of a monocular vision system is characterized by comprising a matching point acquisition unit, a camera coordinate system generation unit and a parallax determination unit;

the matching point acquisition unit acquires a matching point obtained by using the hardware accelerator according to any one of claims 1 to 4 and 6 to 17;

20. A depth information extraction hardware accelerator, comprising:

an information acquisition unit configured to acquire parallax information obtained by the parallax information extraction hardware accelerator according to one of claims 18 or 19;