KR101625562B1 - Block-based signal processing - Google Patents

Abstract

Signal flows for data processing applications may be implemented to enable each processing node in the flows, if it contains a sufficient amount of input data in its input buffer. In various embodiments, such signal flows can be defined graphically within a GUI tool that automatically generates code that is suitable for later performing signal flow.

Description

Block-based signal processing {BLOCK-BASED SIGNAL PROCESSING}

The present invention generally relates to signal-flow architectures and other data processing applications for images, and in some embodiments, a tool for generating program code for implementing signal flow based on a graphical representation to a tool.

Generally, image processing applications include a plurality of functional processing blocks, hereinafter referred to as "nodes ", which are sequentially executed to provide raw image data to the final images presented to the user Transforms and / or analyzes the image data to extract information about the objects or conditions they capture. In such applications, an algorithm that controls the signal flow necessary to connect the nodes (i.e., manages input, output, temporary data storage and data delivery for multiple functional blocks) typically forms the core of the application , Especially when implemented on digital signal processing (DSP) or in hardware, often consume a significant portion of the processing power. Figure 1 illustrates an exemplary signal flow of an algorithm for foreground " blob "detection, which may be used, for example, to detect people, cars, or other objects in images. The first node 100 ('ABS DIFF') computes the pixel-by-pixel difference of the image values (e.g., gray scale values) between one image and the background reference image. This computed difference is thresholded for a fixed or adaptive threshold and a binary image is generated at the second node 102 ('BINARY THRESHOLD'). The binary image is further processed at the 'EROSION' and 'DILATION' nodes 104 and 106 to reduce noise pixels and improve binary image output. The final node 108 ('CONNECTED LABELING') identifies the connected pixels in the binary image and marks them as "blobs".

Developing the appropriate program code to implement the signal and data flow (whether in a low-level DSP language or in a high-level language such as C or C ++) is usually a daunting task for an algorithm or application programmer, Access control, and so on. It is therefore desirable to automate or semi-automate such tasks. There are available program tools that automatically generate code from a graphical representation of the signal flow created by an application developer in a graphical user interface (GUI). These tools support a cumulative sample-based signal flow structure or a frame-based signal flow structure, where processing nodes operate on separate data samples or entire frames, respectively. The sample-based tools are widely used, for example, for audio signal processing and motor control. However, these sample-based tools typically require faster sample throughput speeds, for example because a single image already contains multiple data samples (i.e., pixels) ), These tools may not be suitable for a large number of image processing applications, often including processing steps that operate on sets of samples. For example, an image-smoothing step may include replacing each pixel with an average for a block of several pixels, wherein the one-dimensional Fourier transform essentially consists of applying the entire Requires a row. Other tools operate on the entire image frames. However, in many circumstances it is not necessary to process complete image frames. Also, in real image processing applications implemented on a DSP or other special purpose processors with limited local memory (rather than on a general purpose computer), frame based architectures require frequent access to external (off-chip) memory, Inefficient.

Thus, there is a need for tools that help application developers to implement such signal flows as well as signal flow structures that facilitate efficient image processing on a DSP or other hardware depending on memory and bandwidth constraints.

The present invention relates to signal flow structures that facilitate block-based data (particularly, image) processing and overcome many of the drawbacks of sample-based signal flow and frame-based signal flow. Block-based signal processing typically operates to reduce the frequency of memory requirements and external memory access associated with individual processing steps, resulting in an overall increase in efficiency over frame-based processing. The data "block" used in the text is a collection of data samples forming a subset of the more complete data set referred to as the "frame ". For example, an image frame includes all of the visual data samples obtained by a camera or other optical detector at a particular point in time, and generally (although a one-dimensional or three-dimensional visual data set is also within the scope of an & Dimensional matrix or array of image pixels. An image block may consist of, for example, one or more rows or columns of image frames, or a sub-array that extends a portion of a plurality of rows and / or columns around a particular pixel. Although image processing is one of the key applications of block-based signal processing and is used extensively throughout the specification for purposes of illustration, the present invention is not limited to image data, but may be applied to block-based processing (e.g., It is to be understood that the present invention is generally applicable to any type of data (including measurement or simulation in the field, or multi-channel audio signals).

The signal flow according to the text typically comprises a plurality of nodes, each node corresponding to the performance of a particular processing function on the functional unit, i. Nodes can typically be implemented in hardware (i.e., circuitry), software (i.e., a set of executable instructions), or a combination thereof. In a software implementation, in some embodiments each node corresponds to a set of instructions executed by a separate functional block or processor; In some embodiments, one or more nodes are each implemented by a plurality of instruction blocks; In some embodiments, two or more nodes are implemented together by a single block of instructions. Similarly, in a hardware implementation, each node may correspond to a single dedicated circuit; Multiple nodes may be implemented by multiple functional circuits; And / or multiple individual circuits may collectively implement a single node. Depending on the context, a "node" may refer to a processing step or function itself in the following text, or may refer to an implementation in hardware and / or software.

In a block-based signal flow, the size of the block needed to generate the output of one unit (which may be a separate sample or block of data) at one node may be different between the nodes. For example, in image processing applications, one image processing step may operate on one row of inputs to produce one row of output, while the other step requires three rows of input for each row of output . The nodes typically have associated input buffers for storing the required amount of data. In some embodiments, each node is triggered as soon as sufficient data is available in its input buffer to produce an output of one unit; As a result, the total local memory requirements and latency are minimized. In embodiments where a single processor or circuit is executing multiple nodes, each node having sufficient data in its input buffer is enabled for execution and executed as soon as the computing capacity of the processor or circuit is allowed. The buffer used in the text refers to any means for storing data and may be any type of storage medium, device or device, such as a random-access memory (RAM), a hardware register, a flip-flop, Lt; / RTI > may be implemented in a structure that includes one or more partitions or sections in a combination of < RTI ID = 0.0 > The buffer need not be a single contiguous unit but may contain multiple portions at different memory locations. In addition, the buffer may store data directly or in pointers to different memory locations indirectly.

In addition to block-based data processing methods and hardware for implementing them, the present invention, in many embodiments, allows application developers to define signal flows graphically and automatically generate appropriate program code based on the graphical signal flow representation We present GUI-based tools that can be used. These tools typically include a library of functional data processing blocks (i. E., Nodes), an editor for drawing a signal flow incorporating nodes from the library, a compiler for generating code from the signal flow, and optionally, And a simulator for testing.

In a first aspect, the present invention provides a method for processing data frames (e.g., image frames, etc.) by a series of processing nodes, each node having an input data block, (E.g., a plurality of rows of image frames) to generate an output of one unit (e.g., a row of image frames). The method includes receiving data in input buffers associated with the nodes and causing execution of each node when the node's associated input buffer has stored sufficient data to produce an output of one unit . The execution of the node can be directly caused by triggering the node as soon as the input buffer has enough data; This may be the case, for example, in embodiments where the node has dedicated circuitry, a processor, or another computing unit that simply waits for a trigger signal before starting processing. Alternatively, the execution of the node may be caused indirectly by changing the state of the node so that the process is enabled or authorized. In such a case, the node will be processed as soon as the processor implementing the series of nodes has free capacity.

In a second aspect, the present invention provides a method of controlling signal flow in a data processing system that implements a series of processing nodes, each node having an input of a unit of node-specific integer-multiple (E.g., one row of data) from an input data block containing the data. The method includes the steps of controlling signal flow through a series of nodes by receiving data in input buffers associated with the nodes, and in the case where the input buffer associated with the node stores data of a unit of a respective node- (I. E., Triggering or enabling the node) to cause the execution of each node.

At each processing node, data may be received from a preceding node and / or a DMA source node. In some embodiments, the first processing node in the series reads data from the DMA source node and the last processing node in the series writes data to the DMA sink node. In certain embodiments, a counter is maintained for each input buffer; The method includes incrementing a counter for each unit of input data received from a preprocessing node or a DMA source node. In some embodiments, the memory allocated to the buffer associated with one of the processing nodes is reused for the buffer associated with the downstream node of the node itself. The processing nodes may be executed in parallel or sequentially.

In a third aspect, the present invention provides a system for processing data frames by a series of processing nodes. Each node is configured to process a node specific block of input data to produce output data of a unit, each block comprising a plurality of data samples and being part of a data frame. The system includes one or more processing blocks that implement a series of processing nodes, a plurality of input buffers associated with the nodes, and a plurality of input buffers associated with the nodes, And a logic translation mechanism that causes execution of each node. In some embodiments, the system includes a plurality of processing blocks, each processing block corresponding to one of the processing nodes.

The processing block (s) may be implemented with processor executable instructions stored in memory. Alternatively, the processing block (s) may be implemented in circuitry. In some embodiments, a single circuit for sequentially executing a series of processing nodes is presented, and in some embodiments, the processing node (which has already been caused or enabled by a logic translation mechanism) A plurality of circuits are shown for executing them in parallel. The term "circuit" used in this context may be a processor core, a self-contained portion of a core, an arithmetic logic unit, or any other general purpose processing unit. The switching mechanism may include a plurality of registers for storing a plurality of input units associated with a node specific block of the node for each node and a counter for a plurality of input units currently stored in a buffer associated with the node. The registers may be hardware registers, or may be stored in local memory associated with the processing block (s). In some embodiments, the system is a digital signal processor (DSP).

In a fourth aspect, the present invention provides a system for generating program code for block-based signal processing from a graphical representation of a signal flow defined in a graphical user interface. The system includes a processor, a memory storing instructions executable by the processor, and optionally a display device (e.g., a computer screen) for displaying a graphical user interface. (Ii) a library of functions implementing the signal-processing nodes, each node configured to generate a unit of output data from a block of input data having a node-unique size; (ii) Instructions for causing a user to graphically define a signal flow comprising a plurality of nodes and a connection therebetween, and implementing an editor to associate each of the nodes with one of the functions from the library; ) Commands that implement a compiler for generating program code from a graphically defined signal flow and associated functions-code is used when the buffer associated with the node stores a block of input data of each node-specific size To cause the node to execute. The editor further allows the user to graphically define direct memory access (DMA) of the signal flow including, for example, DMA sources, DMA sinks, and / or DMA scheduling paths, and the compiler allows the graphically defined DMA Lt; RTI ID = 0.0 > a < / RTI > If the user does not define a scheduling path, the compiler can automatically generate the DMA scheduling path. The compiler includes program code for resolving data parallelism in DMA paths, allocating ping-pong buffers in the source node buffer and the sink node buffer, and program code for allocating the ping- And may further generate code that implements the associated buffers. The editor allows the user to enter parameters into the DMA parameter window, and the compiler can generate DMA register entries based on the parameters.

The foregoing description, in particular, together with the drawings, will be better understood from the following detailed description of the invention.
1 is a signal flow diagram for a conventional example image-processing application.
2A is a conceptual signal flow diagram illustrating buffer requirements associated with each node in one implementation of row based signal processing.
Figure 2B is a conceptual signal flow diagram illustrating reduced buffer requirements associated with nodes in a row-based signal processing implementation including switches between nodes in accordance with an embodiment of the present invention.
3 is a block diagram illustrating a system for row based signal processing in accordance with various embodiments.
Figures 4A and 4B are diagrams illustrating a row-based signal processing flow including switches between nodes in accordance with various embodiments and registers, respectively, that control the data flow and operation of switches.
5 is a block diagram illustrating a GUI-based tool for graphically defining a signal flow according to one embodiment and automatically generating code based thereon.
Figures 6A-6H illustrate user-interface components of a GUI-based tool in accordance with various embodiments.
Figure 7 is a block diagram illustrating a computer system for implementing the GUI-based tool of Figure 5 in accordance with various embodiments.

Data processing algorithms according to various embodiments operate on blocks of data rather than data samples or entire frames. Such blocks may consist of, for example, one or more rows of a two-dimensional data array or one or more slices of a three-dimensional array. Row-based data processing is suitable for many image processing applications, for example, to perform two-dimensional filtering, such as convolution, or two-dimensional filtering, such as erosion and dilation. Or even essential. 2A and 2B illustrate signal flow for an exemplary row-based algorithm including four nodes 200, 202, 204, 206 or processing steps, each step comprising a specific number of input rows (to the left of the node ) Into a single line of output data. For successive output rows, the corresponding blocks of the plurality of input rows overlap so that each frame of the input data is a frame of the same size as the output data. For example, row n of the output frame may be generated from rows of the input frame, n-1, n and n + 1, such that the three row input blocks for two adjacent output rows overlap by two rows. (The input frame is, for example, padded with 0s to provide the required input data in the vicinity of the frame.) Alternatively, if the input frame itself is not padded, Create.)

In the illustrated signal flow, the first node 200, "node 0 " receives input data from the DMA source node 208 via DMA, and the last node 206," node 3 " And writes the output data to the node 210. [ Each node has an associated buffer at its input to temporarily store the output from the immediately previous node. In one embodiment, buffers are sized to store enough data to generate one row of output at node 3, as shown in FIG. 2A. Node 3 requires one row of input to generate a row of output data, and therefore its buffer 212 is configured to store a row of data. Node 2 requires five input lines for one row of output, and thus requires buffer 214 to store the data for five rows. Node 1 requires three lines of input to produce one line of output. However, in order to calculate the input of five rows required by node 2, node 1 requires a total of seven rows of input data, rows 1-3 produce the output of the first row, (2-4) produces the output of the second row, the rows (3-5) produce the output of the third row, the rows (4-6) produce the output of the fourth row, -7) produces the output of the fifth row. Thus, the input buffer 216 at node 1 is configured to store seven rows. Similarly, node 0 requires input of five rows to produce an output of one row, but to provide data for the seven rows required by row 1, input of a total of eleven rows (1-5) for the output of the row, (2-6) for the output of the second row, etc.). In general, a node that computes the output of one row from the input of n rows and that requires input of a total of m rows requires a buffer to store n + m-1 rows. Thus, the required number of input rows is cascaded to the source node 208. [ For signal flows involving a large number of nodes, the resulting buffer requirements may exceed the capacity of local memory, require external memory accesses, and block-based processing is intended to be eliminated.

Figure 2B shows a modified signal flow that improves the memory problem. Wherein the data flow is controlled by a control signal flow that triggers each node as soon as the input buffer of the node contains a sufficient amount of data to produce an output of one row, , Each of which connects the output of the immediately preceding node to the input of the next node and closes if sufficient data is available in the input buffer of the next node. Thus, for example, if node 1 receives the input of three nodes, it processes these rows to produce an output of one row, which is stored in the input buffer of node 2. The input buffer of node 1 may then be overwritten. Specifically, the second and third rows in the input buffer can be shifted up one by one (overwriting the first row) and the next row of inputs can be received (from node 0) And stored in the third row of the input buffer. In this way, the buffer size necessary for each node is reduced to the number of input rows the node needs to produce an output of one row, and in the example of Figure 2b, the nodes (0, 1, 2, and 3) 224, 226, 228 for each row, three rows, five rows, and one row, respectively. As can be readily appreciated by those skilled in the art, the cascading effect experienced by the signal flow of FIG. 2a is eliminated, the overall buffer requirement is approximately proportional to the number of nodes, or the buffers As well as shared between nodes). The buffers for the various nodes may generally be implemented at different levels of the memory hierarchy, based on the buffer size and the required access frequency. For example, the smallest buffers (for one row of data) may be implemented in L1 memory, and the larger buffers may be implemented in L2 or L3 memory or cache memory (associated with larger latencies).

The signal flow shown in FIG. 2B may be executed using a single processor. In this case, the processor is controlled to always run the most distant node towards the DMA sink node 210 among the nodes with sufficient data available in its input buffers. For example, when starting with a new data frame at the input, node 0 is executed three times to generate the minimum amount of data required by node 1. Then, the switch between node 0 and node 1 is closed, and node 1 is executed to generate the output of one row. Thereafter, the second and third rows in the input buffer of node 1 are shifted up by one row, and node 0 is again driven to generate the third row of data for the input buffer of node 1. Next, node 1 is run again to generate a second row of data for the input buffer of node 2. This process is repeated three more times until five rows of input are available at node 2, where node 2 is executed and node 3 is followed. The processor then returns to node 0 and the entire loop repeats until the entire input data frame is processed. During each loop, the memory can be reused among the nodes, i. E. The buffer space can be shared. For example, the output row of node 2 may be stored in memory previously assigned to node 1's input buffer. If the buffers are still storing three already processed rows, the first row may be overwritten by the output of node 2 and will not be needed anymore. On the other hand, if the previous rows 2 and 3 of the input buffer of node 1 have already been copied to rows 1 and 2, then the third row of the buffer can be used to store the output of node 2.

In some embodiments, various nodes of the signal flow are executed in parallel (e.g., in a time-shared interleaved manner) by a plurality of processors, or a single processor that simultaneously runs a plurality of threads. In such a scenario, memory reuse among the buffers is not possible, but the total execution time is significantly slower as long as the switch at its input is closed, i.e., as long as enough data can be utilized in its input buffer, Can be reduced. In general, if the buffer between two nodes is filled (for example, the input buffer of node 1 receives three rows of input) and the switch between nodes is closed, the buffer is filled from one end and is at the same rate Out from the other end, and the switch is closed until the entire frame is processed. In other words, following the initial buffer filling, the data movement across the nodes occurs in a pipelined manner.

The signal flow shown in Figure 2B can be modified in many ways. In addition to including more (or fewer) processing nodes than the illustrated nodes with different data entry requirements, the signal flow may include additional DMA sources and / or sink nodes and / DMA sources and / or sink nodes. In general, each node in the signal flow can read data from the memory (i.e., is the source node) or write data to the memory (i.e., is the sink node). Also, the nodes in the signal flow need not necessarily form a linear chain. In some applications, the signal flow includes two or more parallel nodes, i.e. nodes that process the input data independently, and the aggregate outputs may be required by another node for further downstream in the signal flow. Of course, the signal flow can in principle be branched and rejoined in any complex way. Also, in some embodiments, certain nodes may be optional. For example, in a typical image processing application, a node for reducing resolution and thus reducing the size of image frames may or may not be executed, for example, according to the settings specified by the user of the application. In order to implement this optional node, the signal flow may include, for example, "switches" at the input and output of the node, and user selection (or some other condition, e.g., a numerical value of the metric derived from a previous processing step) And a node by-pass using additional control signal lines to determine switch settings based on the control signal lines.

The use of "switches" for triggering the operation of nodes, as described above as an example of row-based processing, can be generally applied to any kind of block-based processing irrespective of the particular type and size of data blocks. The key is that each node in the signal flow is triggered to execute when a sufficient amount of data is received in its input buffer to generate an output of one unit, and the size of the unit depends on the particular application. For example, consider an image-smoothing step (i.e., node) that replaces the value of each pixel with an average value of the 3x3 block of pixels centered at the issue pixel. This node has an output unit size of only one pixel and executes in its input buffer when, for example, it has a 3x3 block corresponding to a block centered at the coordinates (n, m) of the image frame, The calculated output value is written to the input buffer of the next node in such a manner as to save the coordinates (n, m). The next 3x3 block processed by this node can be shifted by one column to the right (i.e., centered at (n, m + 1) in the image frame) (N, m + 1) in the input buffer of the node.

The size of the input data block for each node is generally determined by the size of the output unit from the immediately preceding node (or the size of the output unit from the preceding node (s)) so that repeated execution of the preceding node (s) The size of the combined output unit of the preceding group of nodes, if taking input from a group of nodes. In various embodiments, the output unit size is the same for all nodes. For example, in the signal flow of FIG. 2B, the output unit of each node is one row of data, and all of the input data blocks required by the nodes consist of integer rows. The use of a single output unit for all nodes does not require saving state variables for each node as required for other block-based processing schemes such as 8x8 overlap blocks across column boundaries , Programming the signal flow, and DMA transfer of the data in the signal flow. In an 8x8 overlap block processing (as compared to row based processing where a node will operate on a buffer of 8xM size, where M is the number of pixels in one row of the image) For a < RTI ID = 0.0 > 8 < / RTI > data block, the previous column from the previous 8x8 blocks needs to be stored as a state variable at that node.

FIG. 3 shows an exemplary DSP implementation of block-based signal processing flows, such as the flow shown in FIG. 2B. DSP 300 implements each node of a signal flow having a dedicated logic unit or a separate hardware (represented by a "processing block"), such as a processor core, Data can be processed in parallel. Specifically, although the illustrated embodiment shows only two processing blocks 302 and 304, it should be appreciated that signal flows having any number of nodes may be implemented. The data moves between the processing blocks 302 and 304 and is temporarily stored in the input / output buffers 306, 308 and 310. Register bank 312 stores control parameters that monitor the fill state of buffers 306, 308 and 310 and also triggers data flow between buffers and processing blocks 302 and 304, Quot; switches "220 between the nodes. The input buffer 306 of the first processing block receives data from an internal or external data source 314, 316, such as a memory, a camera that provides image streams, or some other input device, Thereby facilitating selection among a plurality of data sources. The output buffer 310 of the final processing node 304 sends data (optionally via another multiplexer 320) to an internal or external data sink 322, such as, for example, a memory or display device.

In video or image processing applications that are implemented on a DSP or hardware (rather than software running on a general purpose computer), image frames are typically too large to be stored locally, )). ≪ / RTI > The image data is loaded into the internal memory as rows or blocks. After these rows or data blocks are processed in a series of processing blocks (e.g., blocks 302 and 304), the generated output is similarly processed (corresponding to data sink 322, which may be the same memory as source 316) It is stored in low speed external memory. Movement of data to or from the external memory to the external memory (e. G., Buffers 306 and 310) is preferably through a DMA controller, which is responsible for most DSPs and other Purpose processors, and may be implemented in, for example, multiplexers 318 and 320. [ In this case, the external data source and sinks 316 and 322 are DMA-enabled (DMA-enabled). DMA migration, along with data processing itself, has the additional advantage of parallelizing data movement to and from the processor; In other words, data movement occurs in the background.

Of course, the hardware embodiment of Fig. 3 is merely an example. As will be readily appreciated by those skilled in the art, signal flows in accordance with various embodiments of the present invention may be implemented in a variety of ways. For example, the DSP may use a single processor core to execute sets of instructions stored in a local instruction memory, e.g., corresponding to different nodes. Also, the data buffers can share the same memory space and can be instantly created and / or over-written as needed. The registers for controlling switches between nodes may be hardware registers or, alternatively, may be stored in local memory with buffers and / or instructions. As an alternative to DSP, the signal flow may be on any other type of special purpose processor, including, for example, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or programmable gate arrays (PGAs) . Also, the resulting signal flows can be implemented in software running on a general purpose computer.

Turning to the hardware embodiment (e.g., as shown in FIG. 3), FIGS. 4A and 4B show the operation of the switches implemented by the registers 312 in more detail. In the illustrated example, the signal flow (shown in FIG. 4A) includes two nodes 400, 402. Each of the nodes 400, 402 is associated with one-dimensional or two-dimensional data buffers at input and output. Specifically, as shown, node 0 receives input from two DMA source nodes S0 and S1 connected to the node via switches 404 and 406, which is connected to its data buffers B0 and B1 408 and 410, . The output of node 0 is supplied to a two-dimensional buffer B2 412, which is consequently connected to the input of node 1 via switch 414. [ The output of node 1 goes through switch 418 to data buffer B3 416 coupled to DMA sink S2.

The system maintains four different register arrays 420, 422, 424, 426 that collectively form register banks 312 to control signal flow through nodes 400, 402. Each array may comprise a plurality, e.g., 32-bit registers. The node-to-source address register array 420 includes registers for each input source of the node: two entries for the two input sources of node 0 and one entry for the input source of node 1, . The entries of these registers are the addresses of the data buffers for each input source, i. E. The addresses of buffers B0, B1 and B2. Once initialized, these register entries are not changed during the entire signal processing. The node-destination address register array 422 includes register entries for each output of the node: one entry for the output of node 0 and one entry for the output of node 1, for example. The entries of these registers are the addresses of the data buffers for the respective outputs, e.g., buffers B2 and B3. In one-dimensional buffers at the output of the node, the values in the corresponding registers are not changed once initialized. In the two-dimensional buffer in the output (e.g., a buffer B2 in the node 0 output), the entry being initialized B2 + 2 ^nd line value (that is, the initial buffer address in the register is the 2 ^nd row in the buffer B2 Memory address), after the first iteration, the entry is changed to line B2 + 3 ^rd , and then remains the same. (The output of the previous node serving as the input buffer for the current node will always be written to the same memory address after the first iteration, so no update is needed.) This simplifies data movement for nodes with 2D buffers. Note that the iteration depends on the size of the buffer. For example, if the buffer is 3 × M, the iteration is 3/2 = 1, and if it is 5 × M, the iteration is 5/2 = 2.

The switch value register array 424 determines when the control switch for each node closes: it is in each register that the switch is closed and the minimum value of the input data required in each node ' s input buffer And stores a number of rows (or, more generally, units). For example, switch 404, which couples buffer B0 to node 0, stores one row of data in buffer B0; This value is stored in the register. Switch 414 couples buffer B2 to node 1; The associated buffer requirements of the three rows are stored in the appropriate registers. Finally, within node count register array 426, each register entry is associated with a control switch for the DMA sink to which the input source or node to the node is connected. These register values track the number of data rows processed by each node (and thus are updated by the node each time an input data row is processed) and are the counters that control when the switch to the node closes. Initially, the register values C0, C1 and C3 for buffers B0, B1 and B3 are all zero and the register value C2 for buffer B3 is one. (For the secondary buffers, the initial value is chosen to be the number of rows in the buffer that are divided by 2 and rounded down, resulting in, for example, the value of round (3/2) , If the DMA source node S0 fills the buffer B0 with one row of pixels (samples) and the DMA source node S1 fills the buffer B1 with one row of pixels, the values are updated to C0 = 1 and C1 = 1 . The switch associated with each counter register is closed if its value is greater than or equal to the value stored in the corresponding switch register. In pseudo-syntax, this corresponds to the following "if statement".

if (C0 > = SWITCH0 and C1 > = SWITCH1)

{close switch 0 and switch 1; process node 0; increment C2 by 1}

if (C2 > = SWITCH2)

{close switch 2; process node 1; increment C3 by 1}

In various embodiments, the present invention provides GUI-based tools to aid application programmers in designing and implementing signal flows as described herein. 5, the GUI tool 500 includes a drawing canvas 504 and an associated drawing tool 506 that allow the programmer to generate a depiction of the desired signal flow. Gt; 502 < / RTI > The tool also executes a library 508 of functions or procedures that performs various individual image processing algorithms and, if selected, serves as functional nodes, i.e., processing blocks, of the signal flow - self- contained sets of computer executable instructions. In some embodiments, each function and procedure has an icon or other graphical representation associated with it. The programmer can select desired functions or procedures from the library by dragging and dropping each icon onto the drawing canvas, and then define the signal flow by connecting them, for example, by drawing lines between them. In another embodiment, a programmer may define signal flow using common shapes and symbols, and then assign functions to various symbols.

The functions may be optimized for a particular processor or specific hardware implementation. Indeed, in some embodiments, multiple versions of executable code, optimized for different hardware implementations, are provided for the same functionality, allowing the program developer to select one of them. In addition, functions or procedures may be inherently programmed for unique input block sizes and output unit sizes. Alternatively, the input and output blocks for each function or procedure may vary in size and allow the programmer to specify their sizes based on a particular application. In some embodiments, the library includes both fixed size data blocks and variable size data blocks. In addition to the block size, other user selectable parameters may be associated with the various functional blocks.

The GUI tool 500 also includes a compiler 510 that automatically generates program code 512 to execute the desired signal flow from the graphical depiction. The compiler 510 adds the appropriate functions from the library 508, for example, by directly copying or linking them into the program code 512, and adding the necessary instructions to control the movement of data between the nodes. The compiler 508 may comprise a set of rules for translating graphics elements representing connections, switches and buffers into appropriate executable instructions, for example. In some embodiments, the GUI tool may support multiple programming languages; In this case, the library 508 includes program code for each function in each of the supported languages. In some embodiments, the GUI tool may allow a programmer to test a particular signal flow for evaluation of certain performance parameters, such as memory requirements, execution time for a particular processor, processing latencies, and the like Lt; RTI ID = 0.0 > 514 < / RTI > The simulator 514 may be integrated into the compiler 510.

6A-6H illustrate an exemplary GUI for signal flow programming in accordance with one embodiment. The GUI includes a panel 600 of taps next to the drawing canvas 504. "Shapes" tab 602 may include different graphic elements, such as lines, rectangles, etc., to draw a signal flow schematic. Each of the nodes (e.g., represented by a rectangle) within the block diagram may be assigned to any of the image processing (or other types of) functions available in the "IP Blocks" As shown in FIG. 6B, these functions may be provided, for example, as a drop-down list 605. Once the image processing function is assigned to the node, the parameters of the function can be entered in the parameter window for that function. The parameter windows are implemented within the GUI in association with the various functions available in the library. 6C illustrates exemplary parameter windows 608, 610, 612 for three different image processing functions.

User specified parameters (or default values, if not specified) in the parameter windows 606, 608, 610 are stored in a parameter list or array (e.g., a dual pointer array or linked list) Lt; / RTI > Typically, each node has one or more parameters stored in the list. For example, referring to FIG. 6C, if node 0 is for "thresholding" and node 1 is for "erosion", the first entry in the parameter list is for node 0 Are stored from the parameter window 612 and the next three entries are stored from the parameter window 610 for node one. The compiler 510 uses the list of parameters to return the values of the image processing functions associated with each node. Similarly, these return values are stored in a list or array.

As shown in Figures 6d through 6f, the panel 600 further includes a DMA tab 620 that allows the application developer to graphically define DMA movement and scheduling. Graphics DMA elements such as DMA source and sink nodes 622 and 624 and DMA source and sink scheduling paths 626 and 628 may be used by compiler 510 to generate appropriate code management DMA Lt; / RTI > Some of these parameters may be read directly from the graphical signal flow (e.g., the node in the signal flow to which the DMA node is connected), while others are stored in the parameter window popped up when the graphic DMA element is selected Can be input. For example, based on the user input in the parameter windows 630 and 632 for nodes 0 and 1, the compiler 510 generates two register entries in the memory array for all nodes, Can be set. More specifically, the compiler 510 may allocate memory for buffers having the dimensions "buffer width" and "buffer height " It can then set two register entries in the array or list 634 for that node; The first entry is an address automatically generated from the "buffer name" and "offset" parameters, and the second entry is based on the "DMA Stride". These lists are used by the compiler 510 to set, start, and terminate the various DMA paths. Thus, in various embodiments herein, the DMA is an integral part of the GUI and the code for DMA movement and DMA scheduling is automatically generated from user input of graphics and / or text to eliminate this other tedious task for the user.

The input required by the developer to generate the DMA code is generally an external memory buffer address for each source or sink node, a stride to go to the next row or rows of the image / video buffer, and, optionally, Scheduling associated with one or more source / sink nodes as well as associated processing nodes. If the developer does not specify scheduling, the compiler 510 may automatically generate a DMA scheduling path based on default rules. The compiler can also automatically assign dual ping-pong buffers to the source port of the node when receiving a parallel, overlapping input from the DMA node and the input buffer associated with that node have.

DMA scheduling is further illustrated in Figures 6d through 6f for an exemplary signal flow for an embodiment in which one processor cycles through the nodes. Here, nodes 0 and 1 are associated with DMA source nodes, and nodes 4 and 5 are associated with DMA sink nodes. DMA source nodes fetch data from external memory to internal buffers, and sink nodes fetch data from buffers to external memory. Each of the DMA nodes is coupled to either an output port or a source port of the processing node. The developer may specify the scheduling paths of the DMA sources and sinks (otherwise the compiler 510 will automatically find suitable scheduling paths). There may be one single path 626 for all DMA source nodes and one path 628 for all DMA sink nodes. The paths indicate when the DMA starts or ends; For example, the DMA associated with the source nodes starts before node 3 and ends after node 5; That is, while the nodes 1 to 2 are operating, the output is read through the DMA, while the nodes 3 to 5 are processing, new data comes in through the DMA. Similarly, the DMA for sink nodes starts before node 0 and ends after node 2. This type of scheduling is common in embodiments where only one DMA controller (or DMA enabling hardware / peripheral) is associated with all DMA nodes in the signal flow.

In alternative embodiments, multiple DMA controllers may be associated with DMA nodes; In this case, the DMA paths can be duplicated as shown in FIG. 6E. 6F, the buffer at the source port 2 of the node 1 and the DMA at the source port 1 are duplicated, that is, the DMA transfers data from the next row to the buffer at the source port 2 Node 1 also processes the data in the buffer at source port 2 for the previous row. This causes data parallelism or data corruption in the buffer at the source port 2 for node 1. A dual-state ping-pong buffer (known to those skilled in the art) at the source port 2 of node 1 is a parallel-to-serial ping-pong buffer To solve this problem. The compiler 510 can automatically identify instances of such redundancy and assign dual status ping buffers to the affected source ports.

The GUI tool 500 described above can be implemented in software, for example, running on a general purpose computer. 7 illustrates a system 700 that includes a central processing unit (CPU) 700 and associated system memory 702, one or more non-volatile mass storage devices (and associated device drivers) 704, (e.g., Input / output devices 706), and a system bus 708 in which the processor and memory communicate with each other and with other system components. System memory 702 stores instructions that are conceptually described as a group of modules that control the operation of CPU 700 and its interaction with other hardware components. The operating system 710 directs execution of low-level basic system functions such as memory allocation, file management, and the operation of the storage devices 704. At a higher level, one or more service applications provide a computer function for automatically generating code based on a graphical signal flow indication. These applications may include an editor 502, a compiler 510, and a simulator 514. Of course, such modules may be combined, further divided or otherwise organized; As those skilled in the art will appreciate, the instructions may generally be grouped and organized in many different ways. The system memory 702 may also store a library 508 of processing blocks. The instructions implementing the applications 502, 510, 514 may be programmed into any of a variety of suitable programming languages including, but not limited to, C, C ++, BASIC, Pascal, Fortran, or assembly language .

The terms and expressions employed herein are used in the following non-limiting descriptive terms and expressions, and the use of such terms and expressions is intended to exclude any equivalents of the features shown or described or portions thereof no. It will also be apparent to those skilled in the art that, in describing certain embodiments of the invention, other embodiments, including the concepts described herein, may be used without departing from the spirit and scope of the invention. Accordingly, the described embodiments are to be considered in all respects as illustrative and not restrictive.

Claims (26)

A storage-efficient method of processing frame data,
Receiving data from an input buffer associated with a second processing node in the series of processing nodes from a first processing node of a series of processing nodes implemented by the one or more computing devices, Wherein the node is performing an operation requiring an input that is a frame data block of a first predetermined size to generate a block of frame data of a second predetermined size, each of the first and second predetermined sizes comprising a plurality A frame data sample;
Determining, by the one or more computing devices, that the size of the data in the input buffer is equal to the first predetermined size; And
In response to a determination that the size of the data in the input buffer is equal to the first predetermined size, the second processing node generates a block of frame data of the second predetermined size To operate on the frame data in the input buffer
≪ / RTI > Claim 2 has been abandoned due to the setting registration fee. The method according to claim 1,
Wherein the frame data is image frame data. Claim 3 has been abandoned due to the setting registration fee. 3. The method of claim 2,
Wherein the second predetermined size is an image frame of one row of image frames. Claim 4 has been abandoned due to the setting registration fee. The method of claim 3,
Wherein the first predetermined size is an image frame of a plurality of rows. Claim 5 has been abandoned due to the setting registration fee. The method according to claim 1,
Wherein the first processing node is a direct memory access (DMA) source node. The method according to claim 1,
Further comprising: a reuse memory allocated to the input buffer for another buffer associated with another processing node in the series of processing nodes, wherein the another processing node is associated with the first processing node and the second processing node Different methods. Claim 7 has been abandoned due to the setting registration fee. The method according to claim 1,
Further comprising: operating in parallel a plurality of processing nodes in the series of processing nodes. Claim 8 has been abandoned due to the setting registration fee. The method according to claim 1,
And sequentially operating the processing nodes in the series of processing nodes. Claim 9 has been abandoned due to the setting registration fee. The method according to claim 1,
Wherein the step of determining that the size of the data in the input buffer is equal to the first predetermined size,
Maintaining a counter for the input buffer and incrementing the counter in response to receiving units of data from the first processing node. A storage-efficient system for processing frame data,
At least one computing device for providing a first and a second processing node, the second processing node comprising: means for generating a block of frame data of a second predetermined size, the block being a block of frame data of a first predetermined size Wherein each of the first and second predetermined magnitudes comprises a plurality of frame data samples;
An input buffer associated with the second processing node, the input buffer being sized to store an amount of frame data that is greater than or equal to the first predetermined size; And
A logic switching for causing the second processing node to operate on the frame data in the input buffer, when the input buffer stores an amount of data equal to the first predetermined size, by the at least one computing device Mechanism
≪ / RTI > 11. The method of claim 10,
Wherein the logic switching mechanism comprises: a register for storing, for the second processing node, a counter of the number of blocks of the first predetermined size currently stored in the input buffer and a representation of the first predetermined size Systems Included. Claim 12 is abandoned in setting registration fee. 12. The method of claim 11,
Wherein the register is a hardware register. Claim 13 has been abandoned due to the set registration fee. 12. The method of claim 11,
Wherein the register is stored in a local memory associated with the at least one processing device. Claim 14 has been abandoned due to the setting registration fee. 11. The method of claim 10,
Wherein the at least one computing device comprises a digital signal processor. At least one non-temporary computer-readable medium for generating program code for block-based processing of frame data from a graphical representation of a signal flow defined in a graphical user interface,
Wherein the one or more non-temporary computer-readable media stores instructions, and in response to execution by the one or more computing devices of the system,
Signal processing nodes, each of the nodes providing a library of functions to implement one or more blocks of input frame data having a node specific size comprising a plurality of frame data samples, Performing an operation requiring input;
Providing a user with an editor to graphically define a signal flow comprising a plurality of nodes and connections and to associate each of the nodes with one of the functions from the library;
A compiler provides for generating program code from a graphically defined signal flow and associated functions, the code comprising instructions that, when each of the buffers associated with the node is determined to have stored a block of input frame data of a respective node- Causing the node to run -
One or more non-temporary computer readable media. 16. The method of claim 15,
Wherein the editor causes the user to graphically define direct memory access (DMA) of the signal flow. Claim 17 has been abandoned due to the setting registration fee. 17. The method of claim 16,
Wherein the editor causes the user to define at least one of a DMA source, a DMA sink, or a DMA scheduling path. Claim 18 has been abandoned due to the setting registration fee. 17. The method of claim 16,
Wherein the compiler is for generating program code that implements a graphically defined DMA. Claim 19 is abandoned in setting registration fee. 16. The method of claim 15,
Wherein the instructions further cause the system to provide a display device for displaying the graphical user interface providing the graphical user interface to a display device for a display. Claim 20 has been abandoned due to the setting registration fee. The method according to claim 1,
Wherein the second processing node is to provide the DMA sink node with the generated block of frame data of the second predetermined size. The method according to claim 1,
Wherein the first processing node is to generate the block of frame data of the second predetermined size by performing a filtering operation on the block of frame data of the first predetermined size. Claim 22 is abandoned in setting registration fee. The method according to claim 1,
Wherein the step of receiving data from the first processing node in the series of processing nodes in an input buffer associated with the second processing node in the series of processing nodes includes receiving frame data having a size smaller than the first predetermined size ≪ / RTI > 11. The method of claim 10,
Wherein the second predetermined size is smaller than the first predetermined size. 11. The method of claim 10,
Wherein the first processing node is a DMA source node. 11. The method of claim 10,
Wherein the first predetermined magnitude and the second predetermined magnitude are different magnitudes. Claim 26 is abandoned in setting registration fee. 11. The method of claim 10,
Wherein the first predetermined size is a first number of rows and the second predetermined size is a second number of rows.

Images