KR20190101409A - Program code conversion to improve image processor runtime efficiency - Google Patents

Abstract

The method is described. The method includes constructing an image processing software data flow in which a buffer stores and forwards image data sent from a production kernel to one or more consuming kernels. The method includes recognizing that resources are insufficient for the buffer to store and forward the image data. The method also includes modifying the image processing software data flow to include a plurality of buffers for storing and forwarding the image data while transferring the image data from the production kernel to the one or more consuming kernels.

Description

Program code conversion to improve image processor runtime efficiency

FIELD OF THE INVENTION The field of the present invention generally relates to image processing, and more particularly, to program code conversion for improving image processor runtime efficiency.

Image processing generally involves the processing of pixel values organized into an array. Here, the spatially organized two-dimensional array captures the two-dimensional characteristics of the images (the additional dimension may include time (eg a sequence of two-dimensional images)) and the data type (eg color). In a typical scenario, the aligned pixel values are provided by the camera that generated a still image or a sequence of frames to capture images of motion. Traditional image processors are generally in two extremes.

The first extreme is a software program that runs on a general purpose processor or general purpose like processor (eg, a general purpose processor with vector instruction enhancement) to perform image processing tasks. The first extreme generally provides a wide variety of application software development platforms, but the use of more sophisticated data structures combined with associated overhead (e.g., instruction fetching and decoding, on-chip and off-chip data processing, speculative execution) ultimately leads to programmatic The result is that more energy is consumed per unit of data during the execution of the code.

Second, the opposite extremes, fixed functional wiring circuitry applies to much larger data blocks. The use of larger data blocks applied directly to custom design circuits greatly reduces power consumption per unit of data. However, the use of custom designed fixed function circuits typically results in a limited set of tasks that the processor can perform. As such, a widely used programming environment (associated with the first extreme) is lacking at the second extreme.

Technology platforms that offer a wide variety of application software development opportunities combined with improved power efficiency for each unit of data are still desirable but lack a solution.

The method is described. The method includes constructing an image processing software data flow in which a buffer stores and forwards image data sent from a production kernel to one or more consuming kernels. The method includes recognizing that resources are insufficient for the buffer to store and forward the image data. The method also includes modifying the image processing software data flow to include a plurality of buffers for storing and forwarding the image data while transferring the image data from the production kernel to the one or more consuming kernels.

The following description and the annexed drawings are used to describe embodiments of the present invention. In the drawing:
1 illustrates an embodiment of an image processor hardware architecture.
2A, 2B, 2C, 2D and 2E illustrate the operations performed on a sheet with parsing image data into a group of lines, parsing a group of lines into a sheet and overlapping stencils.
3A shows an embodiment of a stencil processor.
3B illustrates an embodiment of instructions of a stencil processor.
4 illustrates an embodiment of a data computing unit in a stencil processor.
5a, 5b, 5c, 5d, 5e, 5f, 5g, 5h, 5i, 5j and 5k illustrate the use of a two-dimensional shift array and an execution lane array to determine pairs of overlapping stencils and neighboring output pixel values. Illustrated.
6 illustrates an embodiment of unit cells for an integrated execution lane array and a two dimensional shift array.
7A and 7B relate to first program code conversion.
8A, 8B and 8C relate to second program code conversion.
9A and 9B relate to third program code conversion.
10A and 10B relate to fourth program code conversion.
11A and 11B relate to fifth program code conversion.
12 relates to a sixth program code conversion.
13A and 13B relate to a seventh program code conversion.
14 relates to an eighth program code conversion.
15 shows a program code conversion method.
16 relates to a software development environment.
17 relates to a computing system.

i. Introduction

The description below describes a number of new image processing technology platforms that provide a broad application software development environment that uses larger data blocks (e.g., line groups and sheets described further below) to provide improved power efficiency. Examples are described.

1.0 Hardware Architecture Embodiments

a. Image Processor Hardware Architecture and Behavior

1 illustrates an embodiment of an architecture 100 for an image processor implemented in hardware. The image processor may be targeted by, for example, a compiler that translates program code written for a virtual processor in a simulated environment into program code that is actually executed by a hardware processor. As can be seen in FIG. 1, the architecture 100 is configured via a network 104 (eg, a network on chip (NOC), an on-chip ring network, or other type of network including an on-chip switch network), and a plurality of stencil processors. Units 102_1-102_N (hereinafter referred to as "stencil processors", "stencil processor units", "image processing cores", "cores", etc.) and corresponding sheet generating units 103_1-103_N ( Hereinafter, a plurality of line buffer units 101_1 to 101_M (hereinafter referred to as "line buffers", "line buffer units", etc.) interconnected to "sheet generators", "sheet generator units", and the like. ). In one embodiment, any line buffer unit may be connected to any sheet generator and corresponding stencil processor via network 104.

In one embodiment, program code is compiled and loaded onto a corresponding stencil processor 102 to perform a predefined image processing operation by a software developer (e.g., according to design and implementation, the program code is stenciled). The association sheet generator 103 of the processor). In at least some cases, the image processing pipeline loads the first kernel program for the first pipeline stage into the first stencil processor 102_1, and loads the second kernel program for the second pipeline stage into the second stencil processor. By loading to 102_2, the first kernel performs the function of the first stage of the pipeline, the second kernel performs the function of the second stage of the pipeline, and additional control flow methods are installed Pass the output image data from one stage of the pipeline to the next stage of the pipeline.

In another configuration, the image processor may be implemented as a parallel machine with two or more stencil processors 102_1, 102_2 running the same kernel program code. For example, image data of high density and high data rate streams can be processed by spreading frames across multiple stencil processors, each of which performs the same function.

In another configuration, essentially any DAG of kernels configures each of the stencil processors into a kernel of their own respective program code, and directs the output images from one kernel to the input of the next kernel of the DAG design. It can be loaded into the hardware processor by configuring the appropriate control flow hooks in hardware.

As a general flow, frames of image data are received by macro I / O unit 105 and delivered to one or more line buffer units 101 on a frame-by-frame basis. The particular line buffer unit parses its frame of image data into smaller areas of the image data, referred to as "line groups," and then passes the group of lines through the network 104 to a particular sheet generator. A complete or "full" single line group may, for example, consist of data of multiple consecutive full rows or columns of a frame (for brevity, this specification will mainly refer to consecutive rows). The sheet generator parses a group of lines of image data into smaller areas of image data called " sheets, " and presents the sheet to its corresponding stencil processor.

In the case of an image processing pipeline or DAG flow with a single input, in general, the input frames parse the image data into line groups and the line groups correspond to the stencil processor 102_1 executing the code of the first kernel in the pipeline / DAG. The same line buffer unit 101_1 is directed to the sheet generator 103_1 which is in progress. Once the operations by stencil processor 102_1 in the line groups it processes are completed, sheet generator 103_1 sends an output line group to "downstream" line buffer unit 101_2 (in some cases, the output line group). May be sent back to the same line buffer unit 101_1 that previously sent the input line groups).

One or more "consumer" kernels representing the next stage / action in the pipeline / DAG running on its own respective sheet generator and stencil processor (e.g., sheet generator 103_2 and stencil processor 102_2) are down. Image data generated by the first stencil processor 102_1 is received from the stream line buffer unit 101_2. In this way, the "producer" kernel running on the first stencil processor has its output data forwarded to the "consumer" kernel running on the second stencil processor, where the consumer kernel is consistent with the overall pipeline or design of the DAG. Perform the next set of tasks after the producer kernel. Here, the line buffer unit 101_2 stores and forwards the image data generated by the producer kernel as part of the image data transfer from the producer kernel to the consumer kernel.

Stencil processor 102 is designed to operate concurrently on multiple overlapping stencils of image data. The multiple overlapping stencils and internal hardware processing capacity of the stencil processor effectively determine the size of the sheet. Here, within stencil processor 102, arrays of execution lanes operate in unison to simultaneously process the image data surface area covered by multiple overlapping stencils.

As described in more detail below, in various embodiments, sheets of image data are loaded into a two-dimensional register array structure within stencil processor 102. The use of sheets and a two-dimensional register array structure moves large amounts of data into a large register space, for example, since a single load operation with processing operations is performed directly on the data immediately after the execution lane array, thus consuming power. Provide effective improvement. In addition, the use of an execution lane array and a corresponding register array provide different stencil sizes that are easily programmable / configurable.

2A-2E illustrate both the parsing activity of the line buffer unit 101, the more granular parsing activity of the sheet generator unit 103, as well as the stencil processing of the stencil processor 102 coupled to the sheet generator unit 103. The activity is shown in high level embodiments.

2A illustrates an embodiment of an input frame of image data 201. 2A also depicts the outline of three overlapping stencils 202 (each stencil having a dimension of 3 x 3 pixels) designed for the stencil processor to operate. The output pixels where each stencil produces its respective output image data are highlighted in solid black. For brevity, the three overlapping stencils 202 are shown to overlap only in the vertical direction. In practice it is appropriate to recognize that a stencil processor can be designed to have overlapping stencils in both the vertical and horizontal directions.

Because of the vertical overlapping stencils 202 in the stencil processor, as seen in FIG. 2A, there is a wide range of image data in a frame in which a single stencil processor can operate. As described in more detail below, in one embodiment, the stencil processors have data in their overlapping stencil in a left-to-right fashion (and repeating from top to bottom, for the next set of lines) across the image data. To process. Thus, while the stencil processor continues to operate, the number of monochrome black output pixel blocks proceeds to the right in the horizontal direction. As mentioned above, the line buffer unit 101 serves to parse a group of lines of input image data from an incoming frame sufficient for the stencil processor to operate for an extended number of upcoming cycles. An exemplary illustration of a line group is shown by shaded area 203. In one embodiment, as will be described further below, the line buffer unit 101 may include different dynamics for transmitting / receiving line groups to / from the sheet generator. For example, according to one mode called "full group", the full full width lines of the image data are transferred between the line buffer unit and the sheet generator. According to a second mode called "virtual height", the group of lines is initially delivered with a subset of the full width rows. The rest of the rows are passed sequentially in small (smaller than full width) pieces.

By the line group 203 of input image data defined by the line buffer unit and passed to the input generator unit, the sheet generator unit further parses the line group into finer sheets that more accurately fit the hardware limits of the stencil processor. More specifically, as described in more detail below, in one embodiment, each stencil processor is comprised of a two-dimensional shift register array. The two-dimensional shift register array basically shifts the image data “down” in an array of execution lanes, where the pattern of shifting causes each execution lane to operate on data in its own respective stencil (ie, each Execution lanes are processed in its own stencil of information to generate output for that stencil. In one embodiment, the sheets are surface regions of input image data that are "filled" or otherwise loaded into a two-dimensional shift register array.

Thus, as observed in FIG. 2B, the sheet generator parses the initial sheet 204 from the group of lines 203 and provides it to the stencil processor (here, an exemplary sheet of data is generally referred to by reference numeral 204. Corresponding to the 5 × 5 shaded area identified by As observed in FIGS. 2C and 2D, the stencil processor operates on a sheet of input image data by effectively moving the overlapping stencil 202 from left to right on the sheet. As shown in FIG. 2D, the number of pixels (9 in a dark 3 × 3 array) from which the output value can be calculated from the data in the sheet is consumed (other pixel positions may not have an output value determined from the information in the sheet). The border area of the image is ignored for simplicity.

As observed in FIG. 2E, the sheet generator then provides the next sheet 205 for the stencil processor to continue operation. The initial position of the stencils when the stencil begins to operate on the next sheet is the next run from the depletion point on the first sheet to the right (as shown previously in FIG. 2D). With the new sheet 205, when the stencil processor operates on the new sheet in the same manner as the processing of the first sheet, the stencils will simply continue to move to the right.

Note that there is some overlap between the data of the first sheet 204 and the data of the second sheet 205 due to the boundary area of the stencil surrounding the output pixel position. Overlap can be handled simply by the sheet generator retransmitting the overlapping data twice. In another implementation, to supply the next sheet to the stencil processor, the sheet generator proceeds only to send new data to the stencil processor, which can reuse the overlapping data from the previous sheet.

b. Stencil Processor Design and Operation

3A shows an embodiment of a stencil processor unit architecture 300. As observed in FIG. 3A, the stencil processor includes a data computing unit 301, a scalar processor 302, and an associated memory 303 and an I / O unit 304. The data calculation unit 301 includes an array of execution lanes 305, a two-dimensional shift array structure 306 and separate respective random access memories 307 associated with a particular row or column of the array.

The I / O unit 304 loads the "input" sheet of data received from the sheet generator into the data calculation unit 301 and stores the "output" sheet of data from the stencil processor in the sheet generator. In one embodiment, loading the sheet data into the data calculation unit 301 parses the received sheet into rows / columns of the image data and parses the rows / columns of the image data into the two-dimensional shift register structure 306 or the execution lane array. This involves loading into each of the random access memories 307 of the row / column of (described in detail below). Once the sheet is initially loaded into the memories 307, the individual execution lanes in the execution lane array 305 randomly access the sheet data at the appropriate time (e.g., according to the load instruction just before the operation on the data in the sheet). It may load from the memories 307 into the two-dimensional shift register structure 306. When the loading of the data sheet into the register structure 306 (from the sheet generator or directly from the memories 307) is complete, the execution lanes of the execution lane array 305 operate on the data and eventually retrieve the terminated data from the sheet. Either return directly to the sheet generator or "write back" to the random access memories 307. Once the execution lanes are written back to the random access memories 307, the I / O unit 304 fetches data from the random access memories 307 to form an output sheet, which is then forwarded to the sheet generator. do.

The scalar processor 302 includes a program controller 309 that reads instructions of the program code of the stencil processor from the stencil memory 303 and issues instructions to the execution lane of the execution lane array 305. In one embodiment, a single identical instruction is broadcast to all execution lanes in the array 305, affecting single instruction multiple data (SIMD) -like behavior from the data computing unit 301. In one embodiment, the instruction format of instructions read from scalar memory 303 and issued to the execution lanes of execution lane array 305 includes a very long-instruction word (VLIW) that includes one or more operation codes per instruction. Include type format. In another embodiment, the VLIW format is an ALU operation code (as described below, in one embodiment may specify one or more conventional ALU operations) and memory that indicates the mathematical function performed by the ALU of each execution lane. Arithmetic code (indicating a memory operation for a particular execution lane or set of execution lanes).

The term "execution lane" means a set of one or more execution units capable of executing an instruction (eg, a logic circuit capable of executing an instruction). Execution lanes may, in various embodiments, include processor-like functionality beyond just an execution unit. For example, in addition to one or more execution units, the execution lane may also include logic circuitry for decoding received instructions, or logic circuitry for retrieving and decoding instructions for more multi-instruction multiple data (MIMD) -like designs. Can be. Regarding MIMD-like approaches, although a centralized program control approach has been described mostly herein, a more distributed approach can be implemented in various alternative embodiments (eg, each execution lane of array 305). Program code and program controller).

The combination of the execution lane array 305, the program controller 309, and the two-dimensional shift register structure 306 provide a broad adaptable / configurable hardware platform for a wide range of programmable functions. For example, application software developers may consider a wide range of dimensions (e.g., stencil size) as well, given that individual execution lanes can perform a variety of functions and provide easy access to input image data near any output array location. It is possible to program kernels with different functional capabilities.

In addition to acting as a data store for image data operated by execution lane array 305, random access memories 307 may also maintain one or more lookup tables. In various embodiments, one or more scalar lookup tables may also be instantiated in scalar memory 303. Lookup tables are often used by image processing tasks to obtain, for example, filters or transform coefficients for different array positions, and complex functions (eg gamma curves) in which the lookup table provides a function output for input index values, etc. , Sine, cosine). Here, SIMD image processing sequences are often expected to perform lookups with the same lookup table for the same clock cycle. Similarly, one or more constant tables may be stored in scalar memory 303. Here, for example, different execution lanes may require the same constant or different value on the same clock cycle (eg, a specific multiplier applied to the entire image). Thus, access to the constant lookup table returns the same scalar value in each execution lane. Lookup tables are typically accessed using index values.

Scalar look-up involves passing the same data value of the same lookup table from the same index to each execution lane in execution lane array 305. In various embodiments, the aforementioned VLIW instruction format is extended to include scalar operation code that directs a lookup operation performed by a scalar processor to a scalar lookup table. An index specified for use with an opcode can be an immediate operand or retrieved from another data storage location. Nevertheless, in one embodiment, the lookup from the scalar lookup table in the scalar memory involves broadcasting the same data value to all execution lanes in the execution lane array 305 during essentially the same clock cycle. Further details regarding the use and operation of the lookup table are provided below.

3B summarizes the above-described VLIW instruction word embodiment (s). As observed in FIG. 3B, the VLIW instruction word format includes fields for three separate instructions: 1) a scalar instruction 351 executed by a scalar processor; 2) an ALU instruction 352 broadcast and executed in a SIMD manner by respective ALUs in the execution lane array; And 3) a memory instruction 353 broadcast and executed in a partial SIMD manner (e.g., if execution lanes following the same row of an execution lane array share the same random access memory, one execution lane from each of the other rows) This actually executes the instruction (the format of the memory instruction 353 may include an operand that identifies which execution lane from each line executes the instruction).

Also included is a field 354 for one or more immediate operands. Which of the instructions 351, 352, 353 uses which immediate operand information can be identified in the instruction format. Each of the instructions 351, 352, 353 also includes its own respective input operand and result information (eg, local registers for ALU operations and memory addresses for local register and memory access instructions). In one embodiment, the scalar instruction 351 is executed by the scalar processor before the execution lanes in the execution lane array execute one of the other two instructions 352, 353. In other words, the execution of the VLIW word includes a first cycle in which the scalar instruction 351 is executed followed by a second cycle in conjunction with the other instructions 352, 353 (in various embodiments the instructions 352). And 353) may be executed in parallel).

In one embodiment, the scalar instructions executed by the scalar processor 302 are issued to the sheet generator 103 to load / store the sheets to / from memories or the 2D shift register 306 of the data computing unit 301. Contains the command Here, the operation of the sheet generator is a pre-runtime comprehension of the operation of the line buffer unit 101 or the number of cycles it takes for the sheet generator 103 to complete any instruction issued by the scalar processor 302. May depend on other variables that interfere with). As such, in one embodiment, any VLIW word to which the scalar instruction 351 corresponds or otherwise causes the instruction to be issued to the sheet generator 103 is also non-operated in the other two instruction fields 352 and 353. Contains the (NOOP) instructions. The program code then enters a loop of NOOP instructions for the instruction fields 352 and 353 until the sheet generator has finished loading / storing to / from the data computation unit. Here, when issuing an instruction to the sheet generator, the scalar processor may set the bits of the interlock register that the sheet generator resets upon completion of the instruction. During the NOOP loop, the scalar processor monitors the bits of the interlock bits. If the scalar processor detects that the sheet generator has completed the instruction, normal execution resumes.

4 shows an embodiment of a data computing unit 401. As observed in FIG. 4, data calculation unit 401 includes an array 405 of execution lanes logically located “above” of the two-dimensional shift register array structure 406. As mentioned above, in various embodiments, the sheet of image data provided by the sheet generator is loaded into the two-dimensional shift register 406. Execution lanes then operate on the sheet data from register structure 406.

The execution lane array 405 and the shift register structure 406 are fixed at positions relative to each other. However, the data in shift register array 406 is shifted in a strategic and coordinated manner such that each run lane of the run lane array processes another stencil in the data. As such, each execution lane determines an output image value for another pixel in the output sheet that is generated. From the architecture of FIG. 4, it is clear that overlapping stencils are arranged vertically as well as horizontally because the execution lane array 405 includes horizontally adjacent execution lanes as well as vertically adjacent execution lanes.

Some notable architectural configurations of the data calculation unit 401 include a shift register structure 406 having a wider dimension than the execution lane array 405. That is, there is a "halo" of registers 409 outside the execution lane array 405. Although halo 409 is shown as being present on both sides of the execution lane array, halo may be less (one) or more (3 or 4) on the side of the execution lane array 405. . Halo 405 provides a “spill-over” space for data that flows out of the boundary of execution lane array 405 as the data is shifted “down” of execution lane 405. Play a role. As a simple case, a 5x5 stencil centered on the right edge of the execution lane array 405 would require four halo register locations to the right when the leftmost pixel of the stencil is processed. For ease of drawing, FIG. 4 shows, in a nominal embodiment, the registers on the right side of the halo and the vertical shift as having only a horizontal shift connection when the registers on both sides (right, bottom) can have both horizontal and vertical connections. The registers on the bottom side of the halo are shown as having only a connection.

Additional spill-overroom is provided by random access memory 407 coupled to each row and / or each column or portion of the array (e.g., random access memory includes four execution lane rows and two execution lanes). May be assigned to a "region" of the execution lane array that expands to columns). For simplicity, the rest of the application refers primarily to row and / or column based allocation schemes. Here, if kernel operations of an execution lane require processing pixel values outside the two-dimensional shift register array 406 (some image processing routines may require), the plane of the image data is, for example, the halo region 409. ) May further spill over to the random access memory 407. For example, consider a 6 × 6 stencil with hardware having a halo region with only four storage elements to the right of the execution lane at the right edge of the execution lane array. In this case, the data must be shifted further to the right from the right edge of halo 409 to fully process the stencil. Data shifted out of the halo region 409 may spill over to the random access memory 407. Further applications of the random access memories 407 and the stencil processor of FIG. 3 are further provided below.

5A-5K illustrate an embodiment of the manner in which image data is shifted within the two-dimensional shift register array " below " of the execution lane array as described above. As observed in FIG. 5A, the data content of the two-dimensional shift array is shown in the first array 507 and the execution lane array is shown in the frame 505. In addition, two neighboring execution lanes 510 in the execution lane array are shown schematically. In this simple illustration 510, each execution lane may receive data from a shift register, receive data from an ALU output (e.g., operate as an accumulator over a cycle), or write the output data to an output destination. It contains register R1.

In the local register R2, each execution lane can use the content just below the content in the two-dimensional shift array. Thus, R1 is the physical register of the execution lane and R2 is the physical register of the two-dimensional shift register array. The execution lane includes an ALU that can operate on the operands provided by R1 and / or R2. As described in more detail below, in one embodiment, the shift register is actually implemented with multiple ("depth") storage / register elements per array position, but the shifting activity is limited to one plane of storage elements. (Eg, only one plane of storage elements can shift per cycle). 5A-5K illustrate one of these deeper register locations used to store the result X from each of the execution lanes. For ease of explanation, the deeper result register is shown on the side rather than below the corresponding register R2.

5A-5K focus on the calculation of two stencils whose center position is aligned with the pair of execution lane positions 511 shown in the execution lane array 505. For ease of illustration, the pair of execution lanes 510 are actually shown as horizontal neighbors when they become vertical neighbors, according to the following example.

As initially observed in FIG. 5A, execution lanes 511 are centered at their central stencil position. 5B shows the object code executed by both execution lanes 511. As observed in FIG. 11B, the program code of both execution lanes 511 causes the data in the shift register array 507 to be shifted down one position and shifted one position to the right. This aligns both execution lanes 511 to the upper left corner of each stencil. The program code then causes the data located at each location (at R2) to be loaded into R1.

As observed in Fig. 5C, the program code then causes the execution lane pair 511 to unit shift the data in the shift register array 507 to the left, so that the value for the right side of each position of each execution lane is Allow shift to the location of the execution lane. The value of R1 (old value) is added with the new value shifted to position R2 of the execution lane. The result is recorded in R1. As observed in FIG. 5D, the same process as described above with respect to FIG. 5C is repeated so that the resulting R1 contains the A + B + C values in the upper run lanes and the F + G + H values in the lower run lanes. do. At this point, the two execution lanes 511 processed the top row of the respective stencils. A spill-over is written to the halo region (if one exists on the left) of the execution lane array 505 or to the random access memory if there is no halo region to the left of the execution lane array 505.

As observed in FIG. 5E, the program code next shifts the data in the shift register array up one unit, causing the two execution lanes 511 to align with the right edge of the middle row of each stencil. The register R1 of both execution lanes 511 contains the sum of the rightmost values of the upper and middle rows of the current stencil. 5F and 5G show continuing to the left across the middle row of stencils of the two running lanes. Cumulative addition is continued, such that at the end of the process of FIG. 5G both execution lanes 511 include the sum of the values of the top and middle rows of each stencil.

5H shows another shift in aligning each execution lane with the lowest row of the corresponding stencil. 5i and 5j show that we are continuously shifting to complete processing in the stencils of both execution lanes. 5K shows additional shifting to align each execution lane with its exact location in the data array and record the results for it.

In the example of FIGS. 5A-5K, the object code for the shift operation may include an instruction format that identifies the direction and magnitude of the shift expressed in (X, Y) coordinates. For example, the object code for shifting up by one position may be represented by an object code such as SHIFT 0 and +1. As another example, the shift to the right by one position may be represented by an object code such as SHIFT +1, 0. In various embodiments, larger magnitude shifts may also be specified in the object code (eg, SHIFT 0, + 2). Here, if the 2D shift register hardware only supports shift by one position per cycle, the instruction may be interpreted to require multiple cycle executions by the machine or the 2D shift register hardware may be shifted by one or more positions per cycle. Can be designed to support them. The embodiments described below are described in more detail below.

Figure 6 shows another detailed illustration of the unit cell for the array execution lanes and shift register structure (the registers in the halo region do not include corresponding execution lanes). In one embodiment, the register space associated with each position of the execution lanes and the execution lane array is implemented by instantiating the circuits observed in FIG. 6 at each node of the execution lane array. As observed in FIG. 6, the unit cell includes an execution lane 601 connected to a register file 602 composed of four registers R1 to R4. During any cycle, execution lane 601 may read from or write to any registers R0 to R4. For instructions that require two input operands, the execution lane may retrieve all of the operands from any of R0 through R4.

In one embodiment, the two-dimensional shift register structure is such that during a single cycle, the content of any (only one) of registers R1 through R3 is shifted out of one of the neighbor register files via output multiplexer 603. And the neighbors through input multiplexers 604 correspond to if the shift between neighbors is in the same direction (e.g., all execution lanes are shifted to the left, all execution lanes are shifted to the right, etc.). Is implemented by having the content of any (only one) of registers R1 through R3 replaced with content shifted in. It may be common for the same register to shift out its contents and replace it with shifted content in the same cycle, but multiplexer arrangements 603 and 604 may have different shift sources and shifts within the same register file during the same cycle. Allow the target register.

As shown in FIG. 6, the execution lane will shift content from its register file 602 to its left, right, top and bottom neighbors, respectively, during the shift sequence. With the same shift sequence, the execution lane will shift the content from a particular one of the left, right, top and bottom neighbors into its register file. Again, the shift out target and the shift in source must be consistent with the same shift direction for all execution lanes (e.g., shift out to the right neighbor and shift in to the left neighbor).

In one embodiment, only the contents of one register are allowed to be shifted per cycle of execution, while other embodiments may allow the contents of one or more registers to be shifted in / out. For example, if the second instance of the multiplexer circuit 603, 604 observed in FIG. 6 is incorporated in the configuration of FIG. 6, the contents of the two registers may be shifted out / in during the same cycle. Of course, in an embodiment where the contents of only one register is allowed to shift every cycle, by consuming more clock cycles for shifts between mathematical operations, shifts from multiple registers may occur between mathematical operations. (Eg, the contents of two registers may be shifted between mathematical operations by consuming two shift operations between mathematical operations).

If all the contents of the register files of the execution lane are shifted out during the shift sequence, the contents of the unshifted registers of each execution lane remain (do not shift). As such, unshifted content that is not replaced with shift-in content is maintained locally in the execution lane through the shifting cycle. The memory unit ("M") observed in each execution lane is used to load / store data to / from random access memory space associated with the rows and / or columns of the execution lanes in the execution lane array. The M unit here acts as a standard M unit in that it is often used to load / store data that cannot be loaded / stored in / out of its own register space of an execution lane. In various embodiments, the main operation of the M unit is to write data from the local register to memory, read data from the memory, and write to the local register.

In various embodiments, with respect to the instruction set architecture (ISA) opcode supported by the ALU unit of the hardware execution lane 601, the mathematical opcode supported by the hardware ALU is a mathematical opcode supported by the virtual execution lane. (E.g., substantially the same) (e.g. ADD, SUB, MOV, MUL, MAD, ABS, DIV, SHL, SHR, MIN / MAX, SEL, AND, OR, XOR, NOT). As described above, memory access instructions are executed by execution lane 601 to fetch / store data from / to associated random access memory. Hardware execution lane 601 also supports shift operation instructions (right, left, up, down) for shifting data in a two-dimensional shift register structure. As mentioned above, program control instructions are primarily executed by the scalar processor of the stencil processor.

2.0 program code conversion to improve runtime efficiency

As discussed above, application software developed for an image processor may be defined by combining small, granular software programs, referred to herein as kernels, into a larger overall structure, such as directed acyclic graphs. The definition generally includes connecting different kernels to a particular data flow pattern in which multiple "production" kernels provide output image data to one or more "consumer" kernels. At least one kernel receives the full input image on which the application software program runs, and typically one kernel produces the full output image of the application software.

Each kernel is mapped to a specific stencil processor. Each stencil processor has an associated sheet generator that receives image data for the kernel of the associated stencil processor to operate. In various embodiments, the image data is received by the sheet generator in a group of lines. For example, the sheet generator can receive image data as multiple rows over the full width of the input image frame. The sheet generator then forms a two dimensional "sheet" of image data that is provided to the stencil processor and eventually loaded into the two dimensional shift register array of the stencil processor.

In various embodiments, the sheet generator may include dedicated hardware logic circuits (eg, application specific integrated circuit (ASIC) logic circuits), programmable logic circuits (eg, field programmable gate array logic circuits), embedded processor logic circuits, or the like. A combination of implements the functionality of the sheet generator. Dedicated hardware logic circuitry (if any) is associated configuration registers that are set with information generated by the compilation process of the application software that causes the sheet generator to perform sheet generation activities for the kernel of the kernel mapped to the associated stencil processor. Have them. The programmable logic circuitry, if any, is programmed with information generated by the compilation process of the application software that causes the programmable logic circuitry to implement the sheet generator functionality for the kernel running on the stencil processor that was mapped to the sheet generator's associated stencil processor. . Embedded processor circuitry (if any) is generated by a compilation process of application software that, when executed by the embedded processor, causes the embedded processor to implement the sheet generator functionality for the kernel running on the stencil processor that was mapped to the associated stencil processor of the sheet generator. To the generated program code. The scalar processor of the stencil processor may also be programmed to perform, assist or otherwise include various sheet generation activity tasks. The same kind of circuit implementability and associated compiled program code and / or information may also be present in conjunction with the line buffer unit.

The application software development process thus includes not only mapping the kernel to a particular stencil processor, but also the generation of associated configuration information and / or program code used to perform sheet generation activities for the kernel.

In various application software program development environments, a compiler that accepts a high level description of an application software program and in response generates a low level program code (eg, object code) and any associated configuration information for execution by an image processor. It recognizes the various inefficiencies of the software and changes the program code that is compiled to improve or reduce the inefficiency. The program code to be modified may be program code and / or configuration information for one or more sheet generators and / or kernels and / or line buffer units to be supplied by them.

7A relates to a first potential inefficiency. As observed in FIG. 7A, the input image 701 is received by the sheet generator as, for example, a plurality of line groups transmitted by the line buffer unit. As observed in FIG. 7A, the input image is down sampled by the sheet generator before processing by, for example, a kernel K1 running on a stencil processor to which the sheet generator is connected (702). Alternatively, kernel K1 may be programmed to perform down-sampling.

In various embodiments, the stencil processor naturally produces output image sheets having the same dimensions as the execution lane array of the stencil processor. For example, in an embodiment where the execution lane array dimension is 16x16 pixels, the configuration of kernel program code K1 of the stencil processor is initially based on the generation of a 16x16 pixel output image sheet.

Large amounts of buffering space are required if the stencil processor is configured to generate an output sheet of the same dimension as the execution lane array from the downsampled input image. For example, referring to FIG. 7A, if down sampling 702 is performed by a sheet generator to generate a 16 × 16 pixel down-sampled sheet 703 for loading into a two-dimensional shift register array of a stencil processor, The sheet generator needs to queue the entire 32x32 pixel input image 701 to form a 16x16 pixel down-sampled input image 703 for consumption by the kernel K1. Allocating a large amount of memory for this queuing is a form of inefficiency.

As such, in an embodiment, the compiler will reconstruct the application software program as shown in FIG. 7B (eg, including any suitable configuration information). In particular, the compiler organizes the program code so that the kernel K1 does not work as the execution lane array is fully utilized. Continuing with the present example, kernel K1 is designed to operate on 8x8 pixel input sheet 703b which causes kernel K1 to produce an 8x8 pixel output sheet 704b.

By configuring the kernel K1 to operate on the smaller 8x8 pixel input sheet 703b, the down-sampling activity 702b (eg, performed by the sheet generator) is halved relative to the input image data 701A of FIG. 7A. It is necessary to queue positive input image data 701b. Here, the input image data 701a of FIG. 1A corresponds to 32 rows of image data, whereas the input image data 701b of FIG. 7B corresponds to only 16 rows of input image data. With only 16 rows of input image data 701b, downsampling activity 702b performs a 2: 1 downsampling that produces a series of 8x8 pixel input sheets 703b that spans the full width of the image. can do.

8A and 8B show another inefficiency in which downsampling is performed by the same amount after upsampling is performed on the output image data 801 of the kernel K1 but before being executed on the image data by the consuming kernel K2 of K1. Here, as observed in FIG. 8A, production kernel K1 produces a series 801 of output sheets A0-A3. The image data of these output sheets 801 are then interleaved to effectively upsample the output of K1. That is, as shown in FIG. 8A, for example, the upper line of each of the output sheets A0 to A3 is a line buffer 802 that temporarily queues the output data of K1 before being consumed by the consuming kernel K2 of K1. Interleaved to form an upper output line of the upsampled Kl output 803 stored at. In various embodiments, upsampling may be performed by one of K1, a sheet generator connected to a stencil processor executed by K1, or a line 802 buffer where K1 sends its output.

As observed in FIG. 8B, input processing for kernel K2 consuming an output of K1 is configured to downsample its input by the same factor as the output of K1 is upsampled. Thus, the process of supplying K2 with appropriately sized input data must reverse the upsampling process performed on the output of K1. That is, referring to FIG. 8B, interleaved queued data 803 in line buffer 802 is ultimately de-interleaved to reconstruct output images A0-A3 originally formed by K1. Downsampling may be performed by line buffer 802, by a sheet generator connected to the stencil processor that K2 executes, or by K2 itself.

In one embodiment, the compiler is responsible for the same factor (eg, 1: 2 upsample and 2: 1) for the kernel (which may include multiple kernels) where the upsampled output of the production kernel will consume the output of the production kernel. And downsampled by a downsample. In response, the compiler will reconstruct the program code under development to eliminate both producers and upsampling and downsampling on the consumer data path. This solution is shown in FIG. 8C. Here, the non-upsampled output of K1 is simply queued in line buffer 802 coupled between K1 and K2 connections. The non-upsampled K1 output is then fed directly to K2 without downsampling. As such, both the upsampling activity of FIG. 8A and the downsampling activity of FIG. 8B can be avoided.

9A and 9B relate to another inefficiency that may occur, for example in the case of a multi-component output image. As is known, a digital image can have multiple components (eg, RGB, YUV, etc.). Various application software programs may be designed / configured to process different components into different data planes. Here, for example, the complete output image 901 generates one or more data sheets composed solely of the first component R, one or more data sheets composed solely of the second component R, and the third component B By generating one or more data sheets of data consisting solely of). In various embodiments, it may be natural or standard default to queue all data of the image passed between the production kernel and the consuming kernel in the same line buffer 902. Thus, FIG. 9A shows image data of all three components 901 queued to the same line buffer unit 902.

However, for example, for large output images, storing the image data of all three components in the same line buffer unit can otherwise consume a large amount of line buffer memory resources. Thus, in one embodiment, referring to FIG. 9B, a compiler that compiles an application software program will automatically recognize when storing different components of a multi-component image strains line buffer memory resources. For example, the compiler may initially allocate a fixed amount of buffer memory resources for storing and forwarding an image, or may allocate an amount of buffer memory resources that correlates to the size and / or amount of data to be transferred, In terms of, it can be determined that the automatically allocated amount is insufficient or the maximum threshold has been reached. In other approaches, the compilation process may include simulating an application software program and recognizing that the line buffer unit is the bottleneck (e.g., often not having memory space to store line groups generated by the production kernel). Or does not have bandwidth to respond to read requests from the consuming kernel). In response, the compilation process automatically modifies the application software and / or reconfigures the image processor so that different components of the output image data of the production K1 kernel are queued in different line buffer units. Here, FIG. 9B shows R, G, and B image data queued to different line buffer units 902_1, 902_2, and 902_3, respectively.

The solution of FIG. 9B can be used even when the production K1 kernel has many consumers. In this case, if the default solution of FIG. 9A is adopted, all components of the image data 901 will be required because multiple consumers will need multiple loading / readings from the line buffer to receive all the information for a single input image. A single line buffer unit that stores the system can be a bottleneck. Therefore, in one embodiment, the approach of FIG. 9B is adopted where each line buffer only holds data for the same component type. In the example discussed, this would reduce the consumer's read request for a single line buffer resource by 66% compared to the default approach of FIG. 9A. That is, each of the line buffer units 902_1, 902_2, 902_3 in FIG. 9B will only need to support 33% of the consuming read load of the line buffer unit 902 in FIG. 9A. The same reduced demand impact also occurs for activities in which image data from the production kernel is written to the line buffer resource.

Another situation where the approach of FIG. 9B can reduce inefficiency is when a particular consumer consumes only a subset of components. In extreme cases, for example, one consumer consumes an R component, another consumer consumes a G component, and another consumer consumes a G component. In this case, each different consumer is configured with its own dedicated line buffer source, simplifying different component based data flows along different data paths (via different line buffer unit connections). In contrast, if the approach of FIG. 9A is used, different component-based data flows will converge in a single point of line buffer 902 (FIG. 9A), in which case the data flow of one component forwards the other components. There will be a delay due to the large amount of read and write activity at 901.

10A and 10B show another efficiency improvement based on the spread of line buffer resources downstream from a single consumer. Here, if too many consumers exist, it may be necessary to use multiple line buffer units to forward the output image data of a single production kernel. 10A shows the potential inefficiency when four different consumers K2 to K5 consume the output of a single production K1 kernel from a single line buffer unit 1002. Also, the single line buffer unit 1002 can be a bottleneck because it cannot remove the queued data until all consumers have consumed it. In this case, the entire data flow from the line buffer unit 1002 will be reduced to at least the slowest consumer's input speed. In addition, the line buffer unit 1002 will be subjected to a large load of read requests, given the large number of supporting consumers that can overwhelm the resources of the line buffer unit 1002.

As such, as shown in FIG. 10B, the first subset of consumers K2, K3 is allocated to the first line buffer unit 1002_1, and the second subset of consumers K4, K5 is assigned to the second line buffer unit 1002_2. Is assigned. The output image stream of the production kernel K1 is supplied to both line buffer units 1002_1 and 1002_2. Distributing the overall consumer load among multiple line buffer unit resources 1002_2, 1002_2 (compared to the approach of FIG. 10A) reduces the total demand for any particular line buffer unit resource. In addition, the compiler can supply a faster input stream consuming kernel to the same line buffer unit (and / or a slower input stream consuming kernel to another line buffer unit), so that faster consuming kernels It can be prevented from being delayed by the speed of consumption.

FIG. 11A illustrates the " splitting and combining " inefficiencies that may arise from an application software program (or components thereof) designed with DAG. As observed in FIG. 11A, the output of source kernel K1 is fed to two different consuming kernels K2 and K3. In addition, kernel K3 consumes the output of kernel K2. Double dependency of kernel K3 from the output of kernel K1 can lead to run-time computational inefficiencies and modeling / design inefficiencies. With regard to runtime inefficiency, the LB2 line buffer 1102_2 may have to be made very large to queue large amounts of output data of K1. In general, kernel K3 does not request the next line group from LB2 1102_2 until approximately the next line group from LB3 1002_3 that kernel K3 will process is available with the next line group from LB2 1102_3. . Due to the large propagation delay through K2, LB2 1102_2 can be very large. The discrepancy between when data in LB2 1102_2 is ready to be consumed and when its sibling input data from kernel K2 to kernel K3 is available in LB3 1102_3 also causes a modeling or optimization process to occur during the design of the application software. It can be more difficult.

Figure 11B illustrates a solution for the compiler to force the pipeline structure on the partitioning and concatenation structure. Here, the K2 kernel of FIG. 11A is extended to another kernel K2 'which includes the original K2 kernel and a load / store algorithm 1103 that simply consumes content from LB1 1102_1 and forwards it to LB4 1102_4. Importantly, the load / store algorithm 1103 is prepared when the original output data from K1 is ready to be consumed by K3 and when the output data from K2 is ready to be consumed by K3 at LB3 1102_3. It can lead to some propagation delays in the unprocessed stream from K1, which eliminates discrepancies between them.

In various embodiments, it can be seen from the description of FIG. 3A that the scalar memory 303 can be configured to maintain a lookup table or a constant table. In certain applications, the input image data processed by the kernel is a fixed constant rather than variable information (eg, as generated by a source kernel operating on various input data). For example, lens shading correction, in which the correction value of the lens is recorded for a significantly large particle size region of the lens surface. Significantly large particle sizes correspond to low resolution image data (when the recorded data is implemented with different items where each item corresponds to a different particle, the recorded data does not contain many items).

When the image processor processes an image from a camera that includes a lens, one of these recorded correction values corresponds to the image area processed by the execution lane array. The recorded value is therefore applied as an input value to each run lane. In this sense, the lens correction value is implemented similarly to a lookup table or a constant table. In addition, since the total amount of data required to implement the modification value is limited, the modification value does not consume a large amount of memory space. As such, as observed in FIG. 12, in various embodiments, the input image data 1210 that is fixed and small enough to fit the scalar memory 1203 is a scalar (eg, as an initial configuration of application software). It is loaded into memory 1203 and referenced as a lookup table or constant table by the kernel, which is executed in the run lane array of the stencil processor during runtime (not generated by the source kernel and supplied to the kernel through the line buffer unit).

13A illustrates another runtime issue that can potentially result in a large amount of data queued to the line buffer unit and / or sheet generator. Here, FIG. 13A shows three line groups 1301, 1302, 1303, for example queued to a sheet generator after being provided from a line buffer unit. For example, assume that each line group 1301, 1302, 1303 contains 16 rows of image data, and that the dimension 1305 of the execution lane array of the corresponding stencil processor of the sheet generator is also 16x16 pixels. Further, the dimension 1306 of the two-dimensional shift register array is assumed to be 24x24 pixels to support a halo region that forms a four pixel wide border around the periphery of the execution lane array. At least under these circumstances, the natural configuration may be to align 16 rows of execution lane array 1305 with 16 rows of a particular line group. That is, the sheet generator forms a sheet about a specific group of lines. FIG. 13A illustrates this approach in which the execution lane 1305 is aligned to operate with respect to the height of the second line group 1302.

The problem is that, as shown in FIG. 13A, due to the presence of the halo 1306, the complete sheet fed to the two-dimensional shift register array has the lower region of the first line group 1301 and the third line group 1303. You will need data from the top region (the halo region also covers these line groups). As such, in one embodiment, as shown in FIG. 13B, the alignment is altered such that a minimum group of lines need to be present to form a full size sheet. In this example, the alignment of FIG. 13B is shifted up by four pixel values compared to the alignment of FIG. 13A, such that only two line groups 1301 and 1302 need to be present in the sheet generator to form a full size sheet. . This not only reduces the memory space required for the sheet generator (and potentially the line buffer), but the sheet only waits for two groups of lines instead of waiting for three line buffers to begin processing.

14 illustrates a sheet generator as input image data including a plurality of pixels per data lane or, alternatively, as the input process for a kernel supplied with a base unit of data to be processed by the kernel of the sheet generator includes a plurality of pixels. A de-interleaving process performed by the present invention. As an example, FIG. 14 shows pixels of an input image received by a sheet generator structured to include a mosaic 1401 of different color pixels, for example in a Bayer pattern format. Here, the input image is received by the sheet generator as a group of lines provided by the line buffer unit. As such, for example, each row of each line group received by the sheet generator includes R, G, and B pixels. Here, the base unit of the input image data includes a unit cell 1402 of four pixels including an R pixel, a B pixel, and two G pixels.

The sheet generator does not simply parse the sheet directly from the received input image structure 1401 (will generate a sheet with a via pattern), but instead performs a deinterleaving process on the input image data structure 1401, Create a new input structure 1403 for the kernel containing four types of sheets. That is, as observed in FIG. 14, the new input structure 1403 comprises: 1) sheets composed only of R pixels of the input image or otherwise derived only from R pixels; 2) sheets composed only of G pixels or otherwise derived only from G pixels located at the same first location of the unit cell of the unit cell input image; 3) sheets composed only of G pixels or otherwise derived only from G pixels located at the same second position of the unit cell of the unit cell input image; 4) Include sheets that consist only of B pixels of the input image or otherwise derive only from B pixels. The sheet may consist only of input image pixels or may be up-sampled, for example, by interpolating values to input image locations where different colors are located.

The newly structured sheets are then provided to the associated kernel of the sheet generator that processes them, producing an output sheet of the same structure 1403 (one color per sheet) provided back from the sheet generator. The sheet generator performs an interleaving process on the monochrome structure 1403 to generate an output image for consumption having the original structure 1401 including unit cells of mixed color.

In various embodiments, the above-described line buffer or line buffer units may be more generally characterized as a buffer for storing and forwarding image data between production and consuming kernels. That is, in various embodiments, the buffer does not necessarily have to queue line groups. Additionally, the hardware platform of the image processor may include a plurality of line buffer units having associated memory resources, and one or more line buffers may be configured to operate from a single line buffer unit. That is, a single line buffer unit of hardware may be configured to store and forward different image data flows between different production / consumer kernel pairs.

15 illustrates the method described above. The method includes constructing 1501 an image processing software data flow for storing and forwarding image data from which a buffer is sent from a production kernel to one or more consuming kernels. The method includes a step 1502 of recognizing that resources are insufficient for the buffer to store and forward the image data. The method also includes modifying the image processing software data flow to include a plurality of buffers for storing and forwarding the image data while transferring the image data from the production kernel to the one or more consuming kernels. do.

3.0 Low Level Program Code Structure

16 illustrates that a programmer may design a high level image processing function and the application development environment may provide any or all of the aforementioned transformations of section 2.0 such that the developer needs to identify inefficiencies and / or write the transformations from scratch. A pre-runtime development environment is shown.

Here, the development environment automatically recognizes the inefficiency described above, for example, a description of the inefficiency (scan the program code being developed for inclusion in the development environment) and the corresponding modifications (applied if inefficiency is found). By referencing a library 1601 that includes a to automatically impose a corresponding modification improvement. That is, the development environment may automatically insert program code from the library 1601 to perform a more efficient process (eg, as part of the compilation process) or otherwise modify the inefficient code to include a fix for the inefficiency. Modify the program code to replace it with new code.

Thus, program code that performs the above-described operations or alternative embodiments may be represented as a high level program code or a low level object code. In various embodiments, a higher level virtual instruction set architecture (ISA) code may specify data values to be operated upon reading a memory having x, y address coordinates, and the object code may replace these data accesses with 2 instead. It can be understood as dimensional shift register operations (such as any of the shift operations described above or similar embodiments).

The compiler can convert the x, y readings in the development environment into the corresponding shifts of the two-dimensional shift register, which is the specified object code (e.g., in a development environment with x, y coordinates (+2, +2) Implemented by two shifts left and two shifts down in the object code).

Depending on the environment, developers may have visibility into both of these levels (for example, only high virtual ISA levels). In another embodiment, such prewritten routines may be called during runtime rather than pre-runtime (eg, by a just-in-time compiler).

4.0 Conclusion

The previous section should recognize that an image processor such as that described in Section 1.0 can be implemented in the hardware of a computer system (e.g. System On Chip of a handheld device processing data from the camera of the handheld device). Chip, as part of SOC).

It is appropriate to point out that the various image processor architecture configurations described above are not necessarily limited to image processing in the conventional sense, and thus may be applied to other applications that may (or may not) be recharacterized. . For example, if any of the various image processor architecture configurations described above are used in the creation and / or generation and / or rendering of the animation as opposed to the processing of the actual camera image, the image processor may be characterized as a graphics processing unit. Can be. In addition, the image processor architecture configurations described above may be applied to other technical applications such as video processing, vision processing, image recognition and / or machine learning. Applied in this manner, an image processor may be integrated with a more general purpose processor (eg, a CPU or part of a computing system) (eg, as a co-processor), or may be a standalone computer system within a computing system. Can be.

The hardware design embodiments described above may be implemented as descriptions of circuit designs for final targeting within semiconductor chips and / or towards semiconductor fabrication processes. In the latter case, such circuit descriptions (eg, VHDL or Verilog) may take the form of register transfer level (RTL) circuit descriptions, gate level circuit descriptions, transistor level circuit descriptions or mask descriptions or various combinations thereof. Circuit descriptions are generally implemented in computer readable storage media (CD-ROM or other type of storage technology).

The previous section should recognize that an image processor such as that described above may be implemented in the hardware of a computer system (e.g., a system on chip of a handheld device that processes data from a camera of the handheld device). As part of SOC). It should be noted that if the image processor is implemented as hardware circuitry, the image data processed by the image processor may be received directly from the camera. Here, the image processor may be part of an individual camera or part of a computing system with an integrated camera. The image data may later be received directly from the camera or from the system memory of the computing system (eg, the camera sends the image data to system memory rather than to an image processor). In addition, many of the configurations described in the previous section can be applied to graphics processing units (animation rendering).

17 provides an example diagram of a computing system. Many components of the computing system described below are applicable to computing systems with integrated cameras and associated image processors (eg, handheld devices such as smartphones or tablet computers). Those skilled in the art will readily be able to distinguish between the two.

As observed in FIG. 17, the basic computing system includes a central processing unit 1701 (eg, a plurality of general purpose processing cores 1715_1 through 1715_N and a main memory controller 1725 disposed in a multicore processor or application processor). , System memory 1702, display 1703 (e.g., touchscreen, flat panel), local wired point-to-point link (e.g., USB) interface 1704, various network I / O functions (1706). ) (E.g. Ethernet interface and / or cellular modem subsystem), wireless local area network (e.g. WiFi) interface 1706, wireless point-to-point link (e.g. Bluetooth) interface 1707 and GPS interface 1708, various sensors 1709_1 to 1709_N, one or more cameras 1710, a battery 1711, a power management control unit 1724, a speaker and a microphone 1713, and an audio coder / decoder 1714.

The application processor or multicore processor 1750 may include one or more general purpose processing cores 1715, one or more graphics processing units 1716, a memory management function 1725 (eg, a memory controller) within its CPU 1701, I / O control function 1718 and image processing unit 1725. General purpose processing core 1715 generally executes the operating system and application software of the computing system. Graphics processing units 1716 generally execute graphics intensive functions to generate, for example, graphical information presented on display 1703. The memory control function 1917 interfaces with the system memory 1702 to write / read data to / from the system memory 1702. The power management control unit 1724 generally controls the power consumption of the system 1700.

Image processing unit 1719 may be implemented in accordance with any of the image processing unit embodiments described above in the previous section. Alternatively or in combination, the IPU 1719 may be connected as a co-processor to one or both of the GPU 1716 and the CPU 1701. Additionally, in various embodiments, GPU 1716 may be implemented with any of the image processor configurations described above.

Each of the touch screen display 1703, the communication interface 1704-1707, the GPS interface 1708, the sensor 1709, the camera 1710, and the speaker / microphone codec 1713, 1714 are each integrated peripheral devices ( For example, it may be viewed as various forms of I / O (input and / or output) associated with the overall computing system that also include one or more cameras 1710. Depending on the implementation, various of these I / O components may be integrated on the application processor / multicore processor 1750 or may be located outside the die or outside the package of the application processor / multicore processor 1750. have.

In one embodiment, one or more cameras 1710 include a depth camera capable of measuring the depth between the camera and the object in its field of view. Application software, operating system software, device driver software, and / or firmware that executes on an application processor or on a general purpose CPU core (or other functional block having an instruction execution pipeline for executing program code) may include the functions described above. Any of may be performed. Here, many components of the computing system of FIG. 17 are implemented in a high performance computing system (eg, a server) that executes program code corresponding to the application development environment of FIG. 16 including a compiler that performs any / all transformations described above. Can be presented.

Embodiments of the present invention may include various processes as described above. Processes may be implemented in machine executable instructions. The instructions may be used to cause a general purpose or special purpose processor to perform certain processes. Alternatively, these processes may be performed by specific hardware components including wired logic to perform the processes, or by any combination of programmed computer components and custom hardware components.

Elements of the invention may also be provided as a machine readable medium for storing machine executable instructions. Machine-readable media may be floppy diskettes, optical disks, CD-ROMs and magneto-optical disks, flash memory, ROM, RAM, EPROM, EEPROM, magnetic or optical cards, propagation media or other types of media suitable for storing electronic instructions / Including but not limited to machine readable media. For example, elements are computer programs transmitted from a remote computer (such as a server) to a requesting computer (such as a client) by means of data signals contained on a carrier or other propagation medium over a communication link (such as a modem or network connection). Can be downloaded as.

In the foregoing specification, specific exemplary embodiments have been described. However, it will be apparent that various modifications and changes may be made without departing from the broader spirit and scope of the invention described in the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Some examples are described below.

Example 1: A machine-readable storage medium containing program code for causing the computing system to perform a method when processed by a computing system, the method comprising:

a) constructing an image processing software data flow in which a buffer stores and forwards image data sent from the production kernel to one or more consuming kernels;

b) recognizing that resources are insufficient for the buffer to store and forward the image data; And

c) modifying the image processing software data flow to include a plurality of buffers for storing and forwarding the image data while transferring the image data from the production kernel to the one or more consuming kernels. Computing system.

Example 2: The method of Example 1, wherein the method further comprises modifying the image processing software data flow such that different portions of the image data are sent from the production kernel to different ones of the plurality of buffers. Machine-readable storage media.

Example 3: The machine readable storage medium of example 2, wherein the different portions correspond to different color components of the image data.

Example 4: The method of at least one of the preceding examples, wherein the method modifies the image processing software data flow such that the same image data is sent from the production kernel to first and second buffers of the plurality of buffers. Machine readable storage medium further comprises.

Example 5: The method of example 4, wherein the modifying comprises: supplying a first one of the plurality of buffers to a first one of the one or more consuming kernels and a second of the plurality of buffers And supplying a buffer to a second one of the one or more consuming kernels.

Example 6: The method of at least one of the preceding examples, wherein the method comprises:

Recognizing that the image processing software data flow is upsampling an image by a factor to form an upsampled image;

Recognizing that the image processing software data flow is downsampling the upsampled image by the factor;

And removing the upsampling of the image and the downsampling of the upsampled image from the image processing software data flow.

Example 7: The method of at least one of the preceding examples, wherein the method comprises:

Recognizing a partition-and-combination pattern in the image processing software data flow; And

And reconstructing the image processing software data flow as a series of stages to remove the split-and-combination pattern.

Example 8: At least one of the preceding examples, characterized in that it is configured to operate in an architecture having a plurality of line buffer units interconnected with a plurality of stencil processor units and / or at least one corresponding sheet generator units. Machine-readable storage medium.

 Example 9: The machine-readable storage medium of at least one of the preceding examples, configured to process stencils, in particular overlapping stencils.

Example 10: The at least one of the preceding examples, configured to operate in a data computing unit that includes a shift register structure having a wider dimension than the execution lane array, in particular the registers being outside the execution lane array. And a machine readable storage medium.

Example 11: A machine readable storage medium comprising program code for causing the computing system to perform a method when processed by a computing system, the method comprising:

 Recognizing that an image to be processed by a kernel of program code will be downsampled, the kernel of program code will be executed at an image processing core of an image processor, the image processing core comprising a two-dimensional execution lane array;

Structuring the kernel to process the image to process the image with fewer execution lanes than all of the execution lanes of the execution lane array to reduce consumption of memory resources used to support downsampling of the image. Machine-readable storage medium, characterized in that.

Example 12: The apparatus of example 11, wherein the image processor comprises a plurality of image processing cores including the image processing core, and one of the plurality of image processing cores is associated with the constant information for storing constant information. Memory, the method being:

Configuring the associated memory for storing a constant input image, wherein the constant input image is processed by each kernel of program code executed in one of the plurality of image processing cores having the associated memory. And a machine readable storage medium.

Example 13: The image processor of examples 11 or 12, wherein the image processor comprises a plurality of image processing cores including the image processing core, and input image data for one of the plurality of image processing cores is a line of image data. Received as groups of, the method:

And aligning an input image region in which said one image processing core is to operate so that said input image region overlaps with a minimum number of said groups of lines.

Example 14: The at least one of examples 11-13, wherein the image processor comprises a plurality of image processing cores including the image processing core, and input image data for one of the plurality of image processing cores is One image processing core comprises a mosaic of multiple pixels per unit of data to operate, the method comprising:

Configuring the input image data to be de-interleave before being processed by the one image processing core;

And configuring the output image data generated by the one image processing core to interleave into a mosaic of a plurality of pixels per unit of data.

Example 15: The at least one of examples 11-14, configured to operate in an architecture having a plurality of line buffer units interconnected with a plurality of stencil processor units and / or at least one corresponding sheet generator units. Machine-readable storage medium.

Example 16: The machine-readable storage medium of at least one of Examples 11-15, configured to process stencils, in particular overlapping stencils.

Example 17: The at least one of examples 11-16, configured to operate in a data computing unit that includes a shift register structure having a wider dimension than the execution lane array, in particular registers outside the execution lane array. And a machine readable storage medium.

Example 18: As a system,

One or more general purpose processing cores;

System memory;

A memory controller coupled between the system memory and the one or more general purpose processing cores;

A storage medium containing program code for causing the computing system to perform a method when processed by the computing system, the method comprising:

a) constructing an image processing software data flow in which a buffer stores and forwards image data sent from the production kernel to one or more consuming kernels;

b) recognizing that resources are insufficient for the buffer to store and forward the image data; And

c) modifying the image processing software data flow to include a plurality of buffers for storing and forwarding the image data while transferring the image data from the production kernel to the one or more consuming kernels. Computing system.

Example 19: The method of example 18, wherein the method further comprises modifying the image processing software data flow such that different portions of the image data are sent from the production kernel to different ones of the plurality of buffers. Computing system.

Example 20: The computing system of example 19, wherein the different portions correspond to different color components of the image data.

Example 21: The method of at least one of examples 18-20, wherein the method modifies the image processing software data flow such that the same image data is sent from the production kernel to first and second buffers of the plurality of buffers. Computing system further comprising the step.

Example 22: The method of at least one of examples 18-21, wherein the modifying comprises: configuring a first one of the plurality of buffers to supply a first one of the one or more consuming kernels; And supplying a second one of the buffers of to a second one of the one or more consuming kernels.

Example 23: The method of at least one of Examples 18-22, wherein the compiling comprises:

Recognizing that the image processing software data flow is upsampling an image by a factor to form an upsampled image;

Recognizing that the image processing software data flow is downsampling the upsampled image by the factor;

And removing the upsampling of the image and the downsampling of the upsampled image from the image processing software data flow.

Example 24: The method of at least one of Examples 18 to 23, wherein the compiling:

Recognizing a partition-and-combination pattern in the image processing software data flow; And

And reconstructing the image processing software data flow as a series of stages to remove the split-and-combination pattern.

Example 25: The computing system of at least one of examples 18-24, wherein the buffer stores forward groups of lines of the image data.

Example 26: The at least one of examples 18-25, wherein the buffer is implemented in a line buffer unit of the image processor, wherein the image processor comprises the line buffer unit, a production kernel of the program code, and at least one consumption of the program code. A computing system comprising a network coupled between a plurality of processing cores each executing kernels.

Example 27: The at least one of Examples 18-26, comprising a processor having an architecture having a plurality of line buffer units interconnected with a plurality of stencil processor units and / or at least one corresponding sheet generator units Computing system.

Example 28: The computing system of at least one of Examples 18-27, configured to process stencils, in particular overlapping stencils.

Example 29: The at least one of examples 18-28, comprising a data computation unit comprising a shift register structure having a dimension wider than the execution lane array, in particular the registers being outside the execution lane array. Characterized by a computing system.

Claims (29)

A machine-readable storage medium comprising program code for causing the computing system to perform a method when processed by a computing system, the method comprising:
a) constructing an image processing software data flow in which a buffer stores and forwards image data sent from the production kernel to one or more consuming kernels;
b) recognizing that resources are insufficient for the buffer to store and forward the image data; And
c) modifying the image processing software data flow to include a plurality of buffers for storing and forwarding the image data while transferring the image data from the production kernel to the one or more consuming kernels. Machine-readable storage medium. The machine of claim 1, wherein the method further comprises modifying the image processing software data flow such that different portions of the image data are sent from the production kernel to different ones of the plurality of buffers. Readable storage medium. The machine-readable storage medium of claim 2, wherein the different portions correspond to different color components of the image data. The method of claim 1, wherein the method further comprises modifying the image processing software data flow such that the same image data is sent from the production kernel to first and second buffers of the plurality of buffers. And a machine readable storage medium. The method of claim 4, wherein the modifying comprises: configuring a first one of the plurality of buffers to supply a first one of the one or more consuming kernels; And configuring to supply a second one of the one or more consuming kernels. The method according to at least one of the preceding claims, wherein
Recognizing that the image processing software data flow is upsampling an image by a factor to form an upsampled image;
Recognizing that the image processing software data flow is downsampling the upsampled image by the factor;
Removing the upsampling of the image and downsampling of the upsampled image from the image processing software data flow. The method according to at least one of the preceding claims, wherein
Recognizing a partition-and-combination pattern in the image processing software data flow; And
And reconstructing said image processing software data flow as a series of stages to remove said partitioning-and-combining pattern. Machine readable according to at least one of the preceding claims, configured to operate in an architecture having a plurality of line buffer units interconnected with a plurality of stencil processor units and / or at least one corresponding sheet generator units. Storage media. Machine-readable storage medium according to at least one of the preceding claims, configured to process stencils, in particular overlapping stencils. The at least one of the preceding claims, characterized in that it is configured to operate in a data computation unit comprising a shift register structure having a wider dimension than the execution lane array, in particular the registers being outside the execution lane array. Machine-readable storage media. A machine-readable storage medium comprising program code for causing the computing system to perform a method when processed by a computing system, the method comprising:
Recognizing that an image to be processed by a kernel of program code will be downsampled, the kernel of program code will be executed at an image processing core of an image processor, the image processing core comprising a two-dimensional execution lane array;
Structuring the kernel to process the image to process the image with fewer execution lanes than all of the execution lanes of the execution lane array to reduce consumption of memory resources used to support downsampling of the image. Machine-readable storage medium, characterized in that. The system of claim 11, wherein the image processor includes a plurality of image processing cores including the image processing core, and one of the plurality of image processing cores has an associated memory for storing constant information. The method is as follows:
Configuring the associated memory for storing a constant input image, wherein the constant input image is processed by each kernel of program code executed in one of the plurality of image processing cores having the associated memory. And a machine readable storage medium. 13. The method of claim 11 or 12, wherein the image processor comprises a plurality of image processing cores including the image processing core, and input image data for one of the plurality of image processing cores as groups of lines of image data. Received, the method being:
And aligning an input image region in which said one image processing core is to operate so that said input image region overlaps with a minimum number of said groups of lines. 14. The method of claim 11, wherein the image processor comprises a plurality of image processing cores including the image processing core, and input image data for one of the plurality of image processing cores is selected from the The image processing core comprises a mosaic of a plurality of pixels per unit of data to operate, the method comprising:
Configuring the input image data to be de-interleave before being processed by the one image processing core;
And configuring the output image data generated by the one image processing core to interleave into a mosaic of a plurality of pixels per unit of data. The machine read of claim 11, configured to operate in an architecture having a plurality of line buffer units interconnected with a plurality of stencil processor units and / or at least one corresponding sheet generator units. Possible storage medium. The machine-readable storage medium of claim 11, configured to process stencils, in particular overlapping stencils. The at least one of the preceding claims, characterized in that it is configured to operate in a data computation unit comprising a shift register structure having a wider dimension than the execution lane array, in particular the registers being outside the execution lane array. Machine-readable storage media. As a computing system,
One or more general purpose processing cores;
System memory;
A memory controller coupled between the system memory and the one or more general purpose processing cores;
A storage medium containing program code for causing the computing system to perform a method when processed by the computing system, the method comprising:
a) constructing an image processing software data flow in which a buffer stores and forwards image data sent from the production kernel to one or more consuming kernels;
b) recognizing that resources are insufficient for the buffer to store and forward the image data; And
c) modifying the image processing software data flow to include a plurality of buffers for storing and forwarding the image data while transferring the image data from the production kernel to the one or more consuming kernels. Computing system. 19. The computing method of claim 18, further comprising modifying the image processing software data flow such that different portions of the image data are sent from the production kernel to different ones of the plurality of buffers. system. 20. The computing system of claim 19, wherein said different portions correspond to different color components of said image data. The method of claim 18, wherein the method further comprises modifying the image processing software data flow such that the same image data is sent from the production kernel to first and second buffers of the plurality of buffers. Computing system comprising a. 22. The method of claim 18, wherein the modifying comprises: configuring a first one of the plurality of buffers to supply a first one of the one or more consuming kernels and the plurality of buffers. And supplying a second buffer to a second consuming kernel of the one or more consuming kernels. The method of claim 18, wherein the compiling comprises:
Recognizing that the image processing software data flow is upsampling an image by a factor to form an upsampled image;
Recognizing that the image processing software data flow is downsampling the upsampled image by the factor;
And removing the upsampling of the image and the downsampling of the upsampled image from the image processing software data flow. 24. The method of claim 18, wherein the compiling comprises:
Recognizing a partition-and-combination pattern in the image processing software data flow; And
And reconstructing the image processing software data flow as a series of stages to remove the split-and-combination pattern. 25. The computing system of claim 18, wherein the buffer stores forward groups of lines of image data. 26. The system of claim 18, wherein the buffer is implemented in a line buffer unit of the image processor, the image processor configured to produce the line buffer unit, the production kernel of the program code, and the one or more consuming kernels of the program code. A computing system comprising a network coupled between a plurality of processing cores each executing. 27. The computing device of any one of claims 18 to 26, comprising a processor having an architecture having a plurality of line buffer units interconnected with a plurality of stencil processor units and / or at least one corresponding sheet generator units. system. 28. The computing system of any one of claims 18 to 27, configured to process stencils, in particular overlapping stencils. 29. A data processing unit as claimed in any one of claims 18 to 28, comprising a data calculation unit comprising a shift register structure having dimensions wider than the execution lane array, in particular the registers being outside the execution lane array. Computing system.

Images