JP2010237936A - Image processor, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for achieving the efficiency of image processing by restricting the maximum value of the number of difference rectangles in the case of distributed processing. <P>SOLUTION: An image processor 1 is provided with a CPU 10 and a co-processor 20 having a plurality of cores to be operated in parallel, and the CPU 10 divides one screen into a plurality of regions, and those cores are assigned to each of those regions one by one, and as for difference rectangles extracted by the processing of the cores, the difference rectangles extracted in the adjacent regions are merged following a prescribed rule, so that the number of difference rectangles on a whole screen is restricted. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to a technique for improving the efficiency of image processing by merging detected difference rectangles.

  When a home PC is remotely operated from another device at a remote location, an image to be displayed on the screen of the home PC may be desired to be reproduced on the screen of another device at a remote location.

  For example, when starting to display the screen of the home PC on the screen of another device at a remote location, the information of the full screen is transmitted, but after that, only the changed part of the screen of the home PC is transmitted to the remote location. If another device at a remote location draws only the changed portion, the data transmission amount and the drawing processing amount can be reduced.

  If there is a difference in screen display performance (number of display pixels, resolution or number of display colors) between the home PC and another device at a remote location, the application on the other device at the remote location will perform an appropriate conversion process. (Reduction / enlargement of the screen or color conversion) may be performed.

  In order to perform graphics drawing processing at high speed, there is a technique for operating a plurality of cores of a graphics processing unit (hereinafter, “GPU”) in parallel. The core is a semiconductor circuit portion excluding the cache memory at the core portion of the processor circuit created on the processor die. Multi-core is a technology in which a plurality of processor cores are enclosed in one processor package.

  When graphics drawing processing is performed in parallel by a plurality of cores, how to distribute the processing to the plurality of cores becomes a problem. As a method of assigning drawing processing to a plurality of cores, there is a method of dividing a screen into n regions and assigning n cores to the drawing processing of each region. For example, in the case of n = 16, as shown in FIG. 10, the screen is divided into 16 areas, and the 16 GPU cores are allocated to the drawing process of each area of the screen.

  As a related technique, an encoder server receives an MPEG-2 signal transmitted from an MPEG-2 encoder, and converts it into an MPEG-4 signal having a higher information compression rate and a lower transmission rate after encoding than MPEG-2. A technique has been proposed in which a receiving terminal decodes a received MPEG-4 signal by an MPEG-4 decoding unit and displays an image on a display unit (see, for example, Patent Document 1).

JP 2006-136013 A

  However, when the screen is divided into a plurality of areas and distributed processing is performed with a plurality of cores such as GPUs as in the related art described above, the result differs from the result of processing the entire screen as one by one CPU. Will occur.

  As shown in FIG. 11, when the screen is vertically divided into strips that are long in the horizontal direction, the drawing processing of each divided area is distributed by a plurality of cores such as GPUs, and a difference is detected in separate areas. More differential rectangles are extracted than originally. In other words, the difference rectangle that crosses the region is extracted by dividing it into two rectangles, which should be recognized as one rectangle. In FIG. 11, a total of 42 difference rectangles, which should have been 28, have been extracted. As a result, an overhead of an overlapping portion that does not occur if it is originally recognized as one rectangle occurs, and wasteful processing capacity and processing time are consumed for encoding / decoding.

  In addition, since more difference rectangles are extracted, it is difficult to limit the maximum number of difference rectangles in the entire screen.

  The present invention has been made in order to solve the above-described problems. An image processing apparatus, an image processing method, and a program for improving the efficiency of image processing by limiting the maximum value of the number of difference rectangles during distributed processing. The purpose is to provide.

The image processing apparatus of the present invention includes a coprocessor having a plurality of cores suitable for parallel processing with a CPU,
The CPU divides one screen into a plurality of areas, and assigns the core to each area one by one.
Regarding the difference rectangles extracted by the processing of the core, the difference rectangles extracted in adjacent areas are merged according to a predetermined rule to limit the number of difference rectangles in one entire screen.

The image processing method of the present invention is an image processing method using a coprocessor having a plurality of cores suitable for parallel processing with a CPU,
The CPU divides one screen into a plurality of areas, and assigns the core one by one for each area;
The core includes a step of merging the difference rectangles extracted in adjacent areas according to a predetermined rule for the extracted difference rectangles, and limiting the number of difference rectangles in one screen.

Further, the program of the present invention is provided on a computer provided with a coprocessor having a plurality of cores suitable for CPU and parallel processing.
Causing the CPU to divide one screen into a plurality of areas, and to execute a process of assigning the core one by one for each area;
The core executes a process of merging the difference rectangles extracted in adjacent areas according to a predetermined rule for the extracted difference rectangles to limit the number of difference rectangles in one entire screen.

  According to the present invention, it is possible to limit the maximum value of the number of difference rectangles during distributed processing and improve the efficiency of image processing.

It is a figure which shows the structure of the image processing apparatus which concerns on embodiment of this invention. 1 is a functional block diagram of an image processing apparatus according to an embodiment of the present invention. It is a flowchart which shows the processing operation which concerns on embodiment of this invention. It is a conceptual diagram which shows dividing | segmenting the screen which concerns on embodiment of this invention up and down, and dividing | segmenting into the strip shape long in a horizontal direction. It is a conceptual diagram of the state allocated to the core which concerns on embodiment of this invention. It is a conceptual diagram of the state from which the difference rectangle which concerns on embodiment of this invention was extracted. It is a conceptual diagram of the state from which the difference rectangle which concerns on embodiment of this invention was extracted. It is a figure which shows the predetermined rule which merges a difference rectangle. It is a figure which shows other embodiment. It is a conceptual diagram which shows dividing | segmenting the conventional screen into the strip shape long divided | segmented up and down horizontally. It is a conceptual diagram of the state allocated to the conventional core.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. 1 includes a central processing unit (hereinafter referred to as “CPU”) 10 that executes software for generating a plurality of drawing commands by graphic data processing, and a plurality of drawing commands. The GPU 20 and its cores 20-1 to 20-16 as a drawing processing unit that performs drawing processing and generates image drawing data for each area of the divided screen in parallel, a program storage unit 40, a main memory 50, an input unit 60, and an output unit 70. The CPU 10, GPU 20, frame memory 30, program storage device 40, main memory 50, input device 60, and output device 70 are respectively connected via a bus 80 for data transfer and the like. It is connected.

  The CPU 10 executes various middleware such as application software, application program interface (API) middleware, and software such as a device driver to generate a drawing command. Drawing commands generated from the graphic data to be processed by the CPU 10 being controlled by software are transferred to the cores 20-1 to 20-16 of the GPU 20.

The GPU 20 is an example of a coprocessor having a plurality of cores suitable for parallel processing, and is not limited to a GPU as long as it has a similar function. Here, the area of the screen is divided into 16 areas, and the cores 20-1 to 20-16 are in charge of processing each area for each area.
The image drawing data generated by the cores 20-1 to 20-16 is stored in the frame memory 30.

The program storage unit 40 stores program codes of software executed by the CPU 10, such as application software, API middleware, and device drivers. In recent years, it has become possible to execute general-purpose programs on the GPU 20,
The program may be stored.

  The main memory 50 has a graphic data storage area 51 and a command buffer area 52. The graphic data storage area 51 stores graphic data to be processed. The command buffer area 52 stores drawing commands transferred to the cores 20-1 to 20-16. The command buffer area 52 is divided according to the number of cores that the GPU 20 has. In the example illustrated in FIG. 1, the command buffer area 52 includes a first command buffer 5201 to a sixteenth command buffer 5216. As will be described later, the first command buffer 5201 to the sixteenth command buffer 5216 are set by the CPU 10 executing a device driver. The main memory 50 temporarily stores data generated by the processing of the CPU 10 and the like.

  The input unit 60 is a keyboard, a mouse, or the like on which a user inputs data. The output unit 70 is an interface capable of externally transmitting image processing results, output images, and the like, a display as a display unit, a printer, and the like.

  The application software processes the graphic data input from the input unit 60 regardless of the number of cores of the GPU 20 included in the image processing apparatus 1, and issues an API function call (instruction for performing target drawing) obtained as a processing result. Deliver to API middleware. The API middleware processes the API function call to generate a drawing command that can be executed by the hardware of the core 20-1 to 20-16 of the GPU 20. The device driver buffers the drawing command generated by the API middleware in the command buffer area 52, and then at a timing when the cores 20-1 to 20-16 can start a new process, for example, the core 20- The drawing command of the processing unit of 1-20-16 is transferred from the command buffer area 52 to the cores 20-1 to 20-16. The cores 20-1 to 20-16 perform a drawing process using a drawing command, store image drawing data (for example, pixel data) as a drawing process result in the frame memory 30, and then a predetermined output is output from the output unit 70. Made.

In the configuration related to the image processing apparatus 1 described above, the contents of the frame memory 30 are updated as a result of the user's operation on the input unit 60.
Hereinafter, when the image processing device 1 transmits an image output from the frame memory 30 or the output unit 70 to a remote device, the processing for extracting the difference between the image changes will be described in detail. explain.

  FIG. 2 shows a functional block diagram of the image processing apparatus 1 in the present embodiment.

  The image processing apparatus 1 according to the present embodiment includes a screen area dividing unit 111, an assigning unit 112, a difference detecting unit 113, and a rectangle merging unit 114.

  The screen area dividing unit 111 has a function of dividing one screen area into the number of areas that can be processed in parallel according to the number of cores of the GPU 20. The CPU 10 may inquire about the number of cores held in the GPU 20 and divide by a number within the number of cores. In the description of this embodiment, an example in which the screen is divided vertically and divided into strips that are long in the horizontal direction will be described. However, the method of dividing the screen is not limited to the method of dividing vertically, and for example, the screen is Any division method may be used as long as it divides the screen into a plurality of areas.

  The assigning unit 112 has a function of assigning drawing processing and merging processing of each divided area to predetermined cores 20-1 to 20-16 of the GPU 20. Note that a predetermined area may not be assigned with a core for performing the merge process in the present embodiment, that is, the merge process may not be performed.

  The difference detection unit 113 has a function of extracting a difference rectangle for each divided area with respect to two screens (for example, the current screen and the immediately preceding screen). Predetermined cores 20-1 to 20-16 and the like to which the processing of each divided area is assigned extract the difference rectangle.

The rectangle merging unit 114 has a function of merging difference rectangles according to a predetermined rule. The difference rectangles that are extracted across the regions are made one original rectangle, or may not be one rectangle originally, but the difference rectangles that are highly likely to be extracted across the regions are 1 It is a process to make two rectangles. The process of merging rectangular coordinates may be realized by deleting the coordinate data of the start point of the straight line where two rectangles are in contact and further deleting the coordinate data of the end point. Execution of merge processing may be performed by the CPU depending on the availability of the CPU, may be performed by each core, may be performed by a predetermined core, or may be performed in parallel by a combination thereof. Also good.
The difference rectangles are merged so as to be within the maximum value of the total number of extracted rectangles per screen stored in a predetermined storage unit.

Hereinafter, the operation of the present embodiment will be described in detail with reference to the drawings.
Referring to the flowchart of FIG. 3, first, the screen area dividing unit 111 of the image processing apparatus 1 divides one screen area into the number of areas that can be processed in parallel according to the number of cores of the GPU 20 (S301). Specifically, the CPU 10 inquires about the number of cores possessed by the GPU 20 and divides the screen vertically into 16 pieces that are the number of cores, and divides the screen into long strips in the horizontal direction (see FIG. 4).

  Next, the assigning unit 112 assigns the drawing process and the merge process of each area divided into 16 parts to the cores 20-1 to 20-16 of the GPU 20 (S302). A conceptual diagram of the allocated state is shown in FIG. Specifically, the CPU 10 assigns the cores 20-1 to 20-16 in order from the top of the 16-divided area.

  The difference detection unit 113 extracts a difference rectangle for each of the 16 regions divided into the two screens of the current screen and the screen immediately before the current screen (S303). Specifically, the cores 20-1 to 20-16 to which the processing of each area divided into 16 is assigned extract the difference rectangle. FIG. 6 shows a conceptual diagram in a state where the difference rectangle is extracted.

Then, the rectangle merge unit 114 merges the difference rectangles according to a predetermined rule (S304). Specifically, the merge process may be performed by the CPU depending on the CPU availability, may be performed by each core, may be performed by a predetermined core, or a combination thereof. It may be processed in parallel.
In the portion surrounded by the broken line in FIG. 7 showing the state in which the difference rectangle is extracted, the difference rectangle is extracted across the region regardless of the original one rectangle.

Hereinafter, a predetermined rule for merging difference rectangles will be described with reference to FIG. An example in which the maximum number of total extracted rectangles per screen stored in a predetermined storage unit is limited to 32 will be described. That is, the difference rectangles are merged so that the total number of extracted rectangles is 32 or less. Since it is a merging of the vertical direction, focus on the X coordinate basically.
First, as Step 1, merging is performed when the X coordinates of two adjacent rectangles match.
If the length of the longer side B of two adjacent rectangles is less than 1.6 times the length of the shorter side A, then, if the number is still not less than 32 after the processing of Step 1 Merge. The part where the sides are common is the effective pixel area, but the part where the sides are not common is the invalid pixel area, which is not limited to 1.6 times, and the efficiency is improved by the area ratio of the effective pixel area and the invalid pixel area. A value that is not impaired may be used.
If the number is still not less than 32 after the processing of Step 2, then Step 3 is merged when the X coordinates of the two rectangles match and the distance between the Y coordinates of Side A and Side B is equal to or less than a predetermined value. That is, the space where the gap is generated is slightly separated.
If the number is still not less than 32 after Step 3 processing, then, as Step 4, merging is performed when side A and side B of the two rectangles are close.
If the number is still not less than 32 after the processing of Step 4, next, as Step 5, the start point coordinates are searched from the upper left of the screen to the right and downward, and the X and Y directions of the rectangle remaining from the 32nd start point are searched. Merge up to the maximum value. That is, 32 to 42 are merged into one.

According to the above-described embodiment, when the maximum value of the total number of extracted rectangles per screen is 32 and one screen is divided into 16, the maximum number of extracted rectangles per core is two. However, it is possible to avoid a situation where the efficiency is low.
In other words, if the number of extraction rectangles per core is limited to a maximum of two, if the core in charge of a certain divided area has no difference detected and there are zero difference rectangles, For example, the two areas may be used by the core in charge of the divided area to detect four difference rectangles, but these processes are performed in parallel by a plurality of cores. Can not do. For this reason, the number of cores in charge of all the divided areas is limited to two or less, resulting in poor efficiency. This situation can be avoided.

As another embodiment, as shown in FIG. 9, one screen is divided into a moving image area and other areas from the coordinates of the moving image area obtained by the CPU 10 from the OS, and the CPU 10 is assigned to the moving image area. You may allocate the core of GPU20 to 4 area | regions other than a moving image area | region from area ratio. That is, the moving image area changes drastically, and a large number of differences are generated on each screen. Therefore, it is necessary to thin out data in order to secure a band, and therefore, it is required to process it separately.
In the moving image region and other regions, even if the above Step 1 to Step 5 merge processing is not performed, the above Step 1 to Step 5 merge processing is performed at the boundary of the region for the four regions other than the moving image region. It may be done.

  Each of the above-described embodiments is a preferred embodiment of the present invention, and various modifications can be made without departing from the scope of the present invention. For example, a process for realizing the function of each apparatus may be performed by causing each apparatus to read and execute a program for realizing the function of the image processing apparatus 1. Further, the program is transmitted to another computer system by a transmission wave via a computer-readable recording medium such as a CD-ROM or a magneto-optical disk, or via a transmission medium such as the Internet or a telephone line. Also good.

10 CPU
20 GPU
20-1 to 20-16 Core 30 Frame memory 40 Program storage unit 50 Main memory 60 Input unit 70 Output unit 80 Bus unit

Claims (5)

A CPU and a coprocessor having a plurality of cores suitable for parallel processing,
The CPU divides one screen into a plurality of areas, and assigns the core to each area one by one.
An image processing apparatus that limits the number of difference rectangles in one screen by merging the difference rectangles extracted in adjacent areas according to a predetermined rule for the difference rectangles extracted by the processing of the core. The one screen is divided vertically and divided into strips that are long in the horizontal direction,
The predetermined rule is:
Merge when the X coordinates of two adjacent rectangles match,
Merge when the length of the longer side B of two adjacent rectangles is within a predetermined magnification of the length of the shorter side A;
Merge when the X coordinates of the two rectangles match and the distance between the Y coordinates of side A and side B is less than or equal to a predetermined value;
Merge when side A and side B of two rectangles are close,
By searching for the coordinates of the starting point from the upper left to the lower right of the screen, and merging from the number of starting points corresponding to the number of allowable difference rectangles in the entire screen to the maximum value in the X and Y directions of the remaining rectangle. The image processing apparatus according to claim 1, wherein the image processing apparatus is provided.   The image processing apparatus according to claim 1, wherein the merging process is not performed by distinguishing a specific area from other areas. An image processing method using a coprocessor having a plurality of cores suitable for parallel processing with a CPU,
The CPU divides one screen into a plurality of areas, and assigns the core one by one for each area;
An image processing method, wherein the core has a step of merging the difference rectangles extracted in adjacent areas according to a predetermined rule for the extracted difference rectangles, and limiting the number of difference rectangles in one entire screen . In a computer with a coprocessor that has multiple cores suitable for CPU and parallel processing,
Causing the CPU to divide one screen into a plurality of areas, and to execute a process of assigning the core one by one for each area;
A program for limiting the number of difference rectangles in one entire screen by causing the core to execute a process of merging difference rectangles extracted in adjacent areas according to a predetermined rule.

Images