JP2013218631A - Information processing device, information processing method, and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device and an information processing method monitoring parameters operated by a user in an image processing system and properly using an image processing instruction to be executed by a GPU (Graphics Processing Unit), according to an operation state of the parameters.SOLUTION: The information processing device includes: a graphics processing unit capable of executing a plurality of image processing instructions; a detection unit that detects an operation by a user for operation parameters corresponding to the image processing instructions; an instruction set storage unit that retains a combination of image processing instructions to be executed by the graphics processing unit, as an instruction set; and an instruction set selection unit that selects an instruction set to be executed by the graphics processing unit in the instruction sets retained by the instruction set storage unit, on the basis of the comparison result by a parameter comparison unit.

Description

  The present technology relates to an information processing apparatus that performs image effect processing using a GPU (Graphics Processing Unit).

  Also in the field of image processing, the situation where processing that could previously be implemented only with dedicated hardware has been implemented by a program executed on a general-purpose computer has become serious. This is because the processing speed that can satisfy the user can be obtained even with the implementation by the program due to the improvement of the performance of the CPU and the memory.

  In addition to software that is executed by the CPU for further speedup, there is an implementation example in which a DSP is incorporated as described in “Registration 4325123”.

  In recent years, GPUs (Graphics Processing Units) for graphics rendering incorporated in computers are used not only for rendering but also for other numerical operations. General purpose computing on graphics processing units (GPGPU) ) Is a technical field.

  Since GPUs are inexpensive, easy to obtain, and capable of high-speed computation, editing systems incorporating GPUs that perform special effects (effects) of images can be developed at low cost.

  However, simply creating and executing a program that runs on the GPU does not mean that the speed can be easily increased. In order to realize real-time processing of large-size images and image processing with a large amount of calculation, it is necessary to devise image processing algorithms and implementation in the program development.

  Recently, several inventions have been disclosed that mention a device for speeding up operations using a GPU.

  For example, Patent Document 1 relates to the decoding of a video signal by properly using the processing steps between the GPU and the CPU to balance the work load between the GPU and the CPU. A technology is disclosed in which data communication between a CPU and a GPU is minimized, a workload of the CPU and the GPU is balanced, and a module that is offloaded to the GPU can be efficiently realized.

  Further, as disclosed in Patent Document 2, there is an example in which the processing algorithm is adapted to a form suitable for a large number of parallel operations, which is an advantage of the GPU. The hair simulation method disclosed in Patent Document 2 models hair using a particle model and can be operated on a GPU. The particle model used in the hair simulation method disclosed herein is suitable for mounting using a GPU because each particle constituting the particle model can be processed in parallel.

JP 2010-130696 A JP 2009-20874 A

  There are still few inventions related to GPGPU.

  In view of the circumstances as described above, an object of the present technology is to monitor an information processing apparatus in which an image processing command to be executed by a GPU according to an operation state of a parameter is monitored in a image processing system. A processing method and an information processing program are provided.

  (1) In order to achieve the above object, an information processing apparatus according to an aspect of the present technology includes a graphics processing unit that can execute a plurality of image processing instructions, and an operation parameter corresponding to the image processing instructions. Based on a comparison result in the parameter comparison unit, a detection unit that detects an operation by a user, an instruction set storage unit that holds a combination of the image processing instructions executed in the graphics processing unit as an instruction set, and An instruction set selection unit that selects the instruction set to be executed by the graphics processing unit from the instruction set held by the instruction set storage unit;

  In the present technology, when performing image processing, the detection unit detects an operation parameter input by the user using the GUI. The instruction set storage unit holds a combination of image processing instructions executed in the graphics processing unit as an instruction set. Note that the image processing command in the command set corresponds to the operation parameter. The instruction set selection unit selects a corresponding instruction set from the instruction sets held in the instruction set storage unit based on the detection result by the detection unit. The selected instruction set is then sent to the graphics processing unit for execution.

  As a result, an image processing instruction corresponding to an operation parameter that is not operated on the GUI by the user is never included in the instruction set, so that unnecessary processing is prevented from being executed on the graphics processing unit. be able to. Further, it is possible to realize speeding up of the entire image processing.

  (2) In addition, the detection unit compares whether the initial value storage unit that stores the initial value of the operation parameter differs from the operation parameter operated by the user and the initial value stored in the initial value storage unit. A parameter comparison unit may be provided.

  The initial value storage unit holds initial values of operation parameters on the GUI on which the user performs image processing operations. When performing image processing, the operation parameter input by the user using the GUI and the initial value of the operation parameter stored in the initial value storage unit are compared by the parameter comparison unit, and the operation parameter input by the user It is determined whether there is a difference between the initial value and the initial value.

  As a result, an image processing command corresponding to an operation parameter that is not operated by the user on the GUI, that is, not operated from the initial value is not included in the command set. It can be prevented from being executed on the unit. Further, it is possible to realize speeding up of the entire image processing.

  (3) The instruction set storage unit may hold a combination of image processing instructions, that is, an instruction set for the combination of operation parameters. By holding only the combinations of the operation parameters in advance, it is possible to appropriately select the corresponding instruction set and send it to the graphics processing unit for execution regardless of the combination of operation parameters. .

  (4) In the information processing method according to the present technology, the detection unit in the information processing device detects the operation parameter operated by the user, and the instruction set selection unit in the information processing device detects the detection result by the detection unit. Based on the instruction set, which is a combination of image processing instructions corresponding to the operation parameters, which is executed in the graphics processing unit in its own device, held in the instruction set storage unit, from the graphics processing unit The instruction set to be executed is selected.

  (5) An information processing program according to the present technology is an information processing program for operating a computer including a graphics processing unit capable of executing a plurality of image processing instructions, and an operation parameter corresponding to the image processing instructions A detection unit for detecting an operation by a user, an instruction set storage unit for holding a combination of the image processing instructions executed in the graphics processing unit as an instruction set, and a comparison result in the parameter comparison unit, The computer is caused to function as an instruction set selection unit that selects the instruction set to be executed by the graphics processing unit from the instruction set held by the instruction set storage unit.

  As described above, according to the present technology, it is possible to eliminate redundant computations in image processing, and it is possible to increase the processing speed when processing results are obtained by performing a plurality of image processing.

1 is a diagram illustrating a configuration of a color grading system according to an embodiment of the present technology. It is a figure which shows the example of GUI of a color grading system, and an image processing result. It is a block diagram showing the hardware constitutions of information processing apparatus. It is a figure which shows the example of the process result of a color grading process. It is a figure which shows the example of a combination of a some layer process. It is a figure which shows the process block corresponded to one layer process. It is a conceptual diagram showing the difference between "space area designation 1" and "space area designation 2". It is a conceptual diagram of AND composition in layer processing. It is a block diagram of layer composition. It is a conceptual diagram of OR composition. It is a block diagram showing selection processing of a GPU image processing command. It is a block diagram showing selection processing (generalized) of GPU image processing instructions. It is a functional block diagram of an information processor. It is an example of operation GUI of a color grading system. It is an example of operation GUI of a color grading system. It is an example of operation GUI of a color grading system. It is an example of operation GUI of a color grading system. It is an example of operation GUI of a color grading system. It is a figure explaining the thread | sled instruction which shows the application range of this technique.

  Hereinafter, embodiments according to the present technology will be described with reference to the drawings.

  In the following description, it is assumed that the present technology is applied to an effect process for processing an image in a video editing field such as a digital cinema.

<Overview>
In the present technology, when an arithmetic instruction for specifying an area to be processed in an image or an arithmetic instruction for effect processing to be performed on the specified area is sent to a GPU (Graphics Processing Unit), parameters for the area specification and the effect processing are previously set. By checking the above, the processing time in the GPU is shortened by selectively sending instructions while omitting operation instructions that are not executed.

<First Embodiment>
There are various types of effect processing. In the following, a color grading system that freely corrects the hue of a captured image will be described as an example. In color grading, there are a large number of operation parameters that can be adjusted by the user. However, in order to simplify the explanation, here, the operation parameters are “spatial region specification parameter”, “color region specification parameter”, “color correction”. This will be described as three types of parameters.

FIG. 1 is a diagram illustrating a configuration of a color grading system. The user can adjust the parameters while viewing the GUI. The operation input device 18 is an input means such as a mouse or a keyboard. The display device 19 displays a GUI (Graphical User Interface) for manipulating parameters and the result of image processing.
While viewing the GUI displayed on the display device 19, the user can adjust various parameters by operating the GUI with a mouse or inputting numerical values from the keyboard.

The information processing apparatus 10 is a general-purpose computer or the like, and has a calculation GPU mounted therein. The information processing apparatus 10 executes image processing and displays the processing result on the display device 19.
FIG. 2 is a diagram illustrating an example of a GUI displayed on the display device 19 and an image processing result.

[Configuration of Information Processing Apparatus 10]
Next, FIG. 3 is a block diagram illustrating a hardware configuration of the information processing apparatus 10.

  The information processing apparatus 10 includes a CPU unit 11, a GPU unit 12, a storage device 13, a display interface 14, an operation interface 15, a network interface 16, and a bus 17 that connects these components to each other.

  The CPU unit 11 includes a CPU 111 and a memory 112 (hereinafter referred to as “CPU memory”), and executes instructions related to various arithmetic processes on the CPU memory 112 by executing a program stored in the CPU memory 112. . The CPU unit 11 interprets a command input by the user through the operation input device 18 connected to the operation interface 15 and reflects it in the operation of the program. For example, the CPU unit 11 can read out image data stored in the storage device 13 and import the image data into the CPU memory 112 to perform processing such as effects on the image data. The image data held in the CPU memory 112 is supplied to the display interface 14, and is rendered as rendering data by being rendered here, and the rendering data of the image processed by the GPU unit 12, which will be described later, as necessary. And are output to the display device 19. Further, the CPU unit 11 performs control so that the processed image data held in the CPU memory 112 is merged with the image data processed by the GPU unit 12 and written back to the storage device 13 as necessary. Is possible.

  The GPU unit 12 includes a GPU 121 and a memory 122 (hereinafter referred to as “GPU memory 122”), and by executing a program stored in the GPU memory 122, image processing such as effects is performed in parallel on the GPU memory 122. It can be executed by arithmetic processing. The image data held in the GPU memory 122 is supplied to the display interface 14 where it is made visible drawing data by drawing processing, and merged with the drawing data of the image processed by the GPU unit 12 as necessary. And output to the display device 19.

  The display interface 14 is an interface with the display device 19, performs drawing processing of image data supplied from the CPU unit 11 and the GPU unit 12, and draws image drawing data processed by the CPU unit 11 as necessary. The drawing data of the image processed by the GPU unit 12 is merged and supplied to the display device 19 as drawing data of one image. The processing of the display interface 14 is realized by, for example, the above-described GPU 121 or a separately provided GPU (not shown).

  The operation interface 15 is an interface with the operation input device 18, and performs a process of supplying data and commands from the user input from the operation input device 18 to the CPU unit 11.

  The storage device 13 stores, for example, various programs for causing the CPU unit 11 and the GPU unit 12 to store and edit image data before editing and image data after editing.

  The network interface 16 is an interface for connecting to the network 30.

[About color grading and layer processing]
FIG. 4 is a diagram illustrating an example of a processing result of the color grading process. As shown in the processing result image, the color adjustment region is not limited to one place, and usually, several places are adjusted in one screen. (In the image shown on the processing result screen on the right side of FIG. 4, the colors of the areas surrounded by the white lines are corrected.) In the following, processing (color correction, etc.) for each area is referred to as layer processing. Call.

[Combination example of multi-layer processing]
FIG. 5 is a diagram illustrating an example in which a final processing result image is obtained by each layer processing (described later) and layer synthesis of those results. A thick line indicates the flow of the region image. In this example, the result of the layer processing LP1 and the result of the layer processing LP2 are combined by the layer combining LM1 (for example, OR combining, details will be described later), and the layer processing LP3 is performed on the result. Then, an image on which layer processing LP4 has been separately performed and an image on which layer processing LP3 has been performed are synthesized by layer synthesis LM2, and a final result image is obtained.

  The process illustrated in FIG. 5 is merely an example of layer processing and layer synthesis. Actually, there are various combinations of layer processing and layer synthesis depending on the combination of the target image to be corrected and the user's operation. Details of layer processing and layer composition processing will be described later.

[About layer processing]
FIG. 6 is a diagram illustrating a processing block corresponding to one layer process. In the following description, reference numerals in the drawings are shown in parentheses.
First, the shape of the region is calculated in order to determine which region is targeted for color correction. There are roughly two types of methods for giving the shape of a region through a GUI by a user. “Space area designation” and “Color area designation (CS)”. The “space area designation” is further divided into “space area designation 1 (SS1)” and “space area designation 2 (SS2)”. “Space area designation 1 (SS1)” is performed using the space area parameter 1 as an input. Further, “spatial region designation 2 (SS2)” is performed using the processing result of “spatial region designation 1 (SS1)” and the spatial region parameter 2 as inputs. Details will be described later.

  Next, “AND synthesis (AM)” is performed on the shapes of the areas respectively determined by “space area designation” and “color area designation (CS)”. After “AND synthesis (AM)” is performed, the shape of the synthesized area may be output as an area image for other layer processing, or “color correction processing ( CM) ”. Details of “AND composition” will be described later.

  Next, regarding the switch (SW), here, either the area image created by “AND composition (AM)” or the existing area image created by other layer processing is used for color correction processing. Can be switched. The thick line indicates the flow of the area image.

  Finally, there is “color correction processing (CM)”. Here, based on the input region image, color correction processing is performed only on the designated region using the color correction parameter and the input image, and an image of the processing result is output. When the area image is multivalued, the correction amount of color correction is adjusted so as to be proportional to the value.

[About space area specification]
The space area designation will be described. Spatial area designation is an area designation method for directly designating which area in the screen is to be corrected.

  The actual processing in the space area designation is performed in two steps: “space area designation 1 (SS1)” executed on the CPU unit 11 and “space area designation 2 (SS2)” calculated on the GPU. Composed.

  FIG. 7 is a diagram illustrating an image representing a difference between “space area designation 1 (SS1)” and “space area designation 2 (SS2)”.

  First, in the “space region designation 1 (SS1)” step, the user designates the shape SP of the region by directly pointing the boundary portion of the shape constituting the region via the GUI. The CPU unit 11 generates the shape SP of the region by connecting the pointed points P using straight lines or curves.

  This pointing position information corresponds to “spatial region parameter 1” shown in FIG. This region shape generation calculation is not performed in each frame, but is recalculated only when the user makes corrections such as deletion, addition, or movement to the point P that is pointed. That is, the shape SP of the area once formed is held as information on the area image until the user corrects it.

  The calculation for generating or correcting the shape SP of this region is performed on the CPU unit 11. The reason is that parallel processing suitable for the GPU is difficult due to the complicated processing, and it is only necessary to re-calculate only when the user corrects it instead of performing computation in each frame.

  Next, the GPU performs the step of “space area designation 2 (SS2)”. In the “space area designation 2 (SS2)” step, the GPU expands the area shape SP based on the area shape SP generated or modified in the “space area designation 1 (SS1)” step. Or perform operations that reduce, move, or rotate.

  The enlargement ratio, the movement amount, and the rotation angle used in the calculation of enlargement, reduction, movement, and rotation correspond to “spatial region parameter 2” shown in FIG.

  The step of “spatial area designation 2 (SS2)” is used because the user designates the shape of the object in the “space area designation 1 (SS1)” step in order to correct the color of an object. Later. This step is used to correctly correspond to the area where the color correction is performed when the shape or position of the object changes after the frame designated by the user.

  The geometrical operations of area shape enlargement, reduction, movement, and / or rotation are performed frame by frame in the GPU. The reason is that the calculation is performed for each pixel of the image information obtained in the “space area designation 1 (SS1)” step, and thus the calculation can be easily parallelized.

  Instead of performing the “space area designation 1 (SS1)” step, the user selects a regular shape such as a rectangle or an ellipse as the area shape SP, so that the shape in the “space area designation 1 (SS1)” step is performed. Specification can be omitted.

  In addition, parameters such as enlargement, reduction, movement, and rotation used in the step of “space area designation 2 (SS2)” are automatically determined based on the result of a motion detection algorithm such as block matching in addition to being given by the user through GUI operation Can also be entered.

[Color area specification]
Another area designating means in the layer processing is area designation by color space threshold processing. For example, the user designates only pixels in a range that falls within A ≦ Hue ≦ B as a region for Hue representing a hue in the color space. When the user operates the threshold values A and B, the user can specify which color object in the image is to be corrected. The parameters represented by the threshold values A and B correspond to the “color region parameters” shown in FIG.

  The area designation method in the “color area designation (CS)” step is not limited to the hue range designation, but can be variously combined depending on the situation, such as brightness, saturation, and each level of the RGB plane.

[About AND composition]
FIG. 8 is a diagram showing an image of AND composition. An area where the space area SA and the color area CA are overlapped and designated by the two methods of “space area designation” and “color area designation” is the AND area AA.

  As shown in the figure, for example, an area having a value corresponding to each position in the image, for example, a pixel included in the designated area has a value of 1 and other pixels have a value of 0. An image is generated.

  In addition, since the level difference between the corrected portion and the non-corrected portion is conspicuous at the boundary of the region, the value near the boundary of the region is set to a value of 0.5 that is between 0 and 1 in order to reduce the sense of discomfort. The region image may be generated as multi-value information such as

  In the AND composition, AND regions are extracted from two types of designated regions (SA and CA), and are generated as final region images. The extraction of the AND area AA is obtained by AND operation processing of each phase value when each area image is binary, and is obtained by the product of the respective phase values when multi-valued.

[About layer composition]
Next, processing contents of layer synthesis will be described. FIG. 9 is a block diagram thereof.

  In layer synthesis, basically, an OR operation is performed on the color correction results of the respective regions that have been obtained in each layer process. Hereinafter, this synthesis by OR operation is referred to as OR synthesis (OM1). At the same time, OR synthesis (OM2) of these regions is also performed on the region images input from each layer process.

  FIG. 10 shows an image of OR synthesis. In this OR synthesis, when the processing areas of the respective layer processes do not overlap on the image, there is no problem because the color correction result in each layer process may be used as the synthesis result. If the processing areas overlap on the image, in the overlapping area OL, a layer processing result with a higher priority is preferentially selected, or an average value of all layer processing results is taken. Can be synthesized.

[Example of layer processing]
As described above, when the user corrects an image, the user often performs a plurality of layer processes on one image and superimposes the results. In color grading systems, there is no standard usage, and various operation parameters and combinations of layer processing are left to the user depending on the situation.

  As an example when using the color grading system, a specific example of processing performed in each layer processing will be described with reference to FIGS. 5 and 6. Here, the space area designation 1 (SS1) performed on the CPU section 11 and the space area designation 2 (SS2) performed on the GPU section 12 are collectively referred to as space area designation (SS), Region parameter 1 and space region parameter 2 are collectively referred to as a space region parameter.

  In the layer processing LP1, since the user has operated an operation parameter related to space area designation on the GUI, “space area designation (SS)” is executed using the space area parameter, and an area image is generated. Other “color area designation (CS)” and “color correction processing (CM)” are not executed because the user did not operate the related operation parameters on the GUI.

  In the layer processing LP2, “color area designation (CS)” and “color correction processing (CM)” are executed based on the GUI operation by the user.

  In the layer synthesis LM1, the processing results of the layer processing LP1 and the layer processing LP2 are synthesized.

  In the layer processing LP3, the area image obtained from the layer synthesis LM1 is used, and “color correction processing (CM)” is further performed.

  In the layer processing LP4, since the user did not perform the “space area designation (SS)” and “color area designation (CS)” operations, these processes are not performed, and no area is designated. Therefore, only “color correction processing (CM)” is performed uniformly on the entire screen.

  In the layer synthesis LM2, the result of the layer processing LP3 and the result of the layer processing LP4 are synthesized.

Table 1 summarizes the processing contents of the above-described layer processing. ○ indicates that processing such as space area designation is executed in the layer processing, and a column without ○ indicates that processing is not executed.
As described above, in each layer process, not all “space area designation (SS)”, “color area designation (CS)”, and “color correction process (CM)” are executed. That is, for example, in the layer processing LP1, when only the “spatial region designation (SS)” is performed, the “color region designation (CS)” and the “color correction processing ( It was useless to send an image processing command for “CM)”.

[GPU image processing instruction selection process]
Therefore, in the present technology, it is proposed that the processing illustrated in FIG. 11 is performed in the information processing apparatus 10 when each layer process is executed.

  First, the CPU unit 11 performs three processes, “space area designation (SS)”, “color area designation (CS)”, and “color correction process (CM)”, which are processing contents performed in one layer process. Regarding the related operation parameters, the initial values, that is, the operation parameters in a state where the user is not operating the GUI are stored. In the figure, these are portions represented by “space area parameter initial value (IV1)”, “color area parameter initial value (IV2)”, and “color correction parameter initial value (IV3)”.

  Next, when the user operates the GUI, the CPU unit 11 compares the stored initial value with the operation parameter specified by the user for the layer processing, and there is a difference. In this case, 1 is output as a flag, otherwise 0 is output. In the figure, “space area parameter (SAP)”, “color area parameter (CAP)”, and “color correction parameter (CMP)”, and “comparison (CMP1 to CMP3)” correspond.

  Here, since the three processes of “space area designation (SS)”, “color area designation (CS)”, and “color correction process (CM)” are performed independently, flags for the three processes are used. In response, an image processing command to be executed on the GPU unit 12 is selected. Since the three processes can be executed independently, a total of eight patterns of instruction sets can be considered as combinations of the three events.

  In each of the eight types of instruction sets, processing with a flag of 1 executes an operation in the GPU unit 12, so a GPU image processing instruction for the operation is incorporated, and processing with a flag of 0 performs an operation in the GPU unit 12. Since it is omitted, the GPU image processing instruction for the calculation is not included.

  The following Table 2 shows GPU image processing instructions for performing three processes of “spatial area designation (SS)”, “color area designation (CS)”, and “color correction process (CM)”, and those instructions. It is a table | surface which shows the combination with an instruction set.

For example, the instruction set B includes a GPU image processing instruction for space area designation (SS), so that the corresponding column is marked with a circle, and the other fields are not marked with a circle. GPU image processing instructions are not included. In the case of the instruction set A, since no circle is attached, the GPU image processing instruction is not included and the state is substantially through.

  In the figure, eight types of instruction sets from instruction set A to instruction set H are held in the “8 types of GPU image processing instructions” part (CS). Then, the “space area parameter (SAP)”, “color area parameter (CAP)”, and “color correction parameter (CMP)” are “compared (CMP 1 to 3)” with the respective initial values, and the value of each flag is obtained. Based on this, in the “switching (SW1)” portion, an appropriate instruction set is read out and sent to the GPU unit 12. In the GPU unit 12, the instruction set sent from the CPU unit 11 is executed on the input image and output as an output image.

  By using these instruction sets in detail for each layer process, redundant operations in each layer process can be eliminated. Further, as in the above-described example of color grading, when a multi-layer process is performed to obtain a processing result, the processing speed can be increased.

  In addition, the comparison processing with the initial value of the operation parameter is not performed on the CPU unit 11, but is incorporated into the instruction set as a GPU instruction and is performed by conditional branching on the GPU, and the GPU image processing instruction may be used properly. It can be considered, but it is not appropriate.

  The reason is that the comparison process with the initial value of the operation parameter itself is a light operation and there is almost no processing delay even when executed on the CPU unit 11, or a parallel operation is performed by arranging a pixel unit thread. In the implementation form of the GPU image processing, the operation parameter is compared with the initial value for each pixel and the processing is switched, which is rather redundant.

[GPU image processing command selection processing (generalization)]
FIG. 12 is a generalized block diagram related to the present technology, not limited to the present embodiment.

  Here, only the difference between FIG. 11 showing the present embodiment and FIG. 12 showing the generalized form will be described.

  The “color area designation (CS)” part is generalized to “area setting by pixel level determination”, and the initial value is “area setting parameter set initial value (IV4) by pixel level determination”. The parameter is “region setting parameter set (LAP) by pixel level determination”.

  The “color correction processing (CM)” portion is generalized to “effect processing”, the initial value is “effect processing parameter set initial value (IV5)”, and the operation parameter is “effect processing parameter set ( EPP) ”.

  Note that the types of instruction sets held in the “GPU image processing instruction set (CS2)” are not limited to eight.

[Functional Block Diagram of Information Processing Apparatus 10]
In FIG. 13, a functional block diagram of the information processing apparatus 10 is shown.

  The information processing apparatus 10 includes the CPU unit 11 and the GPU unit 12 as described above. The CPU unit 11 includes an initial value storage unit 51, a parameter comparison unit 52, an instruction set storage unit 53, and an instruction set selection unit 54. Instead of the initial value storage unit 51 and the parameter comparison unit 52, a configuration may be provided in which a detection unit is provided. In this case, the detection unit detects the GUI operation status by the user, and gives the detection result to the instruction set selection unit 54.

  The initial value storage unit 51 stores initial values of “spatial region parameter”, “color region parameter”, and “color correction parameter” that are operated by the user using the GUI at the time of layer processing, and uses these initial values as parameters. It outputs to the comparison part 52.

  The parameter comparison unit 52 compares the operation parameter input by the user via the GUI with the initial value of the operation parameter acquired from the initial value storage unit 51, and sends the flag value to the instruction set selection unit 54 as a comparison result. Output. As a result of the comparison, if there is a difference between the operation parameter input by the user and its initial value, the parameter comparison unit 52 outputs 1 as the flag value, and outputs 0 if there is no difference. The comparison by the parameter comparison unit 52 is performed for each operation parameter of “space area designation (SS)”, “color area designation (CS)”, and “color correction processing (CM)”. Flags are also provided for the operation parameters of “space area designation (SS)”, “color area designation (CS)”, and “color correction processing (CM)”.

  The initial value storage unit 51 and the parameter comparison unit 52 correspond to operation parameters related to three processes of “space area designation (SS)”, “color area designation (CS)”, and “color correction process (CM)”. You may have three each. That is, the initial value storage unit 51 for “spatial region designation (SS)” stores only the initial value related to the spatial region parameter, and the parameter comparison unit 52 for “spatial region designation (SS)” relates to the spatial region parameter. Alternatively, the configuration may be such that the initial value and the parameter operated by the user are compared, and the flag value relating to only the spatial domain parameter is set.

  The instruction set storage unit 53 holds as many instruction sets corresponding to combinations of flags as combinations of image processing instructions executed on the GPU. In the example of the present embodiment, there are eight types of instruction sets from instruction set A to instruction set H. The instruction set storage unit 53 outputs the requested instruction set to the instruction set selection unit 54 in accordance with the request from the instruction set selection unit 54.

  Based on the value of the flag received from the parameter comparison unit 52, the instruction set selection unit 54 reads an instruction set corresponding to the value from the instruction set storage unit 53, and outputs the read instruction set to the GPU unit 12.

  For example, when the value of the flag is “010”, the values of the respective positions are “space area designation (SS)”, “color area designation (CS)”, and “color correction processing (CM)” in order from the left. If the operation parameter comparison result is expressed, this flag indicates that only the operation parameter of “color area designation (CS)” is different from the initial value. Therefore, from Table 2, the instruction set C is read from the instruction set storage unit 53 as an instruction set corresponding to the value of this flag, and is sent to the GPU unit 12 via the instruction set selection unit 54.

[Specific examples of operation parameters]
In the above example, the combination of instruction sets is focused on the usage status of the operation parameters related to the three processes of “space area designation (SS)”, “color area designation (CS)”, and “color correction process (CM)”. Set. Further, it is possible to subdivide the calculation contents of the processing and switch more instruction sets in detail.

  Actually, in the details of each process of “spatial area designation (SS)”, “color area designation (CS)”, and “color correction process (CM)”, operations of many processes including similar operation parameters are performed. There are many parameters and a number of computing means using them. The reason for this is that the user uses different operation parameters and correction methods depending on the situation in order to more closely approximate the expected output image result for an input image under various conditions.

As a specific example of the spatial domain parameter,
・ Information indicating whether it is a fixed shape such as an ellipse or a rectangle, or an arbitrary shape , Position information, etc. ・ Softness information that adapts to the vicinity of the boundary of the area.

As a specific example of the color area parameter,
-Each threshold value in a color space such as HSL, YUV, RGB, etc.-Inclination information that adds multi-value information to a level near the threshold value.

As a specific example of the color correction parameter,
-Hue and saturation of each area of high, medium, and low luminance-Upper and lower limits of luminance-γ correction values for luminance-Input / output curves for each level of R, G, B, and luminance-Hue of the entire screen, For example, a flag for setting the saturation in a range other than the saturation adjustment / region image to 0 is used.

  As described above, there are many operation parameters, but in a single layer process, all of the operations related to these operation parameters are rarely executed, and operations corresponding to a few operation parameters are often performed. .

  As described above, the color grading system requires a large number of layer processings as shown in FIG. 5, but only a small part of the operation parameters are calculated in each layer processing. Therefore, the present technology is effective. is there.

  Note that the concept of the present technology is not limited to the color grading system, but can be applied to other image processing for editing in which a large number of operation parameters exist, such as blur processing combining mosaic designation and mosaic processing.

[Specific examples of GUI for operation]
FIGS. 14 to 18 show an example of a GUI operated by the user in the color grading system. In addition, the description which is not related to this technique is not performed.

  In FIG. 14, an upper left area AR1 is a tab for selecting a layer to be operated. The upper right area AR2 is written as “Use Key of Previous Node”, and designates whether or not to use the image area of the previous layer. When selected, the space area designation and the color area designation are automatically turned off. The lower areas AR3 to AR5 are related to the color correction process, and correct the hue and saturation of the low luminance area, medium luminance area, and high luminance area, respectively, from the left.

  In FIG. 15, the uppermost area AR6 relates to the color correction process, and each slider performs brightness correction of the low brightness area, the middle brightness area, and the high brightness area, respectively. The second area AR7 is also related to the color correction processing, and each slider is arranged in order from the top, including overall color correction, overall saturation correction, low brightness area saturation correction, high brightness area saturation correction, and luminance offset. The brightness gain and the brightness γ correction value are adjusted. The third area AR8 is also related to the color correction process and adjusts the output level. In the input method, the horizontal axis represents the input level, the vertical axis represents the output level, and the output curve is directly manipulated. Operations can be performed on R, G, B, and luminance.

  In FIG. 16, an uppermost area AR9 relates to the color area designation, and performs area designation by hue threshold designation. The second area AR10 is also related to the color area designation, and designates the area by specifying the saturation threshold value. The third area AR11 is also related to the color area designation, and performs area designation by luminance threshold designation.

  In FIG. 17, an uppermost area AR12 relates to the space area designation 1 and selects a shape for designating the space area. As the shape, a rectangular shape, an ellipse shape, an arbitrary polygon shape, or an arbitrarily designated closed surface by a Bezier curve can be selected.

  The second area AR13 is also related to the spatial area designation 1, and the operation GUI shown here is for the case where the shape selected in the upper area AR12 is an arbitrary polygon or an arbitrary closed phase, The shape SP is generated by adding or moving the point P. When the shape selected in the upper area AR12 is a rectangle or an ellipse, the user directly operates the shape. An example of the operation GUI when the selected shape is a rectangle or an ellipse is shown in FIG.

  The third area AR14 is also related to the space area designation 1, and is adapted to blend the boundaries of the space area. The third area AR14 generates a multi-valued image by blurring a binary (0 and 1) spatial area image and smoothes the boundary.

  The fourth area AR15 relates to the space area designation 2 and designates the amount of movement, enlargement / reduction, and rotation for the space area having the shape designated in the space area designation 1.

[Applicability of this technology]
This technique is not limited to the example of color grading shown in the present embodiment as long as a large number of thread instructions can be uniformly applied to the entire image or the entire area, such as parallel operation on a GPU. Applicable. This will be supplementarily described with reference to FIG.

  As in “case A”, a thread instruction A (corresponding to the above instruction set) in which an image processing instruction for performing an operation A, an operation B, and an operation C is incorporated is calculated by arranging a thread for each pixel. Assume a case.

  In this case, when it is determined that the operation A is not performed on the entire image, the thread instruction is omitted from the thread instruction A, as in “case B”, and is changed to the thread instruction B that performs only the operations B and C. Can change. In this case, since the operation A is not executed, the processing speed can be increased. Changing the state of “case A” to the state of “case B” corresponds to the present embodiment.

  Note that as shown in “case C”, a thread instruction with a conditional branch that performs or does not perform operation A for each pixel unit does not perform operation A when waiting until all thread processing is completed. The thread process for the pixel ends early. However, since this thread instruction waits until the thread processing for the pixel performing the operation A is completed, the overall speed is not increased. In addition, since conditional branching calculation is performed for each pixel, this point is also inefficient.

[Other configurations of this technology]
In addition, this technique can also take the following structures.

  (1) A graphics processing unit capable of executing a plurality of image processing instructions, a detection unit for detecting an operation by a user of an operation parameter corresponding to the image processing instructions, and the graphics processing unit. An instruction set storage unit that holds a combination of the image processing instructions as an instruction set; and based on a detection result of the detection unit, from the instruction set held by the instruction set storage unit, the graphics processing An information processing apparatus comprising: an instruction set selection unit that selects the instruction set to be executed by a unit.

  (2) The information processing apparatus according to (1), wherein the detection unit includes an initial value storage unit that stores an initial value of the operation parameter, the operation parameter operated by a user, and the initial value storage unit. An information processing apparatus comprising a parameter comparison unit that compares whether or not the initial value held by is different.

  (3) The information processing apparatus according to any one of (1) to (2), wherein the instruction set storage unit holds the instruction set for a combination of the operation parameters. apparatus.

  (4) The detection unit in the information processing apparatus detects an operation parameter operated by the user, and the instruction set selection unit in the information processing apparatus holds the instruction set storage unit based on the detection result by the detection unit. The instruction set to be executed by the graphics processing unit is selected from an instruction set which is a combination of image processing instructions corresponding to the operation parameters and is executed in the graphics processing unit in the apparatus. Information processing method.

  (5) An information processing program for operating a computer including a graphics processing unit capable of executing a plurality of image processing instructions, and detecting a user operation of an operation parameter corresponding to the image processing instruction. , An instruction set storage unit that holds the combination of the image processing instructions executed in the graphics processing unit as an instruction set, and the instruction that the instruction set storage unit holds based on the detection result of the detection unit An information processing program for causing the computer to function as an instruction set selection unit that selects the instruction set to be executed by the graphics processing unit from the set.

DESCRIPTION OF SYMBOLS 10 ... Information processing apparatus 11 ... CPU part 12 ... GPU part 18 ... Operation input device 19 ... Display apparatus 51 ... Initial value memory | storage part 52 ... Parameter comparison part 53 ... Instruction set memory | storage part 54 ... Instruction set selection part

Claims (5)

A graphics processing unit capable of executing multiple image processing instructions;
A detection unit for detecting an operation by a user of an operation parameter corresponding to the image processing command;
An instruction set storage unit that holds a combination of the image processing instructions executed in the graphics processing unit as an instruction set;
Information including an instruction set selection unit that selects the instruction set to be executed by the graphics processing unit from the instruction sets held by the instruction set storage unit based on the comparison result in the parameter comparison unit. Processing equipment. The information processing apparatus according to claim 1,
The detector is
An initial value storage unit for storing initial values of the operation parameters;
An information processing apparatus comprising: a parameter comparison unit that compares whether the operation parameter operated by a user is different from an initial value held by the initial value storage unit. The information processing apparatus according to claim 1,
The instruction set storage unit is an information processing apparatus that holds the instruction set for a combination of the operation parameters. The detection unit in the information processing device detects the operation parameter operated by the user,
The image corresponding to the operation parameter executed by the graphics processing unit in the own device held by the instruction set storage unit, based on the detection result by the detection unit, by the instruction set selection unit in the information processing apparatus. An information processing method for selecting the instruction set to be executed by the graphics processing unit from an instruction set which is a combination of processing instructions. An information processing program for operating a computer including a graphics processing unit capable of executing a plurality of image processing instructions,
A detection unit for detecting an operation by a user of an operation parameter corresponding to the image processing instruction;
An instruction set storage unit that holds a combination of the image processing instructions executed in the graphics processing unit as an instruction set, and the instruction set that the instruction set storage unit holds based on a comparison result in the parameter comparison unit An information processing program for causing the computer to function as an instruction set selection unit that selects the instruction set to be executed by the graphics processing unit.