JP2013225756A - Information processing device and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device and an information processing method, capable of making image processing faster by optimization of a processing range, and omitting rendering processing onto memory even when performing realtime moving image preview.SOLUTION: An information processing device comprises: an image input unit which inputs an image having identification information; an image processing unit which processes the input image; a detection unit which compares the input image used as a first image and the image output from the image processing unit used as a second image and detects an image-processed region; a region information generation unit which generates region information indicating the detected region; a storage unit which associates the generated region information with identification information of the first image and stores the information; and a processing range specification unit which specifies a region to be processed in the first image when the image processing unit reprocesses the first image on the basis of the information stored in the storage unit.

Description

  The present technology relates to an information processing apparatus that performs image effect processing using a GPU (Graphics Processing Unit).

  Also in the field of image processing, the situation where processing that could previously be implemented only with dedicated hardware has been implemented by a program executed on a general-purpose computer has become serious. This is because the processing speed that can satisfy the user can be obtained even with the implementation by the program due to the improvement of the performance of the CPU and the memory.

  Further, in recent years, a technique called GPU (General-Purpose computing on Graphics Processing Units) that uses a graphics rendering GPU incorporated in a computer not only for rendering but also for other numerical operations. There is a field.

  Since GPUs are inexpensive, easy to obtain, and capable of high-speed computation, editing systems incorporating GPUs that perform special effects (effects) of images can be developed at low cost.

  However, simply creating and executing a program that runs on the GPU does not mean that the speed can be easily increased. In order to realize real-time processing of large-size images and image processing with a large amount of calculation, it is necessary to devise image processing algorithms and implementation in the program development.

  Recently, in moving image editing, an invention that refers to a device for optimizing preview display quality for displaying the result of image processing has been disclosed.

  For example, in Patent Document 1, when the editing effect of a moving image being edited is confirmed in real time in the moving image editing system, even if the preview display quality required differs depending on the editing purpose, a preview display corresponding to the editing purpose is performed. A video editing system that can do this is disclosed.

JP 2006-101236 A

  When previewing a moving image that has been subjected to image processing, the processing result of the image processing is generally once rendered on a memory in order to ensure real-time performance. Since rendering on the memory takes an enormous amount of time, the user has to wait for a long time to complete the rendering process when performing a preview operation. In addition, since a lot of memory is used to render a long video material, the system memory for video editing has been under pressure.

  Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique for performing a preview display in accordance with an editing purpose when previewing, but a technique for shortening a waiting time until the preview starts, for example, a rendering process before starting the preview is not performed. No technique for previewing is disclosed.

  In view of the circumstances as described above, the object of the present technology is to speed up image processing by optimizing the processing range, and to omit rendering processing on the memory even when performing real-time video preview. It is to provide an information processing apparatus and an information processing method capable of performing the above.

  (1) In order to achieve the above object, an information processing apparatus according to an aspect of the present technology includes an image input unit that inputs an image having identification information, an image processing unit that processes the input image, The input image is the first image, the image output from the image processing unit is the second image, the first image and the second image are compared, and the region where the processing has been performed A detection unit that detects, a region information generation unit that generates region information indicating the detected region, a storage unit that associates and stores the generated region information and the identification information of the first image, Based on information stored in the storage unit, a processing range specifying unit that specifies a processing target area in the first image when the image processing unit processes the first image again. It comprises.

  In the present technology, the image processing unit first performs image processing on the input image received by the image input unit. Then, the detection unit compares the input image with the processing result image obtained as a result of the image processing, and detects a region where the image processing has been performed. Then, based on the detected area, the area information generation unit generates area information indicating the processing range of the image processing so that the amount of calculation of the image processing can be reduced when the same image processing is performed again on the same image. . The generated region information is stored in the storage unit in association with the identification information of the input image (first image) so that it can be used when reprocessing the same image. When the same image is received again, the processing range specifying unit specifies the processing range of the image processing based on the information acquired from the storage unit. The image processing unit performs image processing only on the specified processing range. Thereby, in this technique, the processing time concerning image processing can be shortened. Also, due to the shortening of the image processing time, rendering processing on the memory can be omitted even when performing a real-time video preview.

  (2) In the information processing apparatus according to an aspect of the present technology, the image processing unit performs processing according to a condition including at least a range designated by the user with respect to the first image. In the first image processing, the processing range is designated by the user. When the process is performed again, the image process is performed without the designation by the user, so that it is possible to save the user from specifying the range at the time of reprocessing.

  (3) Moreover, in the information processing apparatus according to an aspect of the present technology, the processing range specifying unit performs the processing based on the region information associated with the identification information of the first image to be processed again. An area to be processed in the first image is specified. When the same image is received again, the processing range specifying unit acquires the corresponding region information from the storage unit based on the identification information of the input image (first image) in order to specify the processing range of the image processing. Therefore, the area information can be easily acquired.

  (4) In the information processing apparatus according to an aspect of the present technology, the first image is one of a plurality of images continuously changing along a time axis, and the identification information is the first information This is time information on the time axis corresponding to one image. Since an information processing apparatus according to an embodiment of the present technology is assumed to be a video editing apparatus, each frame of a moving image has a time code that is time information on the corresponding time axis. By using this time information as identification information, it is possible to save the trouble of providing identification information separately.

  (5) Moreover, in the information processing device according to an aspect of the present technology, the processing range specifying unit performs the time when the time information of the first image to be processed again is not stored in the storage unit. A region to be processed in the first image is specified based on the region information associated with the time information closest to the information. Since the time information is continuously arranged along the time axis, even if there is no area information associated with the time information of the image to be processed in the storage unit, it is associated with the time information closest to the time information. If there is area information, the area of image processing can be specified using the area information.

  (6) In the information processing apparatus according to an embodiment of the present technology, the image processing unit is a graphics processing unit, and the region information generation unit is based on characteristics of the graphics processing unit. The region information may be generated. According to this configuration, by generating the region information based on the characteristics of the graphics processing unit, the image processing can be optimized according to the characteristics of the graphics processing unit to be used.

(7) In the information processing apparatus according to an aspect of the present technology, the graphics processing unit can execute the image processing by dividing the image processing into a plurality of processing units.
The region information generation unit may generate region information that decomposes the detected region into spatial units of image processing by the graphics processing unit. According to this configuration, area information is created as information obtained by disassembling the detected area into spatial units of image processing by the graphics processing unit, so that image processing in the graphics processing unit is optimized. Can do.

  (8) In the information processing method according to another aspect of the present technology, the image input unit in the information processing apparatus inputs an image having identification information, and the image processing unit in the information processing apparatus The detected image is processed, and the detection unit in the information processing apparatus uses the input image as the first image and the image output from the image processing unit as the second image. And the second image are detected to detect a region where the processing has been performed, and a region information generation unit in the information processing device generates region information indicating the detected region, and the information processing The storage unit in the apparatus stores the generated region information and the identification information of the first image in association with each other, and the processing range specifying unit in the information processing apparatus stores the information stored in the storage unit Based on the above, the image processing unit processes the first image again. Cormorant identify areas of the processing target in the first image in the case.

  As described above, according to the present technology, it is possible to speed up the image processing by optimizing the processing range, and it is possible to omit the rendering process on the memory when performing the real-time moving image preview.

It is a block diagram showing the hardware constitutions of information processing apparatus. It is a figure showing the execution form containing the hardware of application software in a color grading process. It is a figure which shows the specific workflow in a color grading process. It is a flowchart which shows the specific example in a color grading process. It is a figure which shows the specific example in a color grading process. It is a figure which shows the specific example in a color grading process. It is a figure which shows the specific example in a color grading process. It is a flowchart which shows the specific example in a color grading process. It is a figure which shows the specific example in a color grading process. It is a figure which shows the specific example in a color grading process. It is a figure which shows the specific example in a color grading process. It is a flowchart which shows the specific example in a color grading process. It is a figure which shows the specific example in a color grading process. It is a figure which shows the specific example in a color grading process. It is a figure which shows the specific example in a color grading process. It is a functional block diagram of a video processing plug-in program to which the present technology is applied. It is a figure which shows the specific example of an effect process control parameter. It is a figure which shows the example of process area limitation information. It is a flowchart which shows the outline of the whole among the flow of the process in embodiment of this technique. It is a flowchart which shows the part of the "new registration / update of a process area limitation information" process among the process flows in embodiment of this technique. It is a flowchart which shows the part of the "extraction of process area | region limitation information" process among the flow of the process in embodiment of this technique. It is a figure which shows the example of effect effect area | region information. It is a figure which shows the example which divided | segmented effect effect area | region information into the processing unit of GPU kernel. It is an example of a mask image. It is a flowchart which shows the part of a "use of process area limitation information" process among the process flows in embodiment of this technique.

Hereinafter, embodiments according to the present technology will be described with reference to the drawings.
A color grading process is given as an application example of the present technology. First, a hardware configuration using a GPU (Graphics Processing Unit) will be described, then a general color grading process will be described, and finally, a case where the present technology is applied to the color grading process will be described.

[Hardware configuration of information processing device]
FIG. 1 is a block diagram illustrating a hardware configuration of the information processing apparatus 10. Specific examples of the information processing apparatus 10 include a video editing apparatus that performs video editing such as color grading processing.

  The information processing apparatus 10 includes a CPU unit 11, a GPU unit 12, a storage device 13, a display interface 14, an operation interface 15, a network interface 16, and a bus 17 that connects these components to each other.

  The CPU unit 11 includes a CPU 111 and a memory 112 (hereinafter referred to as “CPU memory”), and executes instructions related to various arithmetic processes on the CPU memory 112 by executing a program stored in the CPU memory 112. . The CPU unit 11 interprets a command input by the user through the operation input device 18 connected to the operation interface 15 and reflects it in the operation of the program. For example, the CPU unit 11 can read out image data stored in the storage device 13 and import the image data into the CPU memory 112 to perform processing such as effects on the image data. The image data held in the CPU memory 112 is supplied to the display interface 14, and is rendered as rendering data by being rendered here, and the rendering data of the image processed by the GPU unit 12, which will be described later, as necessary. And are output to the display device 19. Further, the CPU unit 11 performs control so that the processed image data held in the CPU memory 112 is merged with the image data processed by the GPU unit 12 and written back to the storage device 13 as necessary. Is possible.

  The GPU unit 12 includes a GPU 121 and a memory 122 (hereinafter referred to as “GPU memory 122”), and by executing a program stored in the GPU memory 122, image processing such as effects is performed in parallel on the GPU memory 122. It can be executed by arithmetic processing. The image data held in the GPU memory 122 is supplied to the display interface 14 where it is made visible drawing data by drawing processing, and merged with the drawing data of the image processed by the GPU unit 12 as necessary. And output to the display device 19.

  The CPU unit 11 and the GPU unit 12 are connected by a bus 17, and when performing arithmetic processing on video, the CPU memory 112, the bus 17, and the GPU memory 122 are between the CPU unit and the GPU unit 12. Data transfer is performed via

  The display interface 14 is an interface with the display device 19, performs drawing processing of image data supplied from the CPU unit 11 and the GPU unit 12, and draws image drawing data processed by the CPU unit 11 as necessary. The drawing data of the image processed by the GPU unit 12 is merged and supplied to the display device 19 as drawing data of one image. The processing of the display interface 14 is realized by, for example, the above-described GPU 121 or a separately provided GPU (not shown).

  The operation interface 15 is an interface with the operation input device 18, and performs a process of supplying data and commands from the user input from the operation input device 18 to the CPU unit 11.

  The storage device 13 stores, for example, various programs for causing the CPU unit 11 and the GPU unit 12 to store and edit image data before editing and image data after editing.

  The network interface 16 is an interface for connecting to the network 30.

  CPU111 and GPU121 implement | achieve each function block mentioned later. The CPU 111 and the GPU 121 execute programs stored in the CPU memory 112, the GPU memory 122, and the like, and control the respective members as necessary. Thereby, the information processing apparatus 10 of this Embodiment can implement | achieve various functional blocks, and can operate the said each member as the information processing apparatus 10. FIG.

[Execution form of application software]
FIG. 2 is a diagram illustrating an execution form including hardware of application software in a color grading process which is one specific example to which the present technology is applied.

  The application software includes an application main body AP that is a video editing program that reads out, draws, and outputs a video, and a plug-in program that is a subroutine that performs various effect processes on the video. Here, the plug-in program is a video processing plug-in program that applies an effect such as filter processing to the input video.

  A video editing program that is the application main body AP is executed in the CPU unit 11. Plug-in programs are classified into programs that are executed only by the CPU unit 11 and programs that are executed by the CPU unit 11 and the GPU unit 12.

  As shown in the figure, in the processing flow, the application main body AP schedules effect processing to be applied to the video, calls the video processing plug-in program, and edits the video.

  The flow of processing will be described with reference to the drawing. First, the application main body AP reads the video and calls the first video processing plug-in program 1 (PP1). Since the video processing plug-in program 1 (PP1) is a program (CPU video processing plug-in) executed in the CPU unit 11, resources of the GPU unit 12 are not used.

  The upper right of the figure shows details of processing in the CPU video processing plug-in. The application main body AP calls the video processing plug-in program 1 (PP1) and requests image processing. At this time, the image to be processed is placed in the input buffer IBc on the CPU memory 112 for one frame. The called video processing plug-in program 1 (PP1) receives an image from the input buffer IBc, and performs necessary effect processing from effect processing 1 (EP1c) to effect processing N (EPNc). Then, the video processing plug-in program 1 (PP1) places the processing result in the output buffer OBc on the CPU memory 112, and returns the processing to the application main body AP.

  Following the video processing plug-in program 1 (PP1), the application main body AP calls various video processing plug-in programs and proceeds with the processing. In the figure, the plug-in program called last is the video processing plug-in program N (PPN). The video processing plug-in program N (PPN) is a program executed in the CPU unit 11 and the GPU unit 12.

Details of processing in the GPU video processing plug-in are shown in the lower right of the figure. When called from the application main body AP, the video processing plug-in program N (PPN) first transfers an image for one frame from the CPU memory 112 to the input buffer IBg on the GPU memory 122. Next, the video processing plug-in program N (PPN) performs necessary effect processing from the effect processing EP1g to the effect processing EPNg on the image received via the input buffer IBg on the GPU unit 12. Then, the video processing plug-in program N (PPN) places the processing result in the output buffer OBg on the GPU memory 122. Finally, the video processing plug-in program N (PPN) transfers the contents of the output buffer OBg to the CPU memory 112 and returns the processing to the application main body AP.
The present technology is assumed to be applied to the above plug-in program.

[Color grading process workflow]
FIG. 3 is a diagram showing a specific workflow in the color grading process.

  As shown in this figure, first, the application main body AP receives the video material before editing from the previous step. Next, the application main body AP performs effect processing such as “color adjustment”, “keying”, and “defocus / blur” by calling a plug-in program. Next, the application main body AP performs a “composite” process and synthesizes the adjusted video. Here, if the synthesized result is satisfactory, the application main body AP ends the processing so far, and writes the processing result by the “rendering” processing. The exported video is passed to the next process as edited video material.

  If the synthesized video is not satisfactory after the “composite” processing, the processing of the application main body AP returns to the beginning, “color adjustment”, “keying”, “defocus / blur”, etc. The adjustment process is repeated.

  As described above, the color grading process refers to a process of performing color operation or effect processing on the entire image or a partial area. The user performs an editing operation by combining effect processing such as “color adjustment” and “defocus / blur”, “keying processing”, and “composite” that is a mixer processing, and adjusting effect parameters such as luminance values. .

  In other words, in the color grading process, the application main body AP performs a process of applying an effect with slightly changing effect parameters to the same area in the image and a process of causing the user to preview the processing result as a moving image. It has the feature of repeating until it is satisfied with the result. This technology was created by focusing on this feature.

[Specific example of color grading process]
4a to 6d are diagrams showing specific examples in the color grading process.

  First, a specific example of the “color adjustment” process will be described with reference to the flowchart of FIG. 4A and the diagrams of FIGS. 4B to 4D.

  First, the application main body AP uses the keying processing video processing plug-in program for the image before color adjustment shown in FIG. (Step 1, hereinafter abbreviated as S1)

Next, the application main body AP determines whether or not the area selection is appropriate. (S2)
If the area selection is appropriate, the process proceeds to S3. If not appropriate, the process returns to S1. FIG. 4c is an example in which a region is appropriately selected. In the drawing, a portion expressed in white is a portion selected as a region for performing color adjustment processing.

  If it is determined in S2 that the area selection is appropriate, the application main body AP then applies the selection area and color adjustment video processing plug-in program shown in FIG. 4c to the image before color adjustment shown in FIG. 4b. Use this to perform color adjustment processing of the selected area. (S3)

Next, the application main body AP determines whether or not the result of color adjustment for the selected area is appropriate. (S4)
If the result of the color adjustment is appropriate, the “color adjustment” process is terminated. If not, the process returns to S3 and the color adjustment is repeated.

  FIG. 4d is a diagram showing the result of color adjustment. It can be seen that the color of the tree, which was green (represented by the right oblique line in the figure) in FIG. 4b before processing, has been changed to yellow (represented by the left oblique line in the figure).

  Next, a specific example of the “defocus” process will be described with reference to the flowchart of FIG. 5A and the diagrams of FIGS. 5B to 5D.

First, the application main body AP uses the keying processing video processing plug-in program to select an image area to be defocused with respect to the image before the defocus processing shown in FIG. 5B. (S5)
Next, the application main body AP determines whether or not the area selection is appropriate. (S6)
If the area selection is appropriate, the process proceeds to S7. If not appropriate, the process returns to S5. FIG. 5c is an example in which a region is appropriately selected. In the drawing, a portion expressed in white is a portion selected as an area to be defocused (blurred).

  If it is determined in S6 that the area selection is appropriate, the application main body AP then selects the selection area and defocus video processing plug-in program shown in FIG. 5c for the image before the defocus processing shown in FIG. 5b. Is used to defocus the selected area. (S7)

Next, the application main body AP determines whether or not the result of the defocus processing for the selected area is appropriate. (S8)
If the result of the defocus process is appropriate, the “defocus” process is terminated. If not appropriate, the process returns to S7 and the defocus process is repeated.

  FIG. 5d shows the result of the defocus processing. In FIG. 5b before processing, the boundary line of the cloud is clear (represented by a thick solid line in the figure), but is changed to blurred (represented by a thick dotted line in the figure).

  Next, a specific example of the “composite” process will be described with reference to the flowchart of FIG. 6A and the diagrams of FIGS. 6B to 6D.

First, the application main body AP acquires an image after the color adjustment processing shown in FIG. 6B, and performs a keying process on a portion other than a portion to which clouds are added, using a video processing plug-in program. (S9)
In the figure shown in FIG. 6b, it can be seen that the cloud portion is not blacked because it is not keyed.

Next, the application main body AP acquires the image after the defocus processing shown in FIG. 6c, and performs a keying process on only the cloud portion using the video processing plug-in program. (S10)
In the diagram shown in FIG. 6c, since only the cloud part is keyed, it can be seen that the part other than the cloud is black.

Next, the application main body AP performs image addition processing using the video processing plug-in program, and ends the processing. (S11)
In the figure shown in FIG. 6d, the result of color adjustment and the result of defocus processing are synthesized by adding the non-cloud part keyed in FIG. 6b and the cloud part keyed in FIG. 6c. Has been.

  As described above, in the color grading process, there is a process of changing only the effect parameter in the area without changing the image area in which the effect is reflected in the effect process being changed. In addition, an effect function whose processing content (luminance value manipulation, color replacement, etc.) is determined in the middle of editing work has a feature that an image area in which the effect effect is reflected is determined in each frame. This technology was created by focusing on this feature.

[Outline of this technology]
The present technology proposes a technique for optimizing a range where effect processing is performed in processing in which user operations are repeatedly generated, and realizing improvement in sensation speed and increase in processing speed for each operation.

  The present technology may be realized inside a video processing plug-in program. Each plug-in program detects an area where an effect is applied in an image, and based on the detection result (hereinafter referred to as effect effect area information), a device (CPU, GPU, etc.) that executes video processing Processing time for effect processing is reduced by optimizing device resources and limiting the image processing range.

  Since the present technology may be realized as a part of the video editing system, in this case, a time code is attached to each frame of the material video as identification information. That is, the input image (first image) is one of a plurality of images that continuously change along the time axis, and the identification information is on the time axis corresponding to the input image (first image). Time code (time information). When a plurality of moving images are edited at the same time, a sequence ID is used to distinguish each moving image.

  Time code and sequence ID, and general information for limiting the processing range of the image and controlling the amount of calculation of the effect processing (for example, a mask image, an argument given to the effect processing function, etc .; hereinafter referred to as processing region limited information) ) Is stored as a table (hereinafter referred to as an effect processing control table).

  When effect processing is executed, each plug-in program refers to the effect processing control table to extract processing area limitation information and restricts the area where effect processing is performed. Improve speed and increase effect processing speed. Since device resource optimization is performed, processing can be performed as fast as possible according to the relationship between the number of effects to be executed and the amount of device resources.

  Further, the present technology provides a high-speed preview operation at the actual resolution by increasing the effect processing speed. Since the preview operation is speeded up, the present technology does not require a memory rendering operation for performing the preview in real time. Therefore, there is an effect that the system memory for video editing is not compressed.

[Functional block diagram of this technology]
FIG. 7 is a functional block diagram of a video processing plug-in program 200 to which the present technology is applied.

  The video processing plug-in program 200 includes an input buffer 210, an effect processing unit 220, an output buffer 230, and a processing area determination unit 240 as functional blocks. In addition, the processing region determination unit 240 includes a difference detection unit 241, a processing region limitation information extraction unit 242, and an effect processing control table 243.

  The effect processing unit 220 is executed by either the CPU 111 or the GPU 121, and the input buffer 210 and the output buffer 230 are on the CPU memory 112 when the effect processing unit 220 is executed on the CPU 111, and the effect processing unit 220. Is executed on the GPU 121, it is on the GPU memory 122. The processing area determination unit 240 is executed on the CPU 111.

  The input buffer 210 receives the input image passed from the application main body AP and passes it to the effect processing unit 220 and the difference detection unit 241.

  The effect processing unit 220 performs effect processing on the input image passed from the input buffer 210 and passes the output image as the processing result to the output buffer 230. Examples of effect processing include keying processing, color adjustment processing, defocus / blur processing, and the like. The effect processing unit 220 makes an inquiry to the effect processing control table 243 based on the time code and sequence ID of the frame that is the input image. Then, the effect processing unit 220 acquires processing area limitation information corresponding to the time code and sequence ID of the input image, and performs effect processing based on the acquired processing area limitation information.

  The output buffer 230 passes the output image passed from the effect processing unit 220 to the difference detection unit 241 and passes it to the application main body AP when the processing of the video processing plug-in program ends and the processing returns to the application main body AP. .

  The difference detection unit 241 compares the input image before effect processing passed from the input buffer 210 with the output image after effect processing passed from the output buffer 230, and detects a difference. Then, the difference detection unit 241 passes the detected difference to the processing region limitation information extraction unit 242 as effect effect region information.

  The processing area limitation information extraction unit 242 converts the effect effect area information received from the difference detection unit 241 into processing area limitation information, and passes the converted processing area limitation information to the effect processing control table. Details of the conversion from the effect effect area information to the processing area limited information will be described later.

  The effect processing control table 243 associates the time code or sequence ID of the input image received from the effect processing unit 220 with the processing region limitation information received from the processing region limitation information extraction unit 242 and stores it as an effect processing control parameter. To do. When the effect processing unit 220 inquires about the effect processing control parameter using the time code and sequence ID of the input image as keys, if the corresponding time code and sequence ID exist, the effect processing control table 243 displays the time Processing area limitation information corresponding to the code and sequence ID is returned to the effect processing unit 220.

  A specific example of the effect processing control parameter is shown in FIG. In this example, a plurality of processing area limitation information is associated with one combination of time code and sequence ID, indicating that a plurality of processing area limitation information is used for effect processing at a time.

  In addition, since processing area limitation information is associated with a combination of one time code and sequence ID, even if the area where effect processing is performed changes for each frame, high-speed processing is possible if the effect has been operated once. It becomes possible.

The information processing apparatus according to the present embodiment includes an input buffer 210 (image input unit) that inputs an image having a time code (identification information), and an effect processing unit 220 (image) that processes the input image. Processing unit), the input image as a first image, and the image output from the effect processing unit 220 (image processing unit) as a second image, the first image and the second image A difference detection unit 241 (detection unit) that detects a region in which the processing is performed, and a processing region limitation information extraction unit 242 (region information generation unit) that generates region information indicating the detected region; An effect processing control table 243 (storage unit) that stores the generated region information and the identification information of the first image in association with each other;
Based on the information stored in the effect processing control table 243 (storage unit), the first image in the case where the effect processing unit 220 (image processing unit) processes the first image again. The image processing apparatus may include an effect processing unit 220 (processing range specifying unit) that specifies a processing target area in an image.

[Processing area limited information]
FIG. 9 is a diagram illustrating an example of the processing area limitation information. The processing area limitation information refers to general information that limits the processing range of an image and controls the calculation amount of effect processing. The example shown on the left side of the figure is a mask image, and the example shown on the right side of the figure is a GPU execution parameter.

  The mask image aligns the effect effect area information with the processing unit of image processing (the division unit of threads in the GPU 121) that handles the effect effect area information in the effect processing unit 220 in order to cause the effect processing to appropriately operate on the GPU 121. This has the effect of suppressing the generation of unnecessary GPU threads during processing and shortening the computation time.

  The alignment method will be described later. As an outline, the GPU 121 (graphics processing unit) can divide and execute effect processing (image processing) into a plurality of processing units, and a processing region limited information extraction unit 242 (region information generation unit). This means that region information that decomposes the region detected by the difference detection unit 241 into spatial units of image processing by the GPU 121 (graphics processing unit) is generated.

  The GPU runtime parameters include the number of massively parallel processes and GPU kernel execution parameters.

[Processing flow (overall)]
FIG. 10 is a flowchart showing an overall outline of the flow of processing according to the embodiment of the present technology. This flowchart shows processing after the video processing plug-in program 200 is called from the application main body AP.

  First, the effect processing unit 220 executes the first effect process. (S100)

  The first effect process is performed on a range designated by the user. At this point, since the processing area limitation information has not been created, the effect processing unit 220 executes the effect process without using the processing area limitation information.

  Next, the video processing plug-in program 200 determines whether or not there is a change in the processing area for effect processing. (S101)

  If there is a change in the processing area of the effect process, the process proceeds to S102, and if there is no change, the process proceeds to S104.

  The case where there is a change in the processing area of the effect processing is when an effect parameter change occurs that changes the processing area in the image. In color grading processing, the case where the key image is changed is equivalent. To do. It should be noted that the condition for proceeding to the processing of S102 is that the image operation to the effect effect processing area occurs more than the specified number of times specified by the user for an image having the same combination of time code and sequence ID, or the previous step In the effect processing, the processing content may be changed to affect the processing range of the subsequent process.

  If it is determined in S101 that there is a change in the processing area of the effect processing, then the difference detection unit 241 has processed the input image obtained from the input buffer 210 and the effect processing unit 220 obtained from the output buffer 230. The difference with the effect processing result is detected. (S102)

  Next, the video processing plug-in program 200 determines whether or not a difference is detected in the processing of S102. (S103)

  If a difference is detected, the process proceeds to S200. If a difference is not detected, the process proceeds to S104. If no difference is detected, the video processing plug-in program 200 recognizes that there is no effect parameter change by the user, and does not perform update processing (see S205) to the effect processing control table 243, or the current A configuration may be used in which processing area limitation information associated with a time code past the time code of the image is written in the effect processing control table 243.

  If it is determined in S103 that a difference has been detected, the video processing plug-in program 200 next performs “new registration / update of processing area limitation information” processing, and proceeds to processing in S104. (S200)

  Details of the “new registration / update of processing area limitation information” processing will be described later using another flowchart.

  If it is determined in S103 that there is no difference, or after the processing of S200 is completed, the video processing plug-in program 200 next determines whether to use the processing area limitation information. (S104)

  This determination may be based on, for example, an instruction from the user for each effect operation, or a threshold is set in advance, and the processing region limitation information is used when the number of effect processing exceeds the threshold. Also good. When the processing area limitation information is used, the process proceeds to S300. When the processing area limitation information is not used, the process is terminated and the control is returned to the application main body AP.

  If it is determined in step S104 that the processing area limitation information is to be used, the video processing plug-in program 200 performs a “use of processing area limitation information” process. (S300)

  Details of the “use of processing area limitation information” process will be described later using another flowchart.

  In the processing after S300, the effect processing is executed while restricting the area where the effect processing is performed based on the effect processing control parameter recorded in the effect processing control table 243.

  In the process of S300, after acquiring the processing area limitation information to be used, next, the effect processing unit 220 executes the effect process again. (S105)

  In the execution of the effect process this time, the effect process is performed using the processing area limitation information, so that the execution of the process is accelerated.

  As described above, in the effect processing that is repeatedly operated, the processing area limitation information that is a parameter for reducing the amount of calculation is extracted, and the effect processing is performed by using the extracted processing area limitation information. Response improvement is realized.

[Flow of processing (new registration / update of processing area limited information)]
FIG. 11 is a flowchart illustrating a part of the “new registration / update of processing area limitation information” processing in the processing flow according to the embodiment of the present technology.

  First, the processing region limitation information extraction unit 242 performs “processing region limitation information extraction” processing. (S400)

  Details of the “extract processing area information” process will be described later with reference to another flowchart.

  Next, the video processing plug-in program 200 acquires the time code of the input image. (S201)

  Next, the video processing plug-in program 200 acquires the sequence ID of the input image. (S202)

Next, the video processing plug-in program 200 searches the effect processing control table 243 using the acquired time code and sequence ID, and determines whether or not there is processing area limitation information having the same time code and sequence ID. to decide. (S203)
When there is processing area limitation information having the same time code and sequence ID, the video processing plug-in program 200 proceeds to the process of S204, and when there is no processing area limitation information having the same time code and sequence ID, The process proceeds to S205.

  If it is determined in S203 that there is processing area limitation information having the same time code and sequence ID, then the video processing plug-in program 200 converts the contents of the effect processing control table 243 into the time code and sequence ID. , And updated with the processing area limitation information extracted in S400. After updating, the process ends. (S204)

  If it is determined in S203 that there is no processing area limitation information having the same time code and sequence ID, the video processing plug-in program 200 then sends the time code and sequence ID to the effect processing control table 243. , The processing area limitation information extracted in S400 is linked and newly registered as an effect processing control parameter. After registration, the process ends. (S205)

  Note that, instead of the above time code, a frame number of an image to be processed may be used.

  As can be seen from the processing of S101, the effect processing control parameters in the effect processing control table 243 are created and updated for each image in which the processing area of the effect processing has changed.

[Processing flow (extract processing area limited information)]
FIG. 12 is a flowchart illustrating a portion of the “extract processing area limitation information” process in the process flow according to the embodiment of the present technology. Although some of the processing area limitation information is used in the CPU 111, an example of extracting the processing area limitation information used in the GPU 121 is shown here. As an outline of the flow of extraction, the effect processing unit 220 (image processing unit) is a GPU 121 (graphics processing unit), and the processing region limited information extraction unit 242 (region information generation unit) is based on the characteristics of the GPU. In other words, processing area limited information (area information) is extracted (generated).

  In the following description, examples of information and images used in each step in the flowchart are shown and described with reference to FIGS. 13a to 13c.

  First, the processing area limited information extraction unit 242 acquires effect effect area information from the difference detection unit 241. (S401)

  FIG. 13A is an example of effect effect area information. In this example, the difference between the image before and after the color adjustment processing is taken for the tree and cloud images used when the specific example of the color grading processing is shown. In the color adjustment process, only the color of the tree is changed from green to yellow, so in this effect area information, only the part where the color was changed becomes white, and the part where the color was not changed It is black.

  Next, the processing area limitation information extraction unit 242 divides the effect area information for each processing unit of the GPU kernel. (S402)

  The processing area limitation information extraction unit 242 holds the value of the processing unit of the GPU kernel in advance, and the effect effect area information acquired in the processing of S401 based on the value of the processing unit of the GPU kernel is, for example, a lattice shape Divide into two.

  FIG. 13B is an example in which effect effect area information is divided and aligned into processing units of the GPU kernel. In this example, the effect effect area information is divided into a grid pattern. Each square of the lattice corresponds to a GPU kernel processing unit (GPU thread).

  Next, the processing area limitation information extraction unit 242 determines whether a mask image is necessary. (S403)

  If the white part in the effect effect area information, that is, the part to which the effect is applied is complicated, and it must be designated as an image as shown in FIG. 13a, it is determined that a mask image is necessary. When the white portion in the effect area information is a simple square, it is determined that the mask image is not necessary and that the runtime parameters should be set.

  When it is determined that a mask image is necessary, the processing area limitation information extraction unit 242 proceeds to the process of S404, and when it is determined that a mask image is not necessary, the processing area limitation information extraction unit 242 proceeds to the process of S405. .

  If it is determined in S403 that a mask image is necessary, the processing area restriction information extraction unit 242 outputs the effect effect area information division result as a mask image to the effect processing control table 243, and ends the process. (S404)

FIG. 13c is an example of a mask image. In this example, a portion that is a white block represents a portion in which the white portion of the effect effect region information is included in the square of the lattice as a result of dividing the effect effect region information into a lattice shape. This white block portion means that the GPU kernel performs effect processing in the GPU 121.
A black block portion represents a portion in which the white portion of the effect effect region information is not included in the square of the lattice as a result of dividing the effect effect region information into a lattice shape. For the black block portion, the GPU kernel does not perform effect processing in the GPU 121. Thus, by creating a mask image, unnecessary operations of threads in the GPU can be reduced.

  If it is determined in S403 that a mask image is not necessary, then the processing area limitation information extraction unit 242 sets the number of massively parallel processes and kernel execution parameters suitable for the effect effect area information division result. The process is terminated. (S405)

  By adjusting the number of massively parallel processing divisions in the GPU 121, generation of unnecessary threads in the GPU 121 can be prevented.

[Flow of processing (use of processing area limited information)]
FIG. 14 is a flowchart illustrating a portion of the “use of processing area limitation information” process in the process flow according to the embodiment of the present technology. As a configuration using the processing area limited information, the video processing plug-in program 200 (processing range specifying unit) is associated with the time code and sequence ID (identification information) of the input image (first image) to be processed again. Based on the attached processing area limitation information (area information), the area to be processed in the input image (first image) may be limited (specified).

  First, the video processing plug-in program 200 acquires the time code and sequence ID of the image currently being worked on. (S301)

  Next, the video processing plug-in program 200 refers to the effect processing control table 243 using the acquired time code and sequence ID. (S302)

  Next, the video processing plug-in program 200 determines whether or not the effect processing control table 243 is empty based on the reference result. (S303)

  When it is determined that the effect processing control table 243 is empty, the video processing plug-in program 200 proceeds to the processing of S304, and when it is determined that the effect processing control table 243 is not empty, the processing proceeds to S307.

  If it is determined in S303 that the effect processing control table 243 is empty, the video processing plug-in program 200 next processes the processing area associated with the same time code and sequence ID as the acquired time code and sequence ID. It is determined whether or not the limited information exists in the effect processing control table 243. (S304)

  When there is processing area limitation information associated with the same time code and sequence ID as the acquired time code and sequence ID, the video processing plug-in program 200 proceeds to the processing of S305, and when it does not exist, the processing of S306 is performed. Proceed to

  If it is determined that there is processing area limitation information associated with the same time code and sequence ID as the time code and sequence ID acquired in S304, the video processing plug-in program 200 next executes the effect processing control table 243. The subsequent effect processing is performed using the processing area limitation information existing in (S305)

  If it is determined that there is no processing area limitation information associated with the same time code and sequence ID as the time code and sequence ID acquired in S304, the video processing plug-in program 200 then executes the processing area limitation information. Is interpolated and the subsequent effect processing is performed. (S306)

  As a method of interpolating the processing area limitation information, the processing area limitation information linked to the frames before and after the frame that is the currently processed image may be used. Only the attached processing area limitation information may be used. That is, the video processing plug-in program 200 (processing range specifying unit) does not store the time code (time information) of the input image (first image) to be processed again in the effect processing control table 243 (storage unit). In this case, the region to be processed in the input image (first image) is determined based on the processing region limitation information (region information) associated with the time code (time information) closest to the time code (time information). Limit (specify).

  If it is determined in S303 that the effect processing control table 243 is empty, the video processing plug-in program 200 next determines whether to create an effect processing control parameter to be newly registered in the effect processing control table 243. . (S307)

  If an effect process control parameter to be newly registered in the effect process control table 243 is not created, the process proceeds to S308, and if created, the process proceeds to S200. Whether or not to create an effect process control parameter to be newly registered in the effect process control table 243 may be determined based on the contents of the effect process, or may be determined based on a user designation.

  If it is determined in S307 that an effect processing control parameter to be newly registered in the effect processing control table 243 is not to be created, the video processing plug-in program 200 then uses the processing region limitation information of the default value to use the processing region limitation information. Used for subsequent effect processing. (S308)

  The default value processing area limitation information is, for example, processing the entire image equally.

  If it is determined in S307 that an effect processing control parameter to be newly registered in the effect processing control table 243 is to be created, the video processing plug-in program 200 then performs a “new registration / update of processing area limitation information” process. (S200)

  The “new registration / update of processing area limitation information” processing performed here is the same as described above.

  Next, the video processing plug-in program 200 performs the following effect processing as processing region limitation information using processing region limitation information obtained by the “new registration / update of processing region limitation information” process. (S309)

[Differences from the prior art]
Conventionally, in order to increase the perceived speed when operating the video editing device, work was performed while previewing the proxy image with effect processing, and the final effect effect was confirmed at the actual resolution (full resolution) For, once the effect processing results were rendered in memory, a preview was performed.

  That is, in order to preview the result of effect processing on a real resolution video in real time, rendering has to be performed once on a memory, which requires a huge amount of time.

  Rendering in memory requires a memory capacity equivalent to the video sequence. If the resolution of the material video is increased, the length of the renderable scale will be shortened (long video sequences cannot be rendered). ).

In addition, there is a problem that there is a difference in the effect processing result to be previewed between the proxy image and the full-resolution image.
Also, if you change any part of the effect processing effect parameters, it will be necessary to re-render the video sequence part to which the effect process is reflected. There was a problem of becoming.

  The influence of the effect parameter change in the effect processing on the moving image sequence and other effect processing differs depending on the contents of the changed effect parameter and the contents of the effect processing. For this reason, even for effect processing that normally does not need to be re-rendered, re-rendering processing occurs each time the final effect effect is confirmed, and it may take a long time to preview.

  In the present technology, since the time for effect processing is shortened, when previewing a moving image, it is not necessary to render the result of the effect processing once in the memory as before. Therefore, it is not necessary to re-render even when the effect parameter is changed. It is also possible to preview the effect processing result at the actual resolution.

[Other configurations of this technology]
In addition, this technique can also take the following structures.
(1) An image input unit that inputs an image having identification information, an image processing unit that processes the input image, a first image that is the input image, and an image that is output from the image processing unit As a second image, the first image and the second image are compared to detect a region where the processing has been performed, and a region that generates region information indicating the detected region An information generation unit, a storage unit that stores the generated region information and the identification information of the first image in association with each other, and the first image based on the information stored in the storage unit. An information processing apparatus comprising: a processing range specifying unit that specifies a region to be processed in the first image when the image processing unit performs processing again.
(2) The information processing apparatus according to (1), wherein the image processing unit performs processing according to a condition including at least a range designated by a user with respect to the first image.
(3) In the information processing apparatus according to (1) or (2), the processing range specifying unit may store the region information associated with the identification information of the first image to be processed again. An information processing apparatus that identifies an area to be processed in the first image.
(4) The information processing apparatus according to any one of (1) to (3), wherein the first image is one of a plurality of images that continuously change along a time axis. And the identification information is time information on a time axis corresponding to the first image.
(5) The information processing apparatus according to (4), wherein the processing range specifying unit performs the processing again for the time information of the first image. Is not stored in the storage unit, an information processing apparatus that identifies a processing target area in the first image based on the area information associated with the time information closest to the time information.
(6) In the information processing apparatus according to any one of (1) to (5), the image processing unit is a graphics processing unit, and the region information generation unit is An information processing apparatus that generates the area information based on characteristics of a graphics processing unit.
(7) In the information processing device according to (6), the graphics processing unit can execute the image processing by dividing the image processing into a plurality of processing units. An information processing apparatus that generates area information that decomposes the detected area into spatial units for image processing by the graphics processing unit.
(8) An image input unit in the information processing apparatus inputs an image having identification information, and an image processing unit in the information processing apparatus performs processing of the input image, and detection in the information processing apparatus A first image as the input image and a second image as the image output from the image processing unit, and the first image and the second image are compared to perform the process. A region information generation unit in the information processing device generates region information indicating the detected region, and a storage unit in the information processing device detects the generated region information and the region information The identification information of the first image is stored in association with each other, and the processing range specifying unit in the information processing apparatus stores the image on the first image based on the information stored in the storage unit. Processing target area in the first image when the processing unit performs the processing again Information processing method to identify.

DESCRIPTION OF SYMBOLS 10 ... Information processing apparatus 11 ... CPU part 12 ... GPU part 17 ... Bus 18 ... Operation input device 19 ... Display apparatus 111 ... CPU
112 ... CPU memory 121 ... GPU
122 ... GPU memory 200 ... Video processing plug-in program 210 ... Input buffer 220 ... Effect processing unit 230 ... Output buffer 240 ... Processing region determination unit 241 ... Difference detection unit 242 ... Processing region limited information extraction unit 243 ... Effect processing control table

Claims (8)

An image input unit for inputting an image having identification information;
An image processing unit for processing the input image;
The input image is a first image, the image output from the image processing unit is a second image, and the first image and the second image are compared to perform the processing. A detection unit for detecting
A region information generating unit that generates region information indicating the detected region;
A storage unit for storing the generated region information and the identification information of the first image in association with each other;
Based on information stored in the storage unit, a processing range specifying unit that specifies a processing target area in the first image when the image processing unit processes the first image again. An information processing apparatus comprising: The information processing apparatus according to claim 1,
The image processing unit performs processing according to a condition including at least a range designated by a user for the first image. The information processing apparatus according to claim 1,
The processing range specifying unit specifies a processing target area in the first image based on the area information associated with the identification information of the first image to be processed again. The information processing apparatus according to claim 1,
The first image is one of a plurality of images continuously changing along a time axis,
The information processing apparatus, wherein the identification information is time information on a time axis corresponding to the first image. The information processing apparatus according to claim 4,
When the time information of the first image to be processed again is not stored in the storage unit, the processing range specifying unit is based on the region information associated with the time information closest to the time information. An information processing apparatus that identifies a region to be processed in the first image. The information processing apparatus according to claim 1,
The image processing unit is a graphics processing unit,
The area information generation unit is an information processing apparatus that generates the area information based on characteristics of the graphics processing unit. The information processing apparatus according to claim 6,
The graphics processing unit can execute the image processing by dividing it into a plurality of processing units.
The area information generation unit generates area information that decomposes the detected area into spatial units for image processing by the graphics processing unit. An image input unit in the information processing apparatus inputs an image having identification information,
An image processing unit in the information processing apparatus processes the input image,
The detection unit in the information processing apparatus uses the input image as a first image, the image output from the image processing unit as a second image, and the first image and the second image. Compare and detect the area where the above processing was performed,
A region information generation unit in the information processing device generates region information indicating the detected region;
A storage unit in the information processing apparatus stores the generated region information and the identification information of the first image in association with each other;
The first image in the case where the processing range specifying unit in the information processing apparatus performs the process again on the first image based on the information stored in the storage unit Processing method for specifying a region to be processed in.

Images