JP2008268725A - Information processing apparatus and method of increasing resolution of image data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processing apparatus capable of making resolution high through less frame reference and dynamically adjusting the processing load of the increase in resolution. <P>SOLUTION: An edge detection unit 202 acquires edges from a low-resolution image, and a corresponding point search unit 204 searches for a predetermined number of corresponding points having a pattern close to a luminance pattern of one low-resolution pixel and pixels at its periphery as to the acquired edges. Then a high-resolution pixel value calculating unit 206 calculates pixel values of the high-resolution image by using pixel values of the low-resolution image and information on the predetermined number of corresponding points. Further, an edge counting unit 211 counts the total number of edges detected by frames, and a parameter/processing algorithm setting unit 212 estimates a processing load from the total number of edges and suitably executes changing processing for parameters and processing algorithm to be applied to the high-resolution pixel value calculating unit 206. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for increasing the resolution of image data suitable for application to an information processing apparatus such as a personal computer.

  In recent years, various types of personal computers such as a notebook type and a desktop type are widely used. Many of these types of personal computers reproduce moving image data with software. Recently, software that provides a function for generating image data with higher resolution than input image data has also appeared. ing.

For example, the multi-frame inverse transform method focuses on the fact that the subject in the reference frame is also in another frame, and by detecting the subject's movement with high accuracy below the pixel interval, A plurality of sample values whose positions are slightly shifted with respect to the local part of the image are obtained to increase the resolution (for example, see Patent Document 1).
JP 2001-188680 A

  By the way, in the multiple frame degradation inverse transformation method, when low resolution frames are arranged in time series, they can be converted into high resolution frames sequentially, but in order to obtain a sufficient number of sample values, A large number of low resolution images are required, and there is a problem that the amount of memory and the amount of processing increase.

  In particular, when the resolution of image data is increased by software operating on a personal computer, for example, the contents of the processing for increasing the resolution are dynamically controlled so that the occupied resources fall within a predetermined range. Need also arises.

  The present invention has been made in consideration of such circumstances, and can perform high resolution with few frame references and can dynamically adjust the processing load for high resolution. It is an object to provide an apparatus and a method for increasing the resolution of image data.

  In order to achieve the above object, an information processing apparatus according to the present invention includes an input unit that inputs image data having a first resolution composed of one or more frames in which the luminance of each pixel is expressed as a pixel value, and the image Detecting means for detecting an attention image area in which a difference in pixel value between adjacent pixels is within a threshold selected from within one frame selected from the data as a reference frame or within two or more frames including the reference frame; Search means for searching for a corresponding image area that approximates a change pattern of pixel values between pixels in each target image area for each target image area detected by the means, and a pixel of each pixel of the reference frame Higher than the first resolution corresponding to the reference frame based on the value and information of the corresponding image area searched by the search means A resolution increasing means for generating image data of two resolutions, and the number of corresponding image areas to be searched by the search means for each of the target image areas according to the number of target image areas detected by the detection means. And setting means.

  The image data resolution increasing method according to the present invention is a method for increasing the resolution of image data using an information processing apparatus including a data input means, a data output means, and a central processing unit. Image data of a first resolution consisting of one or more frames represented as pixel values is input from the data input means, and is included in one frame selected as a reference frame from the image data or including the reference frame 2 A target image area in which the difference in pixel value between adjacent pixels in the frame is equal to or greater than a threshold is detected by the central processing unit, and the pixels in each target image area are determined according to the number of detected target image areas. The number of search for corresponding image areas that approximate the change pattern of pixel values is set by the central processing unit, and each detected image area of interest is set. And searching for the corresponding image area by the central processing unit from the reference frame by the set number, and based on the pixel value of each pixel of the reference frame and the information of the searched corresponding image area Image data having a second resolution higher than the first resolution corresponding to a frame is generated by the central processing unit and output from the data output means.

  According to the present invention, it is possible to provide an information processing apparatus and a method for increasing the resolution of image data that can increase the resolution with reference to a small number of frames and can dynamically adjust the processing load of the increase in resolution. be able to.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows a configuration example of the information processing apparatus according to the present embodiment. This information processing apparatus is realized as, for example, a notebook personal computer that can be driven by a battery.

  As shown in FIG. 1, this computer includes a CPU 11, a north bridge 12, a main memory 13, a graphics controller 14, a VRAM 14A, an LCD 15, a south bridge 16, a BIOS-ROM 17, an HDD 18, a USB controller 19, a sound controller 20, and a speaker 21. A digital TV tuner 22, an embedded controller / keyboard controller IC (EC / KBC) 23, a keyboard 24, a touch pad 25, a power circuit 26, a battery 27, a network controller 28, and the like.

  The CPU 11 is a processor that controls the operation of each unit in the computer. The CPU 11 executes an operating system (OS) 100 loaded from the HDD 18 to the main memory 13 and various application programs including utilities that operate under the control of the OS 100. The various application programs include a high resolution application 200 described later. The CPU 11 also executes the BIOS stored in the BIOS-ROM 17. The BIOS is a program for controlling various hardware.

  The north bridge 12 is a bridge device that connects the local bus of the CPU 11 and the south bridge 16. The north bridge 12 has a function of executing communication with the graphics controller 14 via a bus, and also includes a memory controller that controls access to the main memory 13. The graphics controller 14 is a display controller that controls the LCD 15 used as a display monitor of the computer. The graphics controller 14 generates a display signal to be sent to the LCD 15 from the image data written in the VRAM 14A.

  The south bridge 16 is a controller that controls various devices on the PCI bus and the LPC bus. The south bridge 16 is directly connected to the BIOS-ROM 17, the HDD 18, the USB controller 19, the sound controller 20, and the digital TV tuner 22, and has a function of controlling them. The HDD 18 is a storage device that stores various software and data. The USB controller 19 is a controller that controls various USB devices connected by a cable. The sound controller 20 is a sound source controller that controls the speaker 21. The digital TV tuner 22 is a receiving device for receiving digital TV broadcast program data from the outside.

  The EC / KBC 23 is a one-chip microcomputer in which an embedded controller for power management and a keyboard controller for controlling the keyboard 24 and the touch pad 25 are integrated. The EC / KBC 23 cooperates with the power supply circuit 26 to control supply of power from the battery 27 or the external AC power supply to each unit. The network controller 28 is a communication device that executes communication with an external network such as the Internet.

  The high resolution application 200 operating on the computer having such a hardware configuration increases the resolution of image data input from the HDD 18, USB controller 19, digital TV tuner 22, network controller 28, etc., and the HDD 18, USB controller 19. , A program for increasing the resolution of so-called image data output from the LCD 15 or the like.

  The CPU 11 and the main memory 13 which are resources of the computer 1 are used for increasing the resolution of the image data by the resolution increasing application 200. The load given to the CPU 11 and the work area that can be secured on the main memory 13 are limited. Therefore, the high resolution application 200 can increase the resolution with reference to a small number of frames, and can dynamically adjust the processing load of the high resolution, which will be described in detail below. Describe. FIG. 2 is a diagram showing a functional configuration of the high resolution application 200.

  The high resolution by the high resolution application 200 is a new focus on the fact that when a part of a subject is taken out and the change pattern of the luminance value is seen, the same pattern exists in the vicinity. In other words, the resolution enhancement by the resolution enhancement application 200 is to detect a plurality of sample values for the same luminance value change pattern in one frame and to detect their positions. First, the basic principle of increasing the resolution of image data executed by the resolution increasing application 200 will be described.

  The buffer unit 201 stores the input low resolution image. The low resolution image may be a moving image or a still image. The target effect can be obtained even if the processing is not limited to the periphery of the edge region. However, in regions other than the edge, low-load processing such as bilinear interpolation or cubic convolution is sufficient. In many cases, a correct image is obtained. Therefore, here, edge detection is performed and processing is limited to the periphery of the edge region, thereby obtaining a high-quality image with a low processing load.

  The edge detection unit 202 acquires an edge from the low-resolution image on the buffer unit 201 and stores information about the edge (for example, a binary image) in the buffer unit 203. As a method for acquiring the edge, for example, a Sobel filter or the like can be used.

  The corresponding point search unit 204 refers to the low-resolution image of the buffer unit 201 for the area determined as the edge stored in the buffer unit 203, and has a pattern in which one low-resolution pixel and the surrounding luminance pattern are close to each other. A plurality of corresponding points are searched, and information on the corresponding points is stored in the buffer unit 205. For the correlation of the luminance pattern, for example, a sum of absolute differences or a sum of squared differences can be used. Further, as a search method for obtaining a plurality of corresponding points, a method of searching for corresponding points by setting a search range in units of one pixel in the X-axis or Y-axis direction and changing the other axis component can be used. In this case, for example, several points obtained by changing the other axis component are selected as candidates for highly correlated points, and the coordinates of the corresponding points in sub-pixel units are selected from the candidates by a method such as parabolic fitting. Can be estimated.

  Then, the high resolution pixel value calculation unit 206 calculates a high resolution pixel value using the low resolution pixel value of the buffer unit 201 and information on a plurality of corresponding points of the buffer unit 205. As this specific method, for example, the POCS method (see P. 29 of S. Park, et.al. “Super-Resolution Image Reconstruction: A Technical Overview”) can be used.

  In the POCS method, a pixel value of a temporary high-resolution image is obtained in advance by bilinear interpolation or cubic convolution. For a pixel of interest in a low-resolution image, the sample value (sub-pixel unit) at the corresponding point of that point needs to be reproduced as a set of pixel values of pixels arranged in units of one pixel in the high-resolution image. Pixel values generally do not satisfy it. With this sample value, when the sample value obtained by mapping the pixel to the high resolution image is used as the temporary sample value, the temporarily obtained pixel value does not match the temporary sample value. It is obtained and updated by adding and subtracting the temporary pixel value so as to eliminate the difference. Since this update is performed also on the target pixel in the vicinity, the temporary pixel value changes from the correct value under the influence. Therefore, this update process is repeated several times for all sample points. By this repetition, the tentative pixel value gradually approaches an accurate value, so that an image obtained by repeating the set number of times is output as a high resolution image.

  FIG. 3 is a flowchart showing a flow of the basic principle of this high resolution. First, edge detection is performed (step A1), and then a predetermined number of corresponding points similar to the detected luminance pattern of the edge are detected (step A2 to step A6). Then, the pixel value of the high resolution image is calculated by using the pixel value of the low resolution image and the information of the detected corresponding point (step A7).

  With the above procedure, the high resolution application 200 limits the search range to a narrow range such as within its own frame and obtains a sharp high resolution image while keeping the amount of memory and computing resources to be used small. High resolution is achieved with few frame references.

  Next, a mechanism provided for the resolution enhancement application 200 to dynamically adjust the processing load for resolution enhancement will be described.

  As described above, since the high resolution application 200 is one program that operates on the computer, the calculation resources that can be used are usually limited. The estimated value Lall of processing load other than edge detection per frame can be expressed by the following equation (1).

Lall = Nedge * Lsearch + Nedge * Nsearch * Lsharpen (1) Equation where
Nedge: Number of edges (depends on image content)
Lsearch: Search load
Lsharpen: High resolution image calculation load
Nsearch: search range of corresponding points at the time of search Since the number of edges varies depending on the content of the image, it can be seen that the amount of calculation required varies depending on the content of the image. Hereinafter, parameters that can be set for the system, such as Nsearch, will be referred to as setting parameters.

  Lsearch and Lsharpen vary depending on the search algorithm and the high-resolution pixel calculation algorithm, respectively. A plurality of algorithms are prepared for the edge search, the search, and the calculation of the high-resolution pixel, and the load for the search and the calculation of the high-resolution pixel can be changed by replacing the algorithms. However, the image quality of the finally generated image may be affected by changing the algorithm for Nsearch, search, and high-resolution pixel calculation.

  Based on the above, reference is again made to FIG. When the edge detection unit 202 stores information about edges in the buffer unit 203, the edge counting unit 211 counts the total number of detected edges for each frame. Then, after the result of counting the edges in the frame is obtained, the parameter / processing algorithm setting unit 212 estimates the processing load and appropriately changes the parameters and the processing algorithm. FIG. 4 is a flowchart showing the flow of high resolution processing considering the dynamic adjustment of the processing load.

  For example, Nsearch estimated to be a load that can be processed with available computing resources can be set in the corresponding point search unit 204, and the number of corresponding points searched can be changed. The maximum Nsearch possible is determined by the measured Nedge. The load allowed as Lall is determined by the system state.

  Here, since Nedge changes depending on the state of the image (the number of edges), there are the following methods for changing the setting parameters. This corresponds to the specific processing contents in step B2 (“setting parameter change determination processing”) in the flowchart of FIG. 4 executed by the parameter / processing algorithm setting unit 212.

  (A) The edge is counted for each processing of one frame, and the setting parameter is determined.

  (B) Even when it is estimated that the number of edges of the current frame can be changed to a setting parameter with a higher load (that is, high image quality), the setting parameter is changed until the state continues for a certain frame period. Absent. However, the setting parameter is immediately changed when it is assumed that the load that is permitted for the high image quality processing is exceeded unless the setting parameter is made a lower load depending on the number of edges.

  According to (b), it is possible to suppress frequent changes in setting parameters and to stabilize the quality of moving images. The change of the setting parameter with the suppression of (b) will be described with reference to FIG.

  The upper limits for Nsearch = 2, 4, 6 are Nsearch [2], Nsearch [4], and Nsearch [6], respectively. At T1 and T2, the upper limit values Nsearch [6] and Nsearch [4] are exceeded, so Nsearch is changed to 4 and 2. In T3, it was lower than Nsearch [4], but Nsearch is not changed to 4 immediately. When a certain amount of time has passed in T4, Nsearch is changed to 4. Even if it temporarily falls below Nsearch [6] at T5, Nsearch [6] is again exceeded at T6, and Nsearch is not changed because a certain time has not passed. The method for determining Nsearch changes can be as follows, for example.

  Nsearch selects as an integer value. Therefore, for some Nsearch values, the upper limit of Nedge is obtained from the processing load Lok allowed by Lall (determined by the system status) and the single processing load of the related processing unit, and observed. Nsearch can be determined by determining what value Nedge is for those values.

  FIG. 6 shows an example of a table for obtaining the upper limit value of Nedge for each Nsearch. Here, Nsearch is an even number, but in the processing performed by the corresponding point search unit 204, a method of performing an axial search symmetrically can be considered. If you follow that approach, you should choose an even value. Using this table, Nsearch can be selected according to which range Nedge is within the upper limit of each Nedge.

  As described above, the resolution increasing application 200 counts the number of edges observed in the frame, and determines the search range of corresponding points so that the estimated processing load does not exceed the allowable processing load. Thus, for example, it is possible to perform high image quality processing while maintaining real-time properties. That is, it is possible to dynamically adjust the processing load for high resolution. In addition, it is possible to obtain a stable image quality of the moving image by making the parameter setting change for determining the search range less frequent. In particular, this can be done by setting the number of corresponding points to be searched (Nsearch).

  By the way, as described above, when the POCS method is used to calculate a high-resolution image, the error value can be made smaller by repeating the error correction update process, and in general, a high-quality image can be obtained. However, the load increases as the number of iterations of the error correction update process increases. On the other hand, if the number of iterations is decreased, the image quality deteriorates. Therefore, by increasing the value for adding / subtracting the error value to one time for decreasing the number of iterations, the degradation in image quality can be suppressed to a smaller value.

  The estimated value Lall of the entire load can be expressed by the following equation (2).

Lall = Nedge * Lsearch + Nedge * Nsearch * Nsharpen * Lsharpen0
... (2) where:
Nedge: Number of edges (depends on image content)
Lsearch: Search load
Lsharpen0: Update processing load for error correction in POCS method
Nsearch: search range of corresponding points during search
Nsharpen: Number of error correction update processes in the POCS method
Nsearch and Nsharpen are one of the setting parameters. So measure Nedge and select the largest possible Nsharpen. FIG. 7 shows an example of a table for obtaining the upper limit value of Nedge for each Nsharpen.

  Regarding the timing for changing the setting parameters, the same policy as the above-mentioned Nsearch can be taken. Nsharpen selects as an integer value. Therefore, Nsharpen is determined by determining the upper limit of Nedge that can be processed for some Nsearch values in advance and determining what value the observed Nedge is for those values. Can do.

  As described above, the high resolution application 200 can adjust the processing load appropriately by changing the number of repetitions (Nsharpen) of the update process especially when the POCS method is used.

  Further, when both Nsearch and Nsharpen are to be adjusted by the parameter / processing algorithm setting unit 212, either of the Nsearch and Nsharpen parameters can be changed with priority. If image quality degradation is expected due to the change in the number of corresponding points due to the change in Nsearch, it may be possible to maintain Nsearch and change the value of Nsharpen. In this way, if both Nsearch and Nsharpen are targeted for adjustment by the parameter / processing algorithm setting unit 212, the high resolution application 200 can be used for a system in which the variation in allowable processing load is severe and the nature of the number of edges of an image. Even when the time fluctuates greatly over time, the processing load can be adjusted flexibly.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

The figure which shows the structural example of the information processing apparatus (personal computer) which concerns on one Embodiment of this invention 2 is a diagram showing a functional configuration of a resolution enhancement application that runs on the computer of the embodiment. A flowchart showing a flow of a basic principle of high resolution executed by a high resolution application running on the computer of the embodiment A flowchart showing a flow of high resolution processing in consideration of dynamic adjustment of processing load, which is executed by the high resolution application that runs on the computer of the embodiment The figure for demonstrating the change of the parameter which added the suppression which the high resolution application which runs on the computer of the embodiment performs The figure which shows the example of the table | surface which calculated | required the upper limit of Nedge for every Nsearch which the high resolution application which operate | moves on the computer of the embodiment sets The figure which shows the example of the table | surface which calculated | required the upper limit of Nedge for every Nsharpen which the high resolution application which operate | moves on the computer of the embodiment sets

Explanation of symbols

  11 ... CPU, 12 ... North Bridge, 13 ... Main Memory, 14 ... Graphics Controller, 14A ... VRAM, 15 ... LCD, 16 ... South Bridge, 17 ... BIOS-ROM, 18 ... HDD, 19 ... USB Controller, 20 ... Sound controller, 21 ... Speaker, 22 ... TV tuner, 23 ... Embedded controller / keyboard controller IC (EC / KBC), 24 ... Keyboard, 25 ... Touchpad, 26 ... Power supply circuit, 27 ... Battery, 28 ... Network controller, 100 DESCRIPTION OF SYMBOLS ... Operating system (OS), 200 ... High resolution application, 201 ... Buffer part, 202 ... Edge detection part, 203 ... Buffer part, 204 ... Corresponding point search part, 205 ... Buffer part, 206 ... High resolution pixel value calculation part 211 ... Tsu di counting unit, 212 ... Parameters / processing algorithm setting unit.

Claims (8)

Input means for inputting image data of a first resolution composed of one or more frames in which the luminance of each pixel is expressed as a pixel value;
Detecting means for detecting an image area of interest in which a difference in pixel value between adjacent pixels is equal to or greater than a threshold value in one frame selected as a reference frame from the image data or in two or more frames including the reference frame;
Search means for searching for a corresponding image area that approximates a change pattern of a pixel value between pixels in each target image area for each target image area detected by the detection unit;
Based on the pixel value of each pixel of the reference frame and information on the corresponding image area searched by the search means, image data having a second resolution higher than the first resolution corresponding to the reference frame is generated. High resolution means to
Setting means for setting the number of corresponding image areas to be searched by the search means for each of the target image areas according to the number of target image areas detected by the detection means;
An information processing apparatus comprising:   The number of corresponding image areas to be searched for by the search means on the condition that the state that can be changed in a direction to increase the number of corresponding image areas to be searched by the search means continues for a predetermined period or longer. The information processing apparatus according to claim 1, wherein the information processing apparatus is changed in a direction to increase the value. The high-resolution means increases the resolution of the first resolution image data to the second resolution image data using the POCS method,
2. The information processing apparatus according to claim 1, wherein the setting means includes means for setting the number of iterations of error correction update processing and an error addition / subtraction value in the POCS method executed by the resolution increasing means.   The information processing apparatus according to claim 1, wherein the setting unit includes a unit that sets the threshold used when the detection unit detects a target image area. A method for increasing the resolution of image data using an information processing device comprising a data input means, a data output means and a central processing unit,
The image data of the first resolution consisting of one or more frames in which the luminance of each pixel is expressed as a pixel value is input from the data input means,
An attention image area whose pixel value difference between adjacent pixels is equal to or greater than a threshold value in one frame selected as a reference frame from the image data or in two or more frames including the reference frame is determined by the central processing unit. Detect
In accordance with the number of detected target image areas, the central processing unit sets the number of search for corresponding image areas that approximate the pixel value change pattern between pixels in each target image area,
For each of the detected image areas of interest, the central processing unit searches for the corresponding image areas of the set number from the reference frame,
Based on the pixel value of each pixel of the reference frame and the information of the searched corresponding image area, image data of the second resolution higher than the first resolution corresponding to the reference frame is the central processing unit. And output from the data output means,
A method for increasing the resolution of image data.   Changing to a direction to increase the number of corresponding image areas to be searched on condition that a situation in which the number of corresponding image areas to be searched can be changed in a direction to increase is continued for a predetermined period or longer. The method for increasing the resolution of image data according to claim 5. The generation of the image data of the second resolution is performed using the POCS method,
6. The method for increasing the resolution of image data according to claim 5, further comprising setting the number of iterations of error correction update processing and an error addition / subtraction value in the POCS method.   The method for increasing the resolution of an image according to claim 5, further comprising setting the threshold value used when detecting the target image area.

Images