JP4163709B2 - Image correction apparatus, image correction method, and image correction program - Google Patents

Description

  The present invention relates to a technique for correcting an image.

  With the spread of the Internet, terrestrial digital broadcasting, wireless LAN, and the like, in recent years, many images (videos) have been transmitted via a network. In video communication using a wireless LAN, packet loss occurs at a higher rate compared to a wired network.

When an error such as packet loss occurs in video communication, for example, an image correction (restoration) is performed by applying an error correction technique (error concealment). The error correction technique is described in Non-Patent Document 1 and Non-Patent Document 2.
Y. Wang and QF. Zhu, “Error control and concealment for video communication: a review”, Proc. The IEEE, vol.86, no.5, 1998.5. “Information technology − coding of audio-visual objects-part 2: visual”, ISO / IEC 14496-2, 2004.

  In error correction technology, for example, an image frame in which a packet loss has occurred is replaced with a previously received image frame in units of frames, or a pixel that has been lost due to the occurrence of a packet loss is replaced with a pixel at the same location in the past image frame. Modify the image by replacing it in units.

  However, when the movement between image frames transmitted in time series is large, or when the pattern of the image is complicated, the corrected part is uncomfortable or inconsistent with other parts, and the image is difficult to see visually. There is a risk. That is, when missing pixels are corrected using a past image frame, the image quality may deteriorate.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to correct a missing pixel in an image without using a past image frame, and to avoid deterioration in image quality after correction. is there.

In order to solve the above-described problem, the present invention provides, for example, an image correction apparatus, a detection unit that detects error data from a plurality of image data received via a network, and the error in an image. A specifying means for specifying a missing area which is a display part of data, and a peripheral pixel displayed around the missing area are specified, and based on the peripheral pixel, the pixel of the missing area is arranged from the outside to the inside of the missing area. And a first correction unit that corrects the missing area in units of pixels, wherein the first correction unit applies a partial differential equation to the peripheral pixels, and the pixels in the missing area Is generated from the outside to the inside of the missing region to correct the missing region in units of pixels, and the partial differential equation is based on an isotropic diffusion equation for isotropic diffusion, and the first correction unit Said Identify the circle surrounding the落領region, calculates an average correction distance to the missing region periphery from the center of the circle, to determine the diffusion coefficient of the anisotropic diffusion equation is proportional to the average correction distance.

The present invention is also an image correction method performed by an information processing apparatus, wherein the information processing apparatus includes a detection step of detecting error data from a plurality of image data received via a network, The step of identifying a missing area which is a display part of the error data in the step of identifying a peripheral pixel displayed around the missing area and, based on the peripheral pixel, determining the pixel of the missing area of the missing area. generated from the outside to the inside, have rows and correcting step, the correcting the missing region in pixel units, wherein the correction step, the applied partial differential equations with respect to the peripheral pixels, the pixels of the missing region The missing area is generated from the outside to the inside to correct the missing area in units of pixels, and the partial differential equation is based on an isotropic diffusion equation that isotropically diffuses. Identifies a circle surrounding the missing area, it calculates the average correction distance to the missing region periphery from the center of the circle, to determine the diffusion coefficient of the anisotropic diffusion equation is proportional to the average correction distance.

The present invention is also an image correction program executed by the information processing apparatus, wherein the information processing apparatus includes a detection step of detecting error data from a plurality of image data received via a network; A step of identifying a missing area which is a display part of the error data, and identifying peripheral pixels displayed around the missing area, and determining the pixels of the missing area based on the neighboring pixels And generating a correction step of correcting the missing area in units of pixels, and the correction step applies a partial differential equation to the peripheral pixels, and sets the pixels in the missing area. The missing area is generated from the outside to the inside to correct the missing area in units of pixels, and the partial differential equation is based on an isotropic diffusion equation that diffuses isotropically, The correction step specifies a circle surrounding the missing area, calculates an average correction distance from the center of the circle to the outer periphery of the missing area, and determines a diffusion coefficient of the isotropic diffusion equation proportional to the average correction distance. To do.

  According to the present invention, it is possible to correct a missing pixel in an image without using a past image frame, and to avoid deterioration in image quality after correction.

  Embodiments of the present invention will be described below.

  FIG. 1 is a schematic configuration diagram of an image correction apparatus 1 to which an embodiment of the present invention is applied. An image correction apparatus 1 shown in the figure is an apparatus that receives image data (packets) from an image transmission apparatus (not shown) connected to the image correction apparatus 1 via a network and corrects the received image data if there is an error. is there. The image data includes moving image data and still image data. The image transmission apparatus encodes image data (encodes / compresses based on a certain rule), and transmits the encoded image data packet to the image correction apparatus 1.

  The image correction apparatus 1 includes an image input unit 11, an image storage unit 12, a decoding unit 13, a packet loss detection unit 14, a missing pixel specification unit 15, a buffering unit 16, and an image fineness detection unit 17. The image correction unit 18 and the display unit 19 are provided.

  The image input unit 11 receives (inputs) a packet of image data transmitted from the image transmission device, and stores the received packet in the image storage unit 12 in a one-dimensional state. It is assumed that the packet received by the image input unit 11 is encoded (compressed).

  The decoding unit 13 decodes (decodes) the data of each packet stored in the image storage unit 12 and extracts the original image data. The packet loss detection unit 14 detects a packet loss that has occurred on the network. The missing pixel specifying unit 15 specifies the display position (missing pixel region) of the pixel that is lost due to packet loss. The buffering unit 16 stores the decoded image data in a two-dimensional buffer.

  The image fineness detection unit 17 detects the fineness of the image in the vicinity of the missing pixel area specified by the missing pixel specifying unit 15 and determines whether the texture is a micro texture (fine pattern) or a macro texture (coarse pattern). . The image correction unit 18 corrects the missing pixel region by a micro-texture generation correction method or a macro-texture generation correction method described later. The display unit 19 displays a corrected image obtained by correcting the missing pixel area on the output device.

  The image correction apparatus 1 described above includes a CPU, a memory, an external storage device, an input device such as a keyboard and a mouse, an output device such as a display, and a communication control device for connecting to other devices. A general-purpose computer system including a bus connecting these devices can be used. In this computer system, each function of the image correction device is realized by the CPU executing a program for the image correction device 1 loaded on the memory. Note that a memory or an external storage device is used as a buffer in which the image storage unit 12 and the buffering unit 16 of the image correction apparatus 1 store image data.

  The program for the image correction apparatus 1 can be stored in a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, or an MO, or can be distributed via a network.

  Here, FIG. 2 shows an example in which the missing pixel region is corrected by a conventional error correction technique, and FIG. 3 shows an example in which the missing pixel region is corrected by the image correction apparatus 1 of the present embodiment.

  FIG. 2 shows an example in which pixels at the same location as where a packet loss has occurred in an image frame (video frame) received in the past are embedded. In FIG. 2, a missing pixel region 210 due to packet loss is displayed in black in an image 200 having a complicated pattern. The missing pixel area 210 is replaced (corrected) with an image 220 cut out from a past image frame. When the movement between image frames transmitted in time series is large, or when the pattern of the image is complicated, the missing pixel area 220 after replacement is uncomfortable with other parts other than the missing pixel area 220. And inconsistencies occur, making the image visually difficult to see.

  On the other hand, in the present embodiment shown in FIG. 3, even if the image 300 has a complicated pattern and the movement between image frames transmitted in time series is large, the missing pixel region 320 is reduced. High quality without any discomfort or inconsistency with other parts can be corrected to an image.

  Next, image correction processing using the image correction apparatus 1 of the present embodiment will be described.

  First, the image input unit 11 sequentially receives packets of image data transmitted by an image transmission device (not shown) via a network. Then, the image input unit 11 stores the received packets in the image storage unit 12 in a one-dimensional state (for example, in the order of packet transmission). In this embodiment, it is assumed that the image data of the packet received by the image input unit 11 is encoded. Then, the decoding unit 13 decodes the image data of each packet stored in the image storage unit 12.

  The packet loss detection unit 14 detects packet loss. Packet loss is the loss of a packet during packet transmission. As a cause of packet loss, for example, it may be due to burstiness or noise. Packet loss due to burstiness occurs, for example, due to a failure of a building or the like in a moving mobile terminal. In packet loss due to burstiness, once packet loss occurs, packet loss occurs continuously, so a relatively large number of packets are concentrated and lost. Packet loss due to noise is caused by communication noise, and a relatively small number of packets are lost.

  FIG. 4 is a diagram for explaining a packet loss detection method.

  FIG. 4A schematically shows a packet transmitted from the image transmission apparatus on the packet transmission side to the image correction apparatus 1. Each packet 800 to 803 includes time stamp information, at least one piece of pixel information, a redundancy code for error determination, and the like. As the time stamp information, it is conceivable to use a predetermined time at which packet continuity can be confirmed, or a packet serial number.

  FIG. 4B schematically shows a packet received by the image correction apparatus 1. In the illustrated example, a packet 802 with time stamp information “T = n + 2” is lost. The packet loss detection unit 14 refers to the time stamp information of each of the packets 810, 811 and 813 stored one-dimensionally in the packet transmission order in the image storage unit 12, and the packet loss of the packet 802 of “T = n + 2” is detected. Detect what happened.

  Then, when the packet loss detection unit 14 detects a packet loss, the missing pixel specification unit 15 specifies the position of the pixel included in the lost packet. Each packet includes at least one piece of pixel information. The pixel information includes, for example, an address (image number) indicating the position of the pixel in the image, an RGB attribute, an image gray value, and the like.

  The missing pixel specifying unit 15 specifies the packets before and after the occurrence of the packet loss (packets 811 and 813 in the case of FIG. 3B), and reads each pixel information of the specified packets before and after. Then, the missing pixel specifying unit 15 determines the position of the missing pixel in the difference from the pixel included in the packet 811 immediately before the occurrence of the packet loss to the pixel included in the packet 813 immediately after the occurrence of the packet loss of the preceding and subsequent packets 811 and 813. Specify based on pixel information (address, image number, etc.).

  Then, the buffering unit 16 buffers the one-dimensional array packet group at this time point with a two-dimensional screen image. That is, the buffering unit 16 stores a packet group (decoded packet group) transmitted in one dimension and accumulated in the image accumulation unit 12 in a two-dimensional array in a buffer such as a memory.

  FIG. 5 is a diagram schematically illustrating the buffering process. The buffering unit 16 buffers each packet 910 transmitted one-dimensionally at a predetermined position (area) in the buffer 920 based on pixel information (address, image number, etc.) of the packet 910. Here, if a packet loss has occurred, the buffering unit 16 cannot place the lost packet in the buffer 920. For this reason, the missing pixel region 930 due to packet loss is generally displayed in black.

  Next, the discrimination process of the image fineness detection unit 17 will be described.

  FIG. 6 is an example of an image stored in a buffer in a two-dimensional array. In the illustrated image 600, a plurality of missing pixels (missing pixel areas) 610 due to packet loss are displayed. The image fineness detection unit 17 extracts a region where no packet loss occurs in the vicinity of the missing pixel region 610 (in the illustrated example, a peripheral region 620 surrounding the missing pixel region 610). Then, the image fineness detection unit 17 detects the fineness of the image in which no packet loss has occurred in the extracted region, and the image fineness detection unit 17 determines whether the texture is a micro-texture (fine pattern / pattern) or a macro-texture (coarse pattern / pattern). It is determined whether it is a symbol).

  If it is determined that the texture is a macro texture, the image correction unit 18 sets an L-shaped operator 630 in the macro / texture generation interpolation method in the vicinity of the missing pixel region 610. The macro / texture generation interpolation method will be described later.

  FIG. 7 is a flowchart of the image fineness detection process.

  As described above, the image fineness detection unit 17 determines which of the micro-texture generation interpolation method and the macro-texture generation interpolation method is applied. Note that it is possible to repair and correct a missing pixel region due to packet loss using either the micro-texture generation interpolation method or the macro-texture generation interpolation method. However, an actual image is complex, and there is also a relative complexity between the complexity of the entire image and the position of the missing pixel region. Therefore, in the present embodiment, in order to restore the missing pixels more elaborately, the image fineness detection unit 17 detects the fineness of the image in the vicinity of the missing pixel region and applies the micro-texture generation interpolation method. It is determined whether to apply the macro / texture generation interpolation method.

  As shown in FIG. 7, the image fineness detection unit 17 inputs an image in the vicinity of the missing pixel region (S10). Then, the image fineness detection unit 17 calculates the image feature amount of the input image (S20). That is, the image fineness detection unit 17 determines the texture complexity per unit area (one pixel or a plurality of pixels) as the spatial primary gradient value (1 The second derivative value) and the second gradient value (second derivative value) are calculated.

  Then, the image fineness detection unit 17 compares the calculated image feature amount with a predetermined threshold value, and determines whether the image of the unit area is fine (smaller than the threshold value) or coarse (larger than the threshold value). It discriminate | determines (S30). When it is determined that the texture has a rough structure, the image fineness detection unit 17 selects a macro / texture generation interpolation method (S40). On the other hand, when it is determined that the texture has a fine structure, the image fineness detection unit 17 selects a micro-texture generation interpolation method.

  In the fineness determination in S30, for example, determination may be made using only one of the primary gradient value and the secondary gradient value. In other words, the image fineness detection unit 17 may determine that, in the primary gradient value or the secondary gradient value, the larger the value, the coarser the image, and the smaller, the finer the image.

  Moreover, in the fineness determination of S30, it is good also as determining using both a primary gradient value and a secondary gradient value. In this case, for example, the image fineness detection unit 17 calculates an average value of each of the primary gradient value and the secondary gradient value for each unit region (the value of each of the primary gradient value and the secondary gradient value in the unit region). Divide the total by the number of pixels in the unit area). The image fineness detection unit 17 determines that the average value of the primary gradient values is small and the average value of the primary gradient values is large when the average value of the primary gradient values is small and the average value of the secondary gradient values is large. When the average value of the secondary gradient value is small, it may be determined that the value is fine. It is assumed that the threshold value for determining the average value is stored in advance in a memory or an external storage device.

  Next, the correction process of the missing pixel area performed by the image correction unit 18 will be described.

  The image correction unit 18 uses the micro / texture generation / complementation method (first correction method) or the macro / texture generation / complementation method (second correction method) according to the determination result of the image fineness detection unit 17. To correct the missing pixel area.

  FIG. 8 is a diagram for explaining a micro-texture generation interpolation method. The image restoration technique is well known by manual work by experts in old paintings and is called inpainting. It is a technology that restores the original paintings and images as close as possible without any sense of incongruity by locally replacing the scratches in new paintings and photos with the presumed original paintings and images. This is an important method from the viewpoint. The replacement is performed using the nearby symbols around the area to be repaired as a clue. In recent years, in the field of computer graphics, automatic restoration by a computer has been attempted by a method using an anisotropic diffusion (Anti-Diffusion: AD) equation. The anisotropic diffusion equation is described in the following references.

M.Beralmio, G.Sapiro, C.Ballester, and V.Caselles, Computer Graphics, “Image inpainting”, Proceeding of SIGGRAPH 2000, pp.417-424, July 2000
FIG. 8 shows this principle and concept. Based on the original image 400, the region 410 where the inner image is missing is interpolated. When the missing area 410 is used as a mask image, the outside is included as a boundary condition. Then, along the normal vector of the mask image, the peripheral image 400 is set as a known (initial value) as a boundary condition. The variable of the AD equation is a pixel in the image area. By inputting the initial value into the AD equation and repeating the time integration, the boundary pattern propagates to the inner mask image along the normal vector. As a result, an area 420 corresponding to the inner mask image is interpolated (repaired). By repeating this, all the areas up to the center are restored with the peripheral image information. Of course, it is possible to apply from monochrome to color designs for the target designs.

  In the image shown in FIG. 6, the mask image 410 in the micro-texture generation interpolation method is a missing pixel region 610 that is automatically detected by the missing pixel specifying unit 15. 4 is a region 620 surrounding the missing pixel region 610 in FIG. Further, the restoration process (correction) is performed with the missing pixel region 610 as an initial mask image. However, when the micro-texture generation interpolation method is repeated, the mask image is gradually restored from the outside to the inside. The original image of the region and the vicinity of the outside of the region changes.

  As the name suggests, the micro-texture generation interpolation method described above is good at expressing fine textures, allows the restoration target area to have any shape, and can handle non-repetitive non-periodic pattern images. It has the feature of being.

  Next, a partial differential equation used in the micro-texture generation interpolation method will be described. The partial differential equations are described in, for example, Akira Shoten, “Nonlinear phenomena of fluids—Mathematical analysis and its application” by Shunsuke Ikeda.

In the partial differential equation of the reference document described above, the following equation 1 is used as a main equation, and an anisotropic diffusion equation, which will be described later, equation 8 (or equation 9) is used in combination for each predetermined number of time integrations. On the other hand, in the partial differential equation of the present embodiment, Equation 1 is used as a main equation, and an isotropic diffusion equation 10 (or Equation 11) described later is used in combination for each predetermined number of time integrations.

In Equation 1, an image gray value of a pixel constituting an image is I, a pixel is (i, j), and a discretized time (or time) is n. Thus, I ⁿ (i, j) is an image gray value at the n-th discretized time of the pixel (i, j). Note that i, j, and n are integers. Further, Δt is a time width, and Ω is an area where image correction is performed. The vertical and horizontal widths of the pixels are Δx and Δy, respectively. Further, in Expression 1, Expressions 2 to 7 shown below are used.

  “→” is a vector. L is information about the image, but as can be seen from Equation 3 or Equation 4, it is a secondary spatial differential value of the image gray value. Expression 5 is an expression representing a unit vector, and is a normal vector of contour lines (concentration lines) of image gradation values. That is, Expression 5 is an amount for determining which direction the calculation proceeds (correction is performed) from the known image 400 into the image 410 to be corrected.

  Expression 7 is an expression representing the edge size of the image gray value. For f and b, which are subscripts on the right side of Equation 7, f represents a forward difference in the difference method, and b represents a backward difference in the difference method. Min selects a smaller value comparing 0 and the differential value, and Max selects a larger value comparing 0 and the differential value. Note that the convergence determination condition so far has been that when the temporal change of the image gray value I in Equation 1 has almost disappeared, or when Equation 2 has become a minute value or less.

Equation 8 shown below is an anisotropic diffusion equation used in the above-mentioned reference.

Further, when the anisotropic diffusion equation of Expression 8 is expressed in a discrete manner, the following Expression 9 is obtained.

  Expression 9 is characterized in that it has two effects of diffusing while preserving the edge shape in the image. K is a diffusion coefficient, and λ is a diffusion coefficient.

  In the reference, Equation 1 and Equation 8 (or Equation 9) are used alternately. At that time, Expression 8 picks up the image gray value of the known image 400 as a boundary condition. After that, the image grayscale value picked up from the boundary condition is propagated and diffused in the correction area 410 by time integration only using Expression 8 (or Expression 9). However, since the calculation process for extracting edge information and the diffusion effect are weak, there is a problem that the time integration takes a long time of several tens of times or more in the case of the isotropic diffusion equation in order to propagate a predetermined distance. Also, the image quality has a problem that the discontinuity with the known image 400 is conspicuous because the edges remain unnatural. When the anisotropic diffusion equation is used, it is difficult to determine the number of time integrations from the convergence condition in the correction region 410.

Equation 10 shown below is an isotropic diffusion equation used in this embodiment.

Further, when the isotropic diffusion equation of Expression 10 is expressed in a discrete manner, the following Expression 11 is obtained.

  K is a diffusion coefficient, and λ is a diffusion coefficient. Further, the left side of Expression 10 is a time change term with respect to the image gray value I, and the right side is a spatial second derivative term. Isotropic diffusion has the effect of expanding the area while diminishing the size of the diffusion coefficient λ and the initial image gray level per unit time.

  Here, when applying the isotropic diffusion equation of Equation 11, it is necessary to determine (set) the diffusion coefficient λ and the number of time integrations. It is conceivable that the diffusion coefficient λ and the number of time integrations are fixedly set to predetermined values based on experience or the like, but there is a risk that non-uniformity is noticeable in the image quality of the corrected image. Therefore, in the present embodiment, the diffusion coefficient λ and the number of time integrations are determined based on the average correction distance of the correction area (missing pixel area) to be corrected. First, the image correction unit 18 calculates an average correction distance in order to determine a diffusion coefficient.

  FIG. 9 is a diagram for explaining a method of calculating the average correction distance. First, the image correction unit 18 specifies the maximum circle 92 that surrounds the correction area 91. Then, the image correction unit 18 samples a plurality of distances (correction distances) 94 from the center point 93 of the circle to the outer periphery of the correction area 91. Then, the image correction unit 18 calculates an average value of the plurality of sampled correction distances and sets it as the average correction distance.

  FIG. 10 is an example of a diffusion coefficient determination graph showing the relationship between the diffusion coefficient and the average correction distance. Since the diffusion equation is linear, the distance through which an image propagates (average correction distance) and the magnitude of the diffusion coefficient are also proportional. Therefore, by performing a preliminary experiment using an image in advance, a graph showing the relationship between the diffusion coefficient and the average correction distance as shown in FIG. 10 can be easily created. It is assumed that a diffusion coefficient determination graph (or a correspondence table of diffusion coefficients and average image distance created from the graph) is stored in the memory or the external storage device of the image correction apparatus 1. The image correction unit 18 refers to a diffusion coefficient determination graph stored in a memory or the like, and determines a diffusion coefficient corresponding to the average correction distance.

  FIG. 11 is an example of a time integration count determination graph showing the relationship between the diffusion coefficient and the time integration count. As the average correction distance increases, the number of time integrations needs to be increased in proportion thereto. In other words, the diffusion coefficient and the number of time integrations are inversely proportional, and when the diffusion coefficient is large, the number of time integrations is small. Therefore, by performing a preliminary experiment using an image in advance, a graph showing the relationship between the diffusion coefficient and the number of time integrations as shown in FIG. 11 can be easily created. It is assumed that a time integration number determination graph (or a correspondence table of the diffusion coefficient and the time integration number created from the graph) is stored in the memory or the like of the image correction apparatus 1. The image correction unit 18 refers to a time integration number determination graph stored in a memory or the like, and determines a time integration number corresponding to the diffusion coefficient determined in FIG.

  In the present embodiment, by using the isotropic diffusion equation, the calculation speed can be improved, the correction can be made uniform, and the image quality can be improved. Further, by determining the diffusion coefficient and the number of time integrations according to the average correction distance, it is possible to ensure high quality image quality and to give a convergence condition (time integration number).

  Next, a macro / texture generation interpolation method will be described.

  FIG. 12 is a diagram for explaining a macro / texture generation interpolation method. According to the macro-texture generation interpolation method of this embodiment (see Li-Yi Wei and M. Levoy, “Texture systhesis over arbitrary manifold surfaces”, Proceeding of SIGGRAPH 2001, pp.355-360, Aug. 2001) Assuming that there is an image 500 rich in the texture structure of a small area as a base, an image 510 of a large area can be generated and interpolated from the image 500 of a small area while maintaining the natural continuity of the texture pattern. In the generation method, an L-shaped operator 520 composed of a plurality of pixels is acquired from the image 500, and the L-shaped operator 520 is copied to a predetermined location in the image 510. By repeatedly using the original small image 500 (the acquired L-shaped operator 520 is composed of a plurality of different pixels each time), an image 510 is generated.

  As its name suggests, the macro-texture generation interpolation method is good at expressing rough textures, is effective for organic repetitive patterns with fluctuations, and has a feature of fast processing time.

  When the image fineness detection unit 17 determines that the image is a coarse image in S30 (image fineness determination) in FIG. 7, the image correction unit 18 corrects the missing pixel region using the macro / texture generation interpolation method described above. Do. In the image shown in FIG. 6, the image correction unit 18 sets an L-shaped operator 630 near the missing pixel region 610.

  After the image correction process described above, the display unit 19 reads out an image in which the missing pixel region is corrected in units of pixels from the buffer and displays the image on the output device. That is, as shown in FIG. 3, the display unit 19 displays the image 320 after correcting the unnatural image quality missing pixel region 310 such that the image has a hole due to the occurrence of packet loss. Thereby, a high quality image without a sense of incongruity can be displayed.

  13 and 14 are graphs showing the results of image quality evaluation experiments.

  FIG. 13 is a graph obtained by experimenting by changing the moving speed of an image from a slow one to a fast one. The vertical axis represents MOS (Mean Opinion Score), and the horizontal axis represents the motion speed of the image. The MOS is a subjective evaluation, and in the illustrated example, the evaluation is made in five stages from “5 (no problem in image quality)” to “1 (very worrisome)”. In the experiment, each of five test subjects evaluated each moving image with different motion speeds by three times.

  In FIG. 13, when the missing pixel region is corrected 131 using the image correction apparatus 1 of the present embodiment 131, a high MOS can be ensured even if the image has a fast motion speed without a sudden decrease in MOS (that is, Image quality deterioration can be suppressed as much as possible). On the other hand, when the missing pixel region is corrected 132 using the error correction technique, it is indicated that the MOS is rapidly decreased in the image having a high motion speed. In the error correction technique, the missing pixel is corrected by simply replacing it with the corresponding pixel in the past image frame.

  FIG. 14 is a graph obtained by experimenting by changing the fineness of the image pattern (texture) from coarse to fine. The vertical axis is the same MOS as in FIG. 13, and the horizontal axis is the fineness of the image pattern. In FIG. 14, when the missing pixel region is corrected using the image correction apparatus 1 of the present embodiment 141, even if the image pattern is a fine image, a high MOS can be ensured without a sudden decrease in MOS (that is, Image quality deterioration can be suppressed as much as possible). On the other hand, when the missing pixel region is corrected 142 using the error correction technique 142, it is indicated that the MOS sharply decreases in an image with a fine image pattern.

  In the present embodiment described above, a missing pixel region due to packet loss is corrected on a pixel basis based on an image in the vicinity of the region without using a past image frame. Thereby, even if it is a moving image with a quick motion, or an image with a fine image pattern, degradation of an image can be suppressed as much as possible, and a high quality image without a visually uncomfortable feeling can be displayed.

  In this embodiment, after detecting a packet loss and specifying a missing pixel position, the received packet is buffered in a two-dimensional array. Thereby, an image in the vicinity of the missing pixel region can be specified, and the known image 400 and the corrected image 410 of the partial differential equation can be acquired.

  In the partial differential equation of this embodiment, an isotropic diffusion equation is used. As a result, the calculation speed can be improved, the correction can be made uniform, and the image quality can be improved. Further, by determining the diffusion coefficient and the number of time integrations according to the average correction distance, it is possible to ensure high quality image quality and to give a convergence condition (time integration number).

  As a result, the present invention is the core of new video communication technology and image conversion technology in the broadband age, and is widely used from home to professional specifications in digital image equipment such as digital cameras, digital video, and digital television. It is thought that it is done.

  In addition, this invention is not limited to said embodiment, Many deformation | transformation are possible within the range of the summary. For example, in the above-described embodiment, a case where one missing pixel region exists in one image has been described as an example (see FIG. 3). However, the present invention is not limited to this, and may be a case where a plurality of missing pixel regions are scattered in one image. In this case, it is conceivable to correct the image by mixing the micro-texture generation interpolation method and the macro-texture generation interpolation method in accordance with the fineness of the image in the vicinity of each missing pixel region.

1 is a diagram illustrating an overall configuration of an image correction apparatus to which an embodiment of the present invention is applied. This is an example in which a missing pixel region is corrected by an error correction technique. This is an example in which a missing pixel region is corrected by an image correction device. It is a figure for demonstrating the detection method of a packet loss. It is a figure for demonstrating a buffering process. It is an example of the image memorize | stored in the buffer by the two-dimensional arrangement | sequence. It is a flowchart of an image fineness detection part. It is a figure explaining the micro texture generation interpolation method. It is a figure for demonstrating the average correction distance of an isotropic diffusion equation. It is a graph which determines the diffusion coefficient of an isotropic diffusion equation. It is a graph which determines the number of time integrations of an isotropic diffusion equation. It is a figure explaining a macrotexture production | generation interpolation method. It is the figure which graphed the image quality evaluation experiment result which changed the motion speed of the image. It is the figure which graphed the image quality evaluation experiment result which changed the fineness of the image.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1: Image correction apparatus, 11: Image input part, 12: Image storage part, 13: Decoding part, 14: Packet loss detection part, 15: Missing pixel specific area | region, 16: Buffering part, 17: Image fineness detection part , 18: Image correction unit, 19: Display unit

Claims (8)

An image correction device,
Detecting means for detecting error data from a plurality of image data received via a network;
A specifying means for specifying a missing area which is a display part of the error data in the image;
A peripheral pixel to be displayed around the missing region is identified, and based on the neighboring pixel, a pixel in the missing region is generated from the outside to the inside of the missing region, and the missing region is corrected in pixel units. 1 correction means,
The first correction unit applies a partial differential equation to the surrounding pixels, generates pixels in the missing region from the outside to the inside of the missing region, corrects the missing region in units of pixels, and The partial differential equation is based on the isotropic diffusion equation that isotropically diffuses,
The first correction means identifies a circle surrounding the missing region, calculates an average correction distance from the center of the circle to the outer periphery of the missing region, and diffuses the isotropic diffusion equation proportional to the average correction distance. Determining the coefficient,
An image correction apparatus characterized by the above. The image correction apparatus according to claim 1,
Buffer storage means for storing the received plurality of image data in two dimensions;
Buffering means for storing each of the image data in a predetermined position of the buffer storage means based on address information of the image data;
The image correction apparatus according to claim 1, wherein the first correction unit reads a buffer storage unit in which the image data is two-dimensionally stored, and specifies peripheral pixels displayed around the missing region. The image correction apparatus according to claim 1,
The partial correction equation is based on the isotropic diffusion equation, and is applied by being discretized on a calculation lattice plane. The image correction apparatus according to claim 1,
Detecting means for detecting the fineness of the image of the peripheral area displayed around the missing area, and determining whether the peripheral area is a micro texture or a macro texture;
A second correction that corrects the missing region using a macro texture interpolation method that generates and interpolates each pattern composed of a plurality of pixels in the vicinity of the missing region when it is determined as a macro texture. And an image correction apparatus. The image correction apparatus according to claim 4 ,
Each pattern used repeatedly in the macro texture interpolation method has an L shape. The image correction apparatus according to claim 4 or 5 , wherein
The discriminating unit discriminates the fineness of the image of the peripheral region based on at least one of the spatial first derivative value and the spatial second derivative value of the image gray value of the peripheral region. Image correction device. An image correction method performed by an information processing apparatus,
The information processing apparatus includes:
A detection step of detecting error data from a plurality of image data received via the network;
A specific step of identifying a missing area that is a display part of the error data in the image;
Correction that identifies peripheral pixels displayed around the missing area, generates pixels of the missing area from the outside to the inside of the missing area based on the surrounding pixels, and corrects the missing area in units of pixels and step, stomach line,
The correction step applies a partial differential equation to the surrounding pixels, generates pixels of the missing region from the outside to the inside of the missing region, corrects the missing region in units of pixels, and the partial differential equation Is based on the isotropic diffusion equation for isotropic diffusion,
The correction step specifies a circle surrounding the missing area, calculates an average correction distance from the center of the circle to the outer periphery of the missing area, and determines a diffusion coefficient of the isotropic diffusion equation proportional to the average correction distance. An image correction method characterized by: An image correction program executed by the information processing apparatus,
In the information processing apparatus,
A detection step of detecting error data from a plurality of image data received via the network;
A specific step of identifying a missing area that is a display part of the error data in the image;
Correction that identifies peripheral pixels displayed around the missing area, generates pixels of the missing area from the outside to the inside of the missing area based on the surrounding pixels, and corrects the missing area in units of pixels and step, to the execution,
The correction step applies a partial differential equation to the surrounding pixels, generates pixels of the missing region from the outside to the inside of the missing region, corrects the missing region in units of pixels, and the partial differential equation Is based on the isotropic diffusion equation for isotropic diffusion,
The correction step specifies a circle surrounding the missing area, calculates an average correction distance from the center of the circle to the outer periphery of the missing area, and determines a diffusion coefficient of the isotropic diffusion equation proportional to the average correction distance. An image correction program characterized by: