JP4241463B2 - Image processing device - Google Patents

Description

  The present invention relates to an apparatus for reproducing a video stream.

  Since Motion-JPEG2000 cannot use frame correlation as compared with moving picture coding specifications such as MPEG (Motion Picture Coding Experts Group), the compression rate is inferior, but it has an editing function and a lossless coding function in units of frames. It is excellent in that it has great potential.

As an example of a technique that utilizes the characteristics of JPEG2000, Patent Document 1 discloses a technique that compares elapsed time and time limit at each stage of image decoding and simplifies decoding processing as necessary. According to this, a relatively high-quality image can be decoded with a relatively small-scale configuration.
JP 2002-325257 A

  In the future, distribution and utilization of video content using JPEG2000 is expected to be actively performed. Of the video to be distributed, for example, there is a case where only a part of the video needs high image quality, such as a surveillance camera, but other areas do not require so much image quality. Further, when the image quality of the area designated by the user is increased, the increase in the processing amount may exceed the capability of the image processing apparatus, resulting in frame dropping or the like.

  The present invention has been made in view of these problems, and an object of the present invention is to provide a technique for adjusting image quality within a range not exceeding the capability of decoding processing.

  One embodiment of the present invention relates to an image processing apparatus. This apparatus decodes moving image data so that the image quality of a plurality of areas on the screen is displayed with a difference, and sets the image quality of each of the plurality of areas so that the processing amount of the entire decoding process does not exceed a predetermined value. adjust. According to this aspect, it is possible to suppress the processing amount when displaying a moving image by changing the image quality of a plurality of areas on the screen to a certain value or less.

  Another embodiment of the present invention also relates to an image processing apparatus. The apparatus includes: an area designating unit that designates an attention area on the screen; and a decoding unit that decodes an image with a difference in image quality between the attention area and the other normal area and outputs moving image data. Prepare. Here, the “attention area” refers to an area that the user wants to increase the resolution in the image and reproduces, and is designated using an input device such as a pointing device. According to this aspect, only the attention area designated by the user can be reproduced with a desired image quality.

  The decoding unit may adjust the image quality of the attention area and the normal area so that the processing amount of the entire decoding process does not exceed a predetermined threshold value. Alternatively, the decoding unit may improve the image quality of the attention area while reducing the image quality of the normal area. According to this, it is possible to reproduce the attention area of interest to the user with high image quality without increasing the processing amount of the image processing apparatus.

  A region specifying unit that determines whether the processing amount of the entire decoding process exceeds a predetermined threshold when the image quality of the region of interest is improved; And an image quality instruction unit that determines whether or not to improve image quality. According to this, when the amount of decoding processing is likely to increase and exceed the processing capability of the image processing device, improvement in the image quality of the attention area is not permitted, so that a frame generated when the processing amount exceeds the upper limit processing capability of the image processing device Occurrence of dropping can be avoided.

  Alternatively, the area designating unit determines whether or not the processing amount related to decoding exceeds a predetermined threshold by improving the image quality of the attention area and the image quality instruction section that improves the image quality of the attention area. And a determination unit. In this case, the image quality instruction unit may reduce the image quality of the normal area when it is determined by the determination unit that the processing amount exceeds a threshold value. According to this, the amount of decoding processing in the attention area is increased to improve the image quality, and the amount of decoding processing in the normal area is reduced accordingly, thereby reducing the processing amount of the entire image processing apparatus to the upper limit processing capacity or less. Can do.

  The image quality instruction unit may display a warning to the user that the image quality of the normal area is lowered when it is found that the processing amount exceeds the threshold value. In this way, the user can understand in advance that the normal area has low image quality.

  The image quality instruction unit may display a message asking the user whether or not to improve the image quality of the region of interest when the processing amount exceeds the threshold value. Accordingly, the user can determine whether or not to improve the image quality of the attention area according to the importance.

  The encoded image data may be multiplexed according to the resolution. In one example, the encoded image data is data conforming to Motion-JPEG2000 in which images compressed for each frame are continuously transmitted. According to this, since the encoded image data is multiplexed according to the resolution, it becomes easy to take out an intermediate image generated in the decoding process and use it as a low-resolution image.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a computer program, a recording medium, and the like are also effective as an aspect of the present invention.

  According to the present invention, it is possible to adjust the image quality within a range that does not exceed the capacity of the decoding process when displaying with a difference in the image quality of a plurality of areas on the screen.

  The present invention relates to a technique for generating moving images having different resolutions or image quality from an encoded image data stream. Hereinafter, preferred embodiments of the present invention will be described. In the embodiment, an image processing apparatus that decodes an encoded image data stream compliant with Motion-JPEG2000 as image processing will be considered as an example.

  First, with reference to FIG. 1, a method for encoding a moving image by the Motion-JPEG2000 method will be briefly described. An image encoding device (not shown) continuously encodes each frame of a moving image in units of frames to generate an encoded data stream of the moving image. At the start of the encoding process, an original image OI (Original Image) 102 corresponding to one frame of a moving image is read into the frame buffer. The original image OI read into the frame buffer is hierarchized by a wavelet transformer.

The wavelet transformer in JPEG2000 uses a Daubechies filter. This filter acts as a high-pass filter and a low-pass filter simultaneously in the x and y directions of the image, and divides one image into four frequency subbands. These subbands include an LL subband having a low frequency component in both the x and y directions, an HL subband having a low frequency component in one of the x and y directions and a high frequency component in the other direction, and An LH subband and an HH subband having a high frequency component in both the x and y directions. This filter also has an action of reducing the number of pixels by half in both the x and y directions. That is, the number of vertical and horizontal pixels of each subband is ½ of that of the image before processing, and a subband image with a resolution, that is, an image size of ¼ can be obtained by one filtering. In this specification, an image that has undergone wavelet transform once with respect to the original image OI is referred to as an image WI ₁ in the first layer, and hereinafter, an image WI _{n in} the nth layer according to the number of times of undergoing wavelet transform I will call it.

As schematically shown in FIG. 1, four subbands LL ₁ , HL ₁ , LH ₁ , and HH ₁ are generated in the image WI ₁ 104 of the first layer. The first layer image WI ₁ 104 is subjected to wavelet transform to generate a second layer image WI ₂ 106. Here, the second and subsequent wavelet transforms are performed only on the LL subband component in the immediately preceding layer image. Accordingly, in the second layer image WI ₂ 106, the LL ₁ subband of the first layer image WI ₁ is decomposed into four subbands LL ₂ , HL ₂ , LH ₂ , and HH ₂ . The wavelet transformer executes this filtering a predetermined number of times and outputs wavelet transform coefficients for each subband. Thereafter, the image encoding device performs quantization and other processing, and finally outputs encoded image data CI (Coded Image).

For simplicity of explanation, in this example, it is assumed that the image encoding device performs wavelet transformation three times on the original image OI. Therefore, for example, if the original image OI 102 is 1440 × 960 pixels, the size of the LL ₁ subband of the first layer image WI ₁ 104 is 720 × 480, and the LL ₂ sub of the second layer image WI ₂ 106. The band is 360 × 240, and the size of the LL ₃ subband of the third-level image WI ₃ 108 is 180 × 120.

It should be noted about the hierarchized image that the low frequency component in the original image OI appears in the upper left in FIG. In the case of FIG. 1, the LL ₃ subband in the upper left corner of the image WI ₃ in the third layer has the lowest frequency, and conversely, if this LL ₃ subband can be obtained, the most basic of the original image OI. Realistic properties can be reproduced. This knowledge is used in the following embodiments.

  In addition to Motion-JPEG2000, the encoded data stream may be, for example, an SVC (Scalable Video Codec) having a high-quality HD stream and a low-quality SD stream in one stream, or Motion-JPEG. . In JPEG, each frame is transmitted from the low-order term of the Fourier coefficient, so that the image quality can be selected depending on which order term is used for decoding.

First embodiment.
The first embodiment is an image processing apparatus that receives an encoded image data stream layered according to resolution and can provide moving images having different resolutions to a plurality of display devices.

  FIG. 2 shows a configuration of the image processing apparatus 100 according to the first embodiment. This configuration can be realized in terms of hardware by a CPU, memory, or other LSI of an arbitrary computer, and in terms of software, it is realized by a program having a decoding function loaded in the memory. The functional block realized by those cooperation is drawn. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

The decoding unit 150 of the image processing apparatus 100 receives a stream of encoded image data CI. The decoding unit 150 receives the encoded image data CI, analyzes the data stream, the stream analysis unit 10, the arithmetic decoding unit 12 that performs arithmetic decoding on the data string to be decoded that is found as a result of the analysis, and the result A bit plane decoding unit 14 that decodes the obtained data in the form of a bit plane for each color component, an inverse quantization unit 18 that inversely quantizes the decoded quantized data, and a first obtained as a result of the inverse quantization A wavelet inverse transform unit 20 that performs wavelet inverse transform on the n-layer image WI _n is included. Each time wavelet inverse transform is performed on the encoded image data CI by the wavelet inverse transform unit 20, a higher-level image is obtained, and finally, decoded image data DI (Decoded Image) can be obtained.

The present embodiment is characterized in that an n-th layer image generated during the wavelet inverse transformation for obtaining the decoded image DI in the wavelet inverse transformation unit 20 is output to the low resolution frame buffer 30. The image processing apparatus 100 provides image data corresponding to the respective resolutions to the low-resolution display device 36 that displays a low-resolution moving image and the high-resolution display device 46 that displays a high-resolution moving image. Operate. For this purpose, the memory control unit 22 acquires resolution information of moving images displayed on the low-resolution display device 36 and the high-resolution display device 46, and performs the wavelet inverse transformation on the encoded image CI several times. And the result is transmitted to the wavelet inverse transform unit 20. In accordance with this information, the wavelet inverse transform unit 20 converts the LL subband of the n-th layer image WI _n in the middle of the wavelet inverse transform process, or the decoded image data DI that is a complete decoding result, to the low resolution frame buffer 30. Alternatively, the high resolution frame buffer 40 is written. This operation will be described later with reference to FIG. Note that the low resolution frame buffer 30 and the high resolution frame buffer 40 are distinguished for convenience and do not necessarily have different buffer sizes.

  The image data written in the low resolution frame buffer 30 is generated as a display signal by the low resolution display circuit 32 and displayed on the low resolution display device 36. Similarly, the image data written in the high resolution frame buffer 40 is generated as a display signal by the high resolution display circuit 42 and displayed on the high resolution display device 46. As described above, according to the image processing apparatus 100, it is possible to simultaneously display moving images having different resolutions on a plurality of display devices using the encoded image data stream.

  Either or both of the low resolution display circuit 32 and the high resolution display circuit 42 may include resolution conversion units 34 and 44. Thus, when the resolution obtained by the wavelet inverse transform processing in the decoding unit 150 is different from the resolution of the moving image displayed on each display device 36, 46, the image of the hierarchy having the closest resolution is decoded. Thereafter, the resolution can be converted to a desired resolution by the resolution converters 34 and 44. These resolution conversion units 34 and 44 are optional, and if it is not desired to display a moving image having a resolution other than the resolution obtained by the wavelet inverse transformation process, the low resolution display circuit 32 or the high resolution display circuit. 42 may not include the resolution converters 34 and 44, respectively.

  FIG. 3 shows a processing procedure in the decoding unit 150. Here, as described above, it is assumed that a stream of encoded image data obtained by performing wavelet transform three times on the original image OI is given to the image processing apparatus 100.

First, the encoded image data CI input to the image processing apparatus 100 undergoes processing of the stream analysis unit 10, the arithmetic decoding unit 12, the bit plane decoding unit 14, and the inverse quantization unit 18, and then the third layer image WI ₃ Return to the state of 122. Subsequently, the first wavelet inverse transformation by the wavelet inverse transformation unit 20 obtains the second layer image WI ₂ 124, and the second wavelet inverse transformation obtains the image WI ₁ 126 corresponding to the first layer. Finally, a decoded image DI 128 is obtained by the third wavelet inverse transform.

By the way, as described above, the LL subband of each layer is obtained by extracting a low frequency component from the image of the layer, and the size is 1/4 of the image of the immediately preceding layer. Therefore, it can be said that these are low-resolution images compared to the original image OI. Therefore, as an example, the LL ₁ subband (720 × 480) of the first layer image WI ₁ 126 obtained by the second wavelet inverse transform is output to the low resolution frame buffer 30 as low resolution image data. The decoded image DI (1440 × 960) obtained by the inverse wavelet transform can be output to the high resolution frame buffer 40 as high resolution image data. Since the x and y directions of the image are each halved by performing the wavelet transform, the larger the number of wavelet transforms performed by the wavelet transformer of the image encoding device, the more types of resolutions. Can be displayed.

  FIG. 4 is a conceptual diagram illustrating generation of moving images having different resolutions from each frame. Based on the command from the memory control unit 22, the wavelet inverse transform unit 20 performs necessary decoding processing on each encoded image frame, and outputs a low resolution image to the low resolution frame buffer 30. Are output to the high-resolution frame buffer 40. Then, a low-resolution or high-resolution moving image is formed from the same encoded image data stream by continuously outputting low-resolution or high-resolution images at an intended frame rate.

  FIG. 5 is a flowchart for explaining the operation of the memory control unit 22. First, the memory control unit 22 acquires resolution information of a moving image displayed on the low resolution display device 36 and the high resolution display device 46 (S10). Alternatively, the user may be allowed to input information on the resolution of a moving image displayed on each display device. Next, the memory control unit 22 determines how many LL subbands the low resolution image displayed on the low resolution display device 36 corresponds to the encoded image CI (S12). Subsequently, the memory control unit 22 determines how many layers of the high-resolution image displayed on the high-resolution display device 46 corresponds to the LL subband of the image in the hierarchy or the decoded image DI that has been completely decoded. Determine (S14). Then, when the wavelet inverse transform processing for obtaining the images of the layers determined in S12 and S14 is completed in the wavelet inverse transform unit 20, the memory control unit 22 converts the subband image LL or the decoded image DI into the low resolution. A command is given to write to the frame buffer 30 or the high-resolution frame buffer 40 (S16). Of course, when there is only one display device that should output image data from the image processing device, either the low resolution frame buffer 30 or the high resolution frame buffer 40 may be used.

  As described above, in JPEG2000, the resolution of the LL subband is decreased by 1/2 and 1/2 by the original image, so that an appropriate resolution that matches the resolution of the desired display device is obtained. In some cases, it may not be possible to obtain the image. Therefore, if the memory control unit 22 determines in S12 or S14 that an appropriate resolution cannot be obtained, the memory control unit 22 applies the resolution conversion units 34 and 44 provided in the low resolution display circuit 32 or the high resolution display circuit 42, respectively. It is also possible to execute the interpolation processing of the resolution as appropriate.

The image processing apparatus 100 can also display moving images having different resolutions on three or more display devices by increasing the number of frame buffers. For example, if the above example is used, the LL2 subband (360 × 240) of the image WI ₂ 124 of the second layer obtained by the first wavelet inverse transform is used as the low resolution frame buffer, and the second wavelet inverse transform is performed. The LI ₁ subband (720 × 480) of the first-level image WI ₁ 126 obtained by the above is used as the medium resolution frame buffer, and the decoded image DI 128 (1440 × 960) obtained by the third wavelet inverse transformation is used as the high resolution. Each of the resolution frame buffers can be output and a corresponding display circuit can display low resolution, medium resolution, and high resolution moving images on the display device.

  As described above, according to the first embodiment, it is possible to simultaneously display moving images having different resolutions on two or more display devices using a single encoded image data stream. Conventionally, each time decoding is performed according to the target resolution, but by outputting an image in the middle of the decoding process to the frame buffer, image data of a plurality of resolutions can be obtained by a single decoding unit. Can be produced and is efficient.

Second embodiment.
FIG. 6 shows a configuration of an image display apparatus 200 according to the second embodiment. The image display device 200 includes two display devices, such as a first display device 222 that displays a high-resolution moving image and a second display device 224 that displays a low-resolution moving image, such as a display and a projector.

  The image decoder 212 of the processing block 210 cooperates with the CPU 214 and the memory 216 to continuously decode the input encoded image data stream. The image decoder 212 has the configuration of the image processing apparatus 100 of the first embodiment. Then, high-resolution image data is output to the first display device 222 via the display circuit 218, and low-resolution image data is output to the second display device 224 via the display circuit 220. On each screen, the image data decoded by the image decoder 212 is continuously displayed at a given frame rate, and a moving image is reproduced. The processing block 210 may acquire the encoded image data stream via a communication interface of a wireless or wired network, or may acquire the encoded image data stream via a reception block that receives broadcast radio waves. Good.

With this image display device 200, for example, the following operations can be realized.
1. Airplane in-flight screening If you have a large screen in front of the cabin and a small personal LCD on the back of each seat, you need only prepare a single encoded image data stream. The moving image can be reproduced on both the screen and the liquid crystal display.
2. Presentation If a single encoded image data stream is prepared at the time of presentation, a moving image can be reproduced on both a large screen projected by a projector and a PC screen.
3. Two-screen mobile phone By incorporating the above screen display device into a mobile phone having a main display and a sub-display, a single encoded image data stream is received and video content is played on both the main display and the sub-display. Is possible.

  Needless to say, the image display device 200 may include three or more display devices that display moving images having different resolutions according to the purpose.

Third embodiment.
In the third embodiment, when a user is instructed to improve the image quality of a partial area of an image in an image processing apparatus that decodes an image stream layered according to resolution or image quality and displays the decoded image stream on a display device In addition, the image processing apparatus controls the processing amount so as not to exceed the maximum capacity of the image processing apparatus.

  FIG. 7 is a diagram illustrating a configuration of an image processing apparatus 300 according to the third embodiment. The image processing apparatus 300 includes a decoding unit 310 that receives a stream of encoded image data CI and decodes the image, and an area designation unit 320 that executes a process related to an attention area designated in the image by the user. The stream analysis unit 10, the arithmetic decoding unit 12, the bit plane decoding unit 14, the inverse quantization unit 18, and the wavelet inverse transformation unit 20 included in the decoding unit 310 are the same as those described in the first embodiment.

  The image data decoded by the decoding unit 310 is displayed on the display device 62 by the display circuit 60. The user uses an input device such as a pointing device (not shown) to designate a region (hereinafter referred to as “region of interest (ROI)”) that is desired to be reproduced with improved image quality. Then, the position information creation unit 50 in the area designation unit 320 creates ROI position information indicating the position of the attention area ROI. This ROI position information is given by the coordinate value of the pixel at the upper left corner of the rectangular area and the number of vertical and horizontal pixels of the rectangular area when the attention area ROI is designated by a rectangle. When the user designates the attention area ROI with a circle or the like, the circumscribed rectangle may be set as the attention area. The attention area may be always set to a predetermined area such as the center area of the original image.

  Based on the created ROI position information, the determination unit 52 calculates the amount of increase in data processing necessary to improve the image quality of the attention area, and the total amount of decoding processing added to the current processing amount is Then, it is determined whether or not the image processing apparatus 300 falls within the maximum processing capacity. Based on the determination result, the image quality instruction unit 54 determines whether or not to allow the image quality of the attention area to be high, or to reduce the image quality of the area other than the attention area (hereinafter referred to as “normal area”). Is output to the ROI mask generator 56. Details of this processing will be described later with reference to FIG. 11 or FIG.

  The ROI mask generation unit 56 generates an ROI mask for specifying a portion corresponding to the region of interest among the wavelet transform coefficients based on the ROI position information from the position information generation unit 50. The generated ROI mask is used by the low-order bit zero replacement unit 58 to adjust the number of low-order bits to be replaced with zero values in the bit string of the wavelet transform coefficients. By performing inverse wavelet transform on this, it is possible to obtain an image in which the region of interest has high image quality. This will be described later.

  Here, a method for generating an ROI mask based on the ROI position information by the ROI mask generation unit 56 will be described with reference to FIGS. As illustrated in FIG. 8A, it is assumed that the attention area 90 is designated by the user on the image 80 decoded and displayed by the image processing apparatus 300. The ROI mask generation unit 56 specifies a wavelet transform coefficient necessary for restoring the attention area 90 selected on the image 80 in each subband.

FIG. 8B shows a converted image 82 of the first layer obtained by performing wavelet transform on the image 80 only once. The converted image 82 of the first layer is composed of four subbands LL ₁ , HL ₁ , LH ₁ , and HH _{1 of} the first level. The ROI mask generation unit 56 outputs wavelet transform coefficients (hereinafter referred to as “ROI transform coefficients”) 91 to 94 on the first-level transformed image 82 necessary for restoring the attention area 90 of the image 80 to the first. It is specified in each sub-band LL ₁ , HL ₁ , LH ₁ , HH ₁ of one level.

FIG. 8C shows a second layer converted image 84 obtained by further wavelet transforming the subband LL ₁ of the converted image 82 of FIG. 8B. As shown in the figure, the second-level converted image 84 includes the first level three subbands HL ₁ , LH ₁ , and HH ₁ , and the second level four subbands LL ₂ , HL ₂ , and LH _2. , HH ₂ . The ROI mask generation unit 56 is a wavelet transform coefficient on the second layer transform image 84 necessary for restoring the ROI transform coefficient 91 in the subband LL ₁ of the first layer transform image 82, that is, the ROI transform coefficients 95 to 95. 98 each subband _LL 2 of the second _level, identified in HL _2, LH 2, HH _2.

  Similarly, the ROI transform coefficient corresponding to the attention area 90 is recursively specified in each layer by the number of wavelet transforms, so that the ROI necessary for restoring the attention area 90 in the converted image of the final hierarchy is obtained. All conversion coefficients can be specified. The ROI mask generation unit 56 generates an ROI mask for specifying the position of the finally specified ROI conversion coefficient on the converted image of the final hierarchy. For example, when the wavelet transform is performed only twice, ROI masks that can specify the positions of the seven ROI transform coefficients 92 to 98 indicated by diagonal lines in FIG. 8C are generated.

  Next, with reference to FIGS. 9 and 10, a method for improving the image quality of the attention area will be described. Here, as shown in FIG. 9A, it is assumed that the encoded image data CI is composed of 5-bit planes from MSB (Most Significant Bit) to LSB (Least Significant Bit).

  The image processing apparatus 300 performs simple reproduction in which the lower bit plane of the wavelet transform coefficient is appropriately discarded and reproduced in order to reduce the processing load during normal times when the attention area is not designated by the user. The image quality at this time is referred to as “medium image quality”. In this case, as shown in FIG. 9B, the lower bit zero replacement unit 58 replaces, for example, the lower 2 bits of the bit plane decoded by the bit plane decoding unit 14 with zero, and only the 3 bit plane. To be decrypted. In order to achieve high image quality only in the attention area from this state, it is only necessary to decode a larger number of bit planes only in the attention area.

  FIGS. 10A to 10C show an example of processing for improving the image quality of a region of interest. As shown in FIG. 10A, at the time of simple reproduction, the lower 2 bits are zero-substituted from the LSB side by the lower bit zero replacing unit 58. When an attention area is designated by the user, the ROI mask generation unit 56 generates an ROI mask corresponding to the attention area. This is indicated by the oblique lines in FIG. Then, as shown in FIG. 10C, the low-order bit zero replacement unit 58 generates a wavelet transform coefficient by referring to the ROI mask and replacing only the low-order 2 bits of the non-ROI portion that is not ROI masked with zero. .

  The inverse quantization unit 18 inversely quantizes the generated wavelet transform coefficient, and the wavelet inverse transform unit 20 inversely transforms the inversely quantized wavelet transform coefficient. Thereby, image data in which only the attention area ROI is improved in image quality is obtained.

  Next, processing in the determination unit 52 will be described with reference to the flowchart in FIG. As a premise, it is assumed that the moving image is displayed with the above-described medium image quality at the normal time when the attention area is not designated by the user.

  First, the determination unit 52 receives ROI position information of a region of interest from the position information creation unit 50 (S30). Next, the area (or the number of pixels) of the attention area is calculated from the ROI position information, and the entire decoding processing amount P of the image processing apparatus 300 is calculated (S32).

Here, the decoding processing amount P can be expressed as a sum of (processing amount of each image quality) × (area of each image quality). When the processing amount per unit area at the time of low image quality is denoted as l _L , the processing amount per unit area at the time of medium image quality is denoted as l _M , the processing amount per unit area at the time of high image quality is denoted as l _H, and the area of the entire image is denoted as S The normal decryption processing amount is
P = 1 _M · S (1)
It becomes.

When the area of the attention area designated by the user is s _H and the attention area is improved in image quality, the decoding processing amount P is as follows:
P = l _H · s _H + l _M (S−s _H ) (2)
Can be calculated with

The determination unit 52 determines whether or not the decoding processing amount P calculated by Expression (2) exceeds the upper limit processing capability P _max that the image processing apparatus 300 can process in one frame period (S34). If the decoding processing amount P is less than or equal to the upper limit processing capacity P _max (NO in S34), the image quality instruction unit 54 permits the image quality of the attention area to be improved (S36). If the decoding processing amount P exceeds the upper limit processing capability P _max (YES in S34), the processing capability sufficient for high-quality decoding does not remain in the image processing apparatus 300. Is not permitted to improve the image quality (S38).

FIGS. 12A and 12B schematically show the state of the screen when it is determined in S34 of the flowchart of FIG. 11 that the decoding processing amount P is equal to or less than the upper limit processing capability _Pmax . In the figure, “L” indicates a low image quality region, “M” indicates a medium image quality region, and “H” indicates a high image quality region. As shown in FIG. 12A, when the user designates a region of interest on the screen while decoding the entire image with medium quality, only the region of interest has high image quality as shown in FIG. (H), and other normal areas remain medium image quality (M).

  As described above, according to the image processing apparatus of the present embodiment, when the user designates a region of interest that the user wants to reproduce with high image quality in the decoded and displayed image, there is a margin in the decoding processing capability of the image processing apparatus. Sometimes, the attention area is improved in image quality, and when the decoding processing capacity is not sufficient, the attention area is not improved in image quality.

  As described above, when the attention area is designated, only the attention area can be reproduced with higher quality while the normal area has the same quality as the simple reproduction. This is particularly useful when a high quality is not required during normal times and a point of interest is to be reproduced with high quality only when an abnormality occurs, as in a surveillance video.

  Next, another example of processing when the processing capability of the image processing apparatus 300 is insufficient when the attention area is improved in image quality will be described with reference to FIGS.

  As shown in FIG. 13A, at the time of simple reproduction, it is assumed that the lower 2 bits are zero-substituted from the LSB side by the lower bit zero replacing unit 58. When an attention area is designated by the user, the ROI mask generation unit 56 generates an ROI mask corresponding to the attention area. This is indicated by the oblique lines in FIG. In this case, the area of the attention area is larger than that in the case of FIG. In this case, as shown in FIG. 13C, the lower bit zero replacement unit 58 refers to the ROI mask and replaces the lower 3 bits instead of the lower 2 bits with zero for the non-ROI portion that is not ROI masked. Generate wavelet transform coefficients.

  Then, the inverse quantization unit 18 inversely quantizes the generated wavelet transform coefficient, and the wavelet inverse transform unit 20 inversely transforms the inversely quantized wavelet transform coefficient. As a result, image data in which the attention area ROI has a high image quality and the normal area has a low image quality can be obtained. As described above, when the ROI masked region of interest is improved in image quality, that is, when the number of bit planes is increased, if the processing capacity of the image processing apparatus is insufficient, the number of bit planes in the normal region that is not ROI masked is reduced. Thus, the processing amount of the entire image processing apparatus is balanced.

  Next, with reference to the flowchart of FIG. 14, processing in the determination unit 52 when the processing capability of the image processing apparatus 300 is insufficient when the attention area is improved in image quality will be described. As in the case of FIG. 11, it is assumed that the moving image is displayed with the above-described medium image quality in the normal time when the attention area is not designated by the user.

Receiving the attention area (S50) and calculating the decoding processing amount P of the entire image processing apparatus 300 (S52) are the same as S30 and S32 in FIG. The determination unit 52 determines whether or not the decoding processing amount P calculated in S52 exceeds the upper limit processing capacity _Pmax that the image processing apparatus 300 can process in one frame period (S54). If the decoding processing amount P is less than or equal to the upper limit processing capability P _max (NO in S54), the image quality instruction unit 54 permits the image quality of the attention area to be improved (S64).

When the decryption processing amount P exceeds the upper limit processing capacity P _max , the determination unit 52
P = l _H · s _H + l _L (S−s _H ) (3)
The processing amount l _L that satisfies the above is calculated, and the image quality of the normal area is determined (S56). Then, the image quality instruction unit 54 displays a warning screen on the display device as to whether or not to allow the user to deteriorate the image quality of the normal area other than the attention area (S58). When the user makes an intention not to accept this with an input device (not shown) (NO in S60), the image quality instruction unit 54 does not allow the image quality of the attention area to be improved (S66). When the intention display is accepted by the user (YES in S60), the image quality instruction unit 54 issues an instruction to improve the image quality of the attention area and reduce the image quality of the normal area (S62). As a result, the decoding processing amount P is set to be equal to or less than the upper limit processing capability P _max .

  FIGS. 15A and 15B are diagrams schematically showing a screen when the user accepts that the image quality of the normal area other than the attention area is deteriorated in S60 of the flowchart of FIG. As shown in FIG. 15A, when the user designates an attention area on the screen while decoding the entire image with medium image quality (M), the attention area is displayed as shown in FIG. The image quality is improved (H), and other normal areas are also reduced in image quality (L).

  In this embodiment, when the user designates an attention area that he / she wants to reproduce with high image quality, the decoding processing amount of the attention area is increased to improve the image quality, and the decoding processing amount of the normal area other than the attention area is reduced accordingly. As a result, the processing amount of the entire image processing apparatus is reduced to the upper limit processing capacity or less. As a result, it is possible to reproduce the region of interest to the user with high image quality without increasing the processing amount of the image processing apparatus. Further, it is possible to avoid the occurrence of frame dropping that occurs when the decoding processing amount exceeds the capability of the image processing apparatus.

  As yet another embodiment, when the attention area is designated by the user, the image quality of the attention area may be maintained at the medium image quality, and the normal area other than the attention area may be reduced to a low image quality. In this case, the low-order bit zero replacement unit 58 can perform decoding by relatively replacing the low-order bits of the wavelet transform coefficient corresponding to the non-ROI region with the image quality of the region of interest higher than that of the normal region. This is shown in FIG. As shown in FIG. 16A, when the user designates an attention area on the screen while decoding the entire image with medium image quality (M), the attention area is displayed as shown in FIG. The normal area is reduced in image quality (L) while maintaining the medium image quality. As a result, a clear display screen can be obtained only in the attention area, and the subjective quality of the user is increased.

  In the above description, the image quality is described as three stages of high, medium, and low, but there may be more image quality stages depending on the number of lower bits to be replaced with zero.

  The user may specify a plurality of attention areas. When two regions of interest are designated, the image quality instruction unit 54 may increase the image quality of one region of interest and the image quality of the other region of interest as it is according to the decoding processing amount. Instead of being designated by the user, the attention area may be set by automatically extracting an important area such as an area in which a person or a character is captured by the position information creation unit 50.

When the decoding processing amount P exceeds the upper limit processing capacity P _max in the determination when the attention area has high image quality, the image quality instruction unit 54 reduces the frame rate of the moving image to be output to the decoding unit 310. May be commanded as follows. As a result, the decoding processing amount per unit time of the entire image processing apparatus is reduced, so that although the time resolution is reduced, it is possible to achieve high image quality of the attention area.

Fourth embodiment.
FIG. 17 is a configuration diagram of an image display apparatus 400 according to the fourth embodiment. The image display device 400 displays a moving image on a display device such as a display, and corresponds to a display control unit such as a television receiver or a surveillance camera as an example.

  An image decoder 412 in the processing block 410 cooperates with the CPU 414 and the memory 416 to continuously decode the input encoded image data stream. The image decoder 412 has the configuration of the image processing apparatus 300 of the third embodiment. Note that the processing block 410 may acquire the encoded image data stream via a communication interface of a wireless or wired network, and acquire the encoded image data stream via a reception block that receives broadcast radio waves. May be.

  The display circuit 418 receives the decoded image from the processing block 410 and outputs it to the display device 420. The decoded image frames are continuously displayed on the display device 420, and a moving image is reproduced.

  The user designates the attention area in the image displayed on the display device 420 using the input device 424 such as a pointing device, or designates the attention area using a contact-type display device such as a touch panel. Information on the region of interest is input to the processing block 410 via the interface 422. The processing block 410 receives the information on the attention area and generates a decoded image in which the image quality of the attention area is different.

  According to the image display device 400, it is possible to display only a region selected by the user in high quality for a movie or video from a surveillance camera.

Fifth embodiment.
The fifth embodiment of the present invention is a display device that receives an encoded image data stream layered according to resolution, continuously decodes each encoded frame by a decoding unit, and displays a low-resolution moving image. And an image display device that provides moving image data to both of the display devices that display a high-resolution moving image, and in either one of the display devices, the user instructs to improve the image quality of a partial area of the image. When this is the case, the image display apparatus executes an improvement in image quality for both the low-resolution moving image and the high-resolution moving image.

  FIG. 18 shows a configuration of an image display system 500 according to the fifth embodiment. Since the display circuits 218 and 220, the first display device 222, and the second display device 224 are the same as those in the second embodiment, they are denoted by the same reference numerals. The decoding unit 512 and the area specifying unit 514 have the same configuration as the decoding unit 310 and the area specifying unit 320 in the third embodiment shown in FIG.

  The decoding unit 512 of the image processing apparatus 510 continuously decodes the input encoded image data stream. Then, the high resolution image data is output to the first display device 222 that displays the high resolution moving image via the frame buffer 516 and the display circuit 218, and the low resolution image data is output to the frame buffer 518 and the display circuit 220. Via the second display device 224 for displaying a low-resolution moving image. This process is executed according to the procedure shown in the first embodiment. Then, the first display device 222 and the second display device 224 continuously display the decoded image data at a given frame rate, and a moving image is reproduced. The image processing apparatus 510 may acquire the encoded image data stream via a communication interface of a wireless or wired network, or may acquire the encoded image data stream via a reception block that receives broadcast radio waves. Also good.

  A user designates a region of interest in an image displayed on the first display device 222 or the second display device 224 using an input device 524 such as a pointing device, or a touch-type display device such as a touch panel. Specify the region of interest using. Information on the attention area is input to the image processing apparatus 510 via the interface 522. The area designation unit 514 receives information on the attention area, determines whether or not the attention area should be improved in image quality, and informs the decoding unit 512 of the result. The decoding unit 512 generates image data with a difference in image quality between the attention area and the other normal area for each of the high resolution image data and the low resolution image data according to the determination result. This process is executed according to the procedure shown in the third embodiment. Thereafter, a moving image is reproduced on the first display device 222 and the second display device 224 in the same manner as described above.

  According to this embodiment, when moving images having different resolutions are displayed on a plurality of display devices at the same time, if the attention area is designated on any one of the display devices, the image quality of the attention area is improved also on the other display devices. be able to. For example, in a presentation, when a moving image is reproduced on both a large screen projected by a projector and a PC screen, it is possible to produce an effect that appeals to the participant the image to be emphasized. Further, in the surveillance camera system, when the same surveillance image stream is displayed on the displays of a plurality of surveillance rooms, an image area that should be noted by other surveillance staff can be aroused.

  Needless to say, the image display system 500 may include three or more display devices that display moving images having different resolutions according to the purpose.

  The present invention has been described based on some embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  In any of the embodiments, the wavelet transform has been described as spatial filtering for image coding, but other spatial frequency transforms may be used. For example, even in the case of the discrete cosine transform used in the JPEG standard, by substituting the lower bits of the transform coefficient of the normal area with zero by the same method, the image quality of the attention area is relatively improved while the image quality of the normal area is improved. By sacrificing, the processing amount of the entire image processing apparatus can be reduced.

It is a figure which shows the procedure of an image encoding process. 1 is a configuration diagram of an image processing apparatus according to a first embodiment. It is a figure which shows the procedure of an image decoding process. It is a figure explaining the process of the frame by an image processing apparatus. It is a flowchart explaining the process in a memory control part. It is a block diagram of the image display apparatus which concerns on 2nd Embodiment. It is a block diagram of the image processing apparatus which concerns on 3rd Embodiment. (A)-(c) is a figure explaining the mask for specifying the wavelet transformation coefficient corresponding to the attention area | region of an original image. (A), (b) is a figure explaining a mode that the low-order bit of a wavelet transform coefficient is zero-substituted. (A)-(c) is a figure explaining the wavelet transform coefficient when an attention area | region is designated to the original image. It is a flowchart explaining the process in a determination part. (A), (b) is a figure which shows a mode that an attention area | region becomes high quality. (A)-(c) is a figure explaining a mode that the low-order bit of a wavelet transform coefficient is zero-substituted when an attention area is specified for an original image and the amount of processing is large. It is a flowchart explaining another Example of the process in a determination part. (A), (b) is a figure which shows a mode that an attention area | region is improved in image quality, and a normal area | region is reduced in image quality. (A), (b) is a figure which shows a mode that an attention area is left as it is, and a normal area | region is reduced in image quality. It is a block diagram of the image display apparatus which concerns on 4th Embodiment. It is a block diagram of the image display system which concerns on 5th Embodiment.

Explanation of symbols

  10 stream analysis unit, 12 arithmetic decoding unit, 14 bit plane decoding unit, 18 inverse quantization unit, 20 wavelet inverse transformation unit, 22 memory control unit, 30 low resolution frame buffer, 32 low resolution display circuit, 34, 44 resolution conversion 36, low-resolution display device, 40 high-resolution frame buffer, 42 high-resolution display circuit, 46 high-resolution display device, 50 position information creation unit, 52 determination unit, 54 image quality instruction unit, 56 ROI mask generation unit, 58 low-order bit zero Replacement unit, 60 display circuit, 62 display device, 100, 300 image processing device, 150, 310 decoding unit, 320 region designation unit, 200, 400 image display device, 500 image display system.

Claims (6)

An area designating part for designating an attention area on the screen;
A decoding unit that decodes an image with a difference in image quality between the attention region and the other normal region, and outputs moving image data;
With
The decoding unit performs simple reproduction by discarding a predetermined number of low-order bits of image data during normal times when the region of interest is not designated, and when the region of interest is designated by the region designation unit The image processing apparatus is characterized in that the image quality of the attention area is improved by decoding the lower bits of the image data corresponding to the attention area by reducing the number of bits to be discarded as compared with the simple reproduction.   The image processing apparatus according to claim 1, wherein the decoding unit adjusts the image quality of the attention area and the normal area so that a processing amount of the entire decoding process does not exceed a predetermined threshold value. . The area specifying unit is
A determination unit that determines whether or not a processing amount related to decoding exceeds a predetermined threshold when the image quality of the attention area is improved;
An image quality instruction unit that determines whether or not to improve the image quality of the region of interest according to the determination result;
The image processing apparatus according to claim 2, further comprising: The area specifying unit is
A determination unit that determines whether or not a processing amount related to decoding exceeds a predetermined threshold when the image quality of the attention area is improved;
When it is determined that the amount of processing exceeds the threshold value as a result of the determination by the determination unit, the decoding unit performs simple reproduction of the number of bits to be discarded for the lower bits of the image data corresponding to the normal area. The image processing apparatus according to claim 2, wherein the image quality of the normal area is reduced by decoding more than the time. 5. The display according to claim 4, wherein when the processing amount is found to exceed the threshold, the decoding unit displays a warning to the user that the image quality of the normal area is reduced. Image processing device. 5. The display according to claim 4, wherein when the processing amount is found to exceed the threshold value, the decoding unit displays an inquiry as to whether or not to improve the image quality of the attention area. 5. The image processing apparatus according to 5.

Images