JP2010206847A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To simultaneously display a plurality of moving images with different resolutions by using an encoded image data stream. <P>SOLUTION: A decoding unit 150 decodes encoded image data. A low-resolution frame buffer 30 records image data of low resolution from output of the decoding unit 150. A high-resolution frame buffer 40 records image data of high resolution from the output of the decoding unit 150. A low-resolution display circuit 32 acquires data from the low-resolution frame buffer 30 and generates a display signal for a low-resolution display device 36 that displays a low-resolution moving image. A high-resolution display circuit 42 acquires data from the high-resolution frame buffer 40 and generates a display signal for a high-resolution display device 46 for displaying a high-resolution moving image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus.

  As display technology for thin displays such as liquid crystal displays and plasma displays has improved and prices have fallen, display devices of various sizes now exist around us and provide moving images. . These display devices have various resolutions ranging from a liquid crystal display of a mobile phone to a large high-resolution display. Each display device performs a decoding process on the encoded image data stream to provide a moving image corresponding to each resolution.

  As an example of such a technique, Patent Document 1 discloses a moving image reproduction processing apparatus that performs decoding processing at a resolution corresponding to a display size. This apparatus includes a plurality of decoding processing units that compare the display size with the size of the original image and decode it into an image having a resolution of the display size. According to this apparatus, it is possible to correspond to display apparatuses having various resolutions by using the same encoded image data stream.

JP 2002-94994 A

  In the future, as the distribution and utilization of video contents become popular, it is considered that there will be a scene where it is necessary to simultaneously display a plurality of moving images having different resolutions using a single encoded image data stream. However, since the technique described in Patent Document 1 outputs an image with a resolution that is fixedly obtained by the decoding processing unit selected by the resolution selection processing unit, a plurality of display devices can be obtained from a single encoded image data stream. In addition, it is not possible to display moving images having different resolutions, and it is only possible to deal with resolutions of types of decoding processing units prepared in advance.

  The present invention has been made in view of these problems, and an object thereof is to provide an apparatus that displays moving images having different resolutions on a plurality of display devices.

  An aspect of the present invention includes a decoding unit that decodes encoded image data to generate a moving image, and an area designation unit that designates a region of interest for a screen. When the image is decoded with medium quality when not specified and the attention area is specified, the image quality of the attention area is set higher than the medium image quality, and the image quality of other normal areas is lower than the medium image quality. An image processing apparatus that decodes an image with low image quality and outputs moving image data.

  The decoding unit may lower the image quality of the other normal areas according to the amount of decoding processing increased to increase the image quality of the attention area.

  The decoding unit decodes the encoded image data to generate two moving images having different resolutions, and generates a display signal for displaying one of the two moving images on the display device. You may further provide a display circuit and the 2nd display circuit which produces | generates the display signal for displaying the other moving image on the display apparatus different from a display apparatus among two moving images.

  In the area designating unit, when receiving an instruction to decode with high image quality for the attention area of either one of the attention areas of the two moving images, the decoding section The image quality of the attention area may be set to high image quality, and the image quality of the other normal areas may be set to low image quality for decoding to output moving image data.

  The encoded image data may be multiplexed according to the resolution.

 It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a computer program, a recording medium, etc. are also effective as an aspect of the present invention.

  According to the present invention, moving images having different resolutions can be displayed on a plurality of display devices, respectively.

It is a figure which shows the procedure of an image encoding process. 1 is a configuration diagram of an image processing apparatus according to a first embodiment. It is a figure which shows the procedure of an image decoding process. It is a figure explaining the process of the frame by an image processing apparatus. It is a flowchart explaining the process in a memory control part. It is a block diagram of the image display apparatus which concerns on 2nd Embodiment. It is a block diagram of the image processing apparatus which concerns on 3rd Embodiment. (A)-(c) is a figure explaining the mask for specifying the wavelet transformation coefficient corresponding to the attention area | region of an original image. (A), (b) is a figure explaining a mode that the low-order bit of a wavelet transform coefficient is zero-substituted. (A)-(c) is a figure explaining the wavelet transform coefficient when an attention area | region is designated to the original image. It is a flowchart explaining the process in a determination part. (A), (b) is a figure which shows a mode that an attention area | region becomes high quality. (A)-(c) is a figure explaining a mode that the low-order bit of a wavelet transform coefficient is zero-substituted when an attention area is specified for an original image and the amount of processing is large. It is a flowchart explaining another Example of the process in a determination part. (A), (b) is a figure which shows a mode that an attention area | region is improved in image quality, and a normal area | region is reduced in image quality. (A), (b) is a figure which shows a mode that an attention area remains as it is, and a normal area | region is reduced in image quality. It is a block diagram of the image display apparatus which concerns on 4th Embodiment. It is a block diagram of the image display system which concerns on 5th Embodiment.

  The present invention relates to a technique for generating moving images having different resolutions or image quality from an encoded image data stream. Hereinafter, preferred embodiments of the present invention will be described. In the embodiment, an image processing apparatus that decodes an encoded image data stream compliant with Motion-JPEG2000 as image processing will be considered as an example.

  First, with reference to FIG. 1, a method for encoding a moving image by the Motion-JPEG2000 method will be briefly described. An image encoding device (not shown) continuously encodes each frame of a moving image in units of frames to generate an encoded data stream of the moving image. At the start of the encoding process, an original image OI (Original Image) 102 corresponding to one frame of a moving image is read into the frame buffer. The original image OI read into the frame buffer is hierarchized by a wavelet transformer.

The wavelet transformer in JPEG2000 uses a Daubechies filter. This filter acts as a high-pass filter and a low-pass filter simultaneously in the x and y directions of the image, and divides one image into four frequency subbands. These subbands include an LL subband having a low frequency component in both the x and y directions, an HL subband having a low frequency component in one of the x and y directions and a high frequency component in the other direction, and An LH subband and an HH subband having a high frequency component in both the x and y directions. This filter also has an action of reducing the number of pixels by half in both the x and y directions. That is, the number of vertical and horizontal pixels of each subband is ½ of that of the image before processing, and a subband image with a resolution, that is, an image size of ¼ can be obtained by one filtering. In this specification, an image that has undergone wavelet transform once with respect to the original image OI is referred to as first-layer image WI _1, and is hereinafter referred to as n-th layer image WI _n according to the number of times of undergoing wavelet transform. I will call it.

As schematically shown in FIG. 1, the image WI ₁ 104 in the first layer includes four subbands L
L ₁ , HL ₁ , LH ₁ and HH ₁ are generated. Wavelet transform is applied to the first layer image WI ₁ 104 to generate a second layer image WI ₂ 106. Here, the second and subsequent wavelet transforms are performed only on the LL subband component in the immediately preceding layer image. Therefore, in the second layer image WI ₂ 106, the first layer image WI _{1 is used.}
LL ₁ subbands are decomposed into four subbands LL ₂ , HL ₂ , LH ₂ , and HH ₂ . The wavelet transformer executes this filtering a predetermined number of times and outputs wavelet transform coefficients for each subband. Thereafter, the image encoding device performs quantization and other processing, and finally outputs encoded image data CI (Coded Image).

For simplicity of explanation, in this example, it is assumed that the image encoding device performs wavelet transformation three times on the original image OI. Thus, for example, the original image OI 102 is 1440 ×
If it is 960 pixels, the size of the LL ₁ subband of the first layer image WI ₁ 104 is 720 × 480, and the LL ₂ subband of the second layer image WI ₂ 106 is 360 ×.
240, the size of the LL ₃ subband of the third layer image WI ₃ 108 is 180 × 120.

It should be noted about the hierarchized image that the low frequency component in the original image OI appears in the upper left in FIG. In the case of FIG. 1, the LL ₃ subband in the upper left corner of the third layer image WI ₃ has the lowest frequency. Conversely, if the LL ₃ subband can be obtained, the most basic of the original image OI. Realistic properties can be reproduced. This knowledge is used in the following embodiments.

In addition to Motion-JPEG2000, the encoded data stream may be, for example, an SVC (Scalable Video Codec) having both a high-quality HD stream and a low-quality SD stream in one stream, or Motion-JPEG. . In JPEG, each frame is transmitted from the low-order term of the Fourier coefficient, so that the image quality can be selected depending on which order term is used for decoding.
First embodiment.
The first embodiment is an image processing apparatus that receives an encoded image data stream layered according to resolution and can provide moving images having different resolutions to a plurality of display devices.

  FIG. 2 shows a configuration of the image processing apparatus 100 according to the first embodiment. This configuration can be realized in terms of hardware by a CPU, memory, or other LSI of an arbitrary computer, and in terms of software, it is realized by a program having a decoding function loaded in the memory. The functional block realized by those cooperation is drawn. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

The decoding unit 150 of the image processing apparatus 100 receives a stream of encoded image data CI. The decoding unit 150 receives the encoded image data CI, analyzes the data stream, the stream analysis unit 10, the arithmetic decoding unit 12 that performs arithmetic decoding on the data string to be decoded that is found as a result of the analysis, and the result A bit plane decoding unit 14 that decodes the obtained data in the form of a bit plane for each color component, an inverse quantization unit 18 that inversely quantizes the decoded quantized data, and a first obtained as a result of the inverse quantization A wavelet inverse transform unit 20 that performs wavelet inverse transform on the n-layer image WI _n is included. Each time wavelet inverse transform is performed on the encoded image data CI by the wavelet inverse transform unit 20, a higher-level image is obtained, and finally, decoded image data DI (Decoded Image) can be obtained.

The present embodiment is characterized in that an n-th layer image generated during the wavelet inverse transformation for obtaining the decoded image DI in the wavelet inverse transformation unit 20 is output to the low resolution frame buffer 30. The image processing apparatus 100 provides image data corresponding to the respective resolutions to the low-resolution display device 36 that displays a low-resolution moving image and the high-resolution display device 46 that displays a high-resolution moving image. Operate. For this purpose, the memory control unit 22 acquires resolution information of moving images displayed on the low-resolution display device 36 and the high-resolution display device 46, and performs the wavelet inverse transformation on the encoded image CI several times. And the result is transmitted to the wavelet inverse transform unit 20. In accordance with this information, the wavelet inverse transform unit 20 uses the LL subband of the n-th layer image WI _n in the middle of the wavelet inverse transform process, or the decoded image data DI that is a complete decoding result as the low resolution frame buffer 30. Alternatively, the high resolution frame buffer 40 is written. This operation will be described later with reference to FIG. Note that the low resolution frame buffer 30 and the high resolution frame buffer 40 are distinguished for convenience and do not necessarily have different buffer sizes.

  The image data written in the low resolution frame buffer 30 is generated as a display signal by the low resolution display circuit 32 and displayed on the low resolution display device 36. Similarly, the image data written in the high resolution frame buffer 40 is generated as a display signal by the high resolution display circuit 42 and displayed on the high resolution display device 46. As described above, according to the image processing apparatus 100, it is possible to simultaneously display moving images having different resolutions on a plurality of display devices using the encoded image data stream.

  Either or both of the low resolution display circuit 32 and the high resolution display circuit 42 may include resolution conversion units 34 and 44. Thus, when the resolution obtained by the wavelet inverse transform processing in the decoding unit 150 is different from the resolution of the moving image displayed on each display device 36, 46, the image of the hierarchy having the closest resolution is decoded. Thereafter, the resolution can be converted to a desired resolution by the resolution converters 34 and 44. These resolution conversion units 34 and 44 are optional, and if it is not desired to display a moving image having a resolution other than the resolution obtained by the wavelet inverse transformation process, the low resolution display circuit 32 or the high resolution display circuit. 42 may not include the resolution converters 34 and 44, respectively.

  FIG. 3 shows a processing procedure in the decoding unit 150. Here, as described above, it is assumed that a stream of encoded image data obtained by performing wavelet transform three times on the original image OI is given to the image processing apparatus 100.

First, the encoded image data CI input to the image processing apparatus 100 undergoes processing of the stream analysis unit 10, the arithmetic decoding unit 12, the bit plane decoding unit 14, and the inverse quantization unit 18, and then the third layer image WI _3. Return to the state of 122. Subsequently, the first wavelet inverse transformation by the wavelet inverse transformation unit 20 obtains the second layer image WI ₂ 124, and the second wavelet inverse transformation obtains the image WI ₁ 126 corresponding to the first layer. Finally, a decoded image DI 128 is obtained by the third wavelet inverse transform.

By the way, as described above, the LL subband of each layer is obtained by extracting a low frequency component from the image of the layer, and the size is 1/4 of the image of the immediately preceding layer. Therefore, it can be said that these are low-resolution images compared to the original image OI. Therefore, as an example, the first layer image WI ₁ obtained by the second wavelet inverse transformation is used.
126 LL ₁ subbands (720 × 480) are output to the low-resolution frame buffer 30 as low-resolution image data, and the decoded image DI (1440 × 960) obtained by the third wavelet inverse transform is a high-resolution image. Data can be output to the high resolution frame buffer 40. Since the x and y directions of the image are each halved by performing the wavelet transform, the larger the number of wavelet transforms performed by the wavelet transformer of the image encoding device, the more types of resolutions. Can be displayed.

  FIG. 4 is a conceptual diagram illustrating generation of moving images having different resolutions from each frame. Based on the command from the memory control unit 22, the wavelet inverse transform unit 20 performs necessary decoding processing on each encoded image frame, and outputs a low resolution image to the low resolution frame buffer 30. Are output to the high resolution frame buffer 40. Then, a low-resolution or high-resolution moving image is formed from the same encoded image data stream by continuously outputting low-resolution or high-resolution images at an intended frame rate.

  FIG. 5 is a flowchart for explaining the operation of the memory control unit 22. First, the memory control unit 22 acquires resolution information of a moving image displayed on the low resolution display device 36 and the high resolution display device 46 (S10). Alternatively, the user may be allowed to input information on the resolution of a moving image displayed on each display device. Next, the memory control unit 22 determines how many LL subbands the low resolution image displayed on the low resolution display device 36 corresponds to the encoded image CI (S12). Subsequently, the memory control unit 22 determines how many layers of the high-resolution image displayed on the high-resolution display device 46 corresponds to the LL subband of the image in the hierarchy or the decoded image DI that has been completely decoded. Determine (S14). Then, when the wavelet inverse transform processing for obtaining the images of the layers determined in S12 and S14 is completed in the wavelet inverse transform unit 20, the memory control unit 22 converts the subband image LL or the decoded image DI into the low resolution. A command is given to write to the frame buffer 30 or the high-resolution frame buffer 40 (S16). Of course, if there is only one display device that should output image data from the image processing device, either the low resolution frame buffer 30 or the high resolution frame buffer 40 may be used.

  As described above, in JPEG2000, the resolution of the LL subband is decreased by 1/2 and 1/2 by the original image, so that an appropriate resolution that matches the resolution of the desired display device is obtained. In some cases, it may not be possible to obtain the image. Therefore, if the memory control unit 22 determines in S12 or S14 that an appropriate resolution cannot be obtained, the memory control unit 22 applies the resolution conversion units 34 and 44 provided in the low resolution display circuit 32 or the high resolution display circuit 42, respectively. It is also possible to execute the interpolation processing of the resolution as appropriate.

The image processing apparatus 100 can also display moving images having different resolutions on three or more display devices by increasing the number of frame buffers. For example, if the above example is used, the LL2 subband (360 × 240) of the image WI ₂ 124 of the second layer obtained by the first wavelet inverse transform is used as the low resolution frame buffer and the second wavelet inverse transform is performed. The LL ₁ subband (720 × 480) of the first-level image WI ₁ 126 obtained by the above is used as a medium resolution frame buffer, and the decoded image DI 128 (1440 × 960) obtained by the third wavelet inverse transformation is used as a high resolution. Each of the resolution frame buffers can be output and a corresponding display circuit can display low resolution, medium resolution, and high resolution moving images on the display device.

As described above, according to the first embodiment, it is possible to simultaneously display moving images having different resolutions on two or more display devices using a single encoded image data stream. Conventionally, each time decoding is performed according to the target resolution, but by outputting an image in the middle of the decoding process to the frame buffer, image data of a plurality of resolutions can be obtained by a single decoding unit. Can be produced and is efficient.
Second embodiment.
FIG. 6 shows a configuration of an image display apparatus 200 according to the second embodiment. The image display device 200 includes two display devices, such as a first display device 222 that displays a high-resolution moving image and a second display device 224 that displays a low-resolution moving image, such as a display and a projector.

  The image decoder 212 of the processing block 210 cooperates with the CPU 214 and the memory 216 to continuously decode the input encoded image data stream. The image decoder 212 has the configuration of the image processing apparatus 100 of the first embodiment. Then, high-resolution image data is output to the first display device 222 via the display circuit 218, and low-resolution image data is output to the second display device 224 via the display circuit 220. On each screen, the image data decoded by the image decoder 212 is continuously displayed at a given frame rate, and a moving image is reproduced. The processing block 210 may acquire the encoded image data stream via a communication interface of a wireless or wired network, or may acquire the encoded image data stream via a reception block that receives broadcast radio waves. Good.

With this image display device 200, for example, the following operations can be realized.
1. Airplane in-flight screening If you have a large screen in front of the cabin and a small personal LCD on the back of each seat, you need only prepare a single encoded image data stream. The moving image can be reproduced on both the screen and the liquid crystal display.
2. Presentation If a single encoded image data stream is prepared at the time of presentation, a moving image can be reproduced on both a large screen projected by a projector and a PC screen.
3. Two-screen mobile phone By incorporating the above screen display device into a mobile phone having a main display and a sub-display, a single encoded image data stream is received and video content is played on both the main display and the sub-display. Is possible.

Needless to say, the image display device 200 may include three or more display devices that display moving images having different resolutions according to the purpose.
Third embodiment.
In the third embodiment, when a user is instructed to improve the image quality of a partial area of an image in an image processing apparatus that decodes an image stream layered according to resolution or image quality and displays the decoded image stream on a display device In addition, the image processing apparatus controls the processing amount so as not to exceed the maximum capacity of the image processing apparatus.

  FIG. 7 is a diagram illustrating a configuration of an image processing apparatus 300 according to the third embodiment. The image processing apparatus 300 includes a decoding unit 310 that receives a stream of encoded image data CI and decodes the image, and an area designation unit 320 that executes a process related to an attention area designated in the image by the user. The stream analysis unit 10, the arithmetic decoding unit 12, the bit plane decoding unit 14, the inverse quantization unit 18, and the wavelet inverse transformation unit 20 included in the decoding unit 310 are the same as those described in the first embodiment.

  The image data decoded by the decoding unit 310 is displayed on the display device 62 by the display circuit 60. The user uses an input device such as a pointing device (not shown) to designate a region (hereinafter referred to as “region of interest (ROI)”) that is desired to be reproduced with improved image quality. Then, the position information creation unit 50 in the area designation unit 320 creates ROI position information indicating the position of the attention area ROI. This ROI position information is given by the coordinate value of the pixel at the upper left corner of the rectangular area and the number of vertical and horizontal pixels of the rectangular area when the attention area ROI is designated by a rectangle. When the user designates the attention area ROI with a circle or the like, the circumscribed rectangle may be set as the attention area. The attention area may be always set to a predetermined area such as the center area of the original image.

  Based on the created ROI position information, the determination unit 52 calculates the amount of increase in data processing necessary to improve the image quality of the attention area, and the total amount of decoding processing added to the current processing amount is Then, it is determined whether or not the image processing apparatus 300 falls within the maximum processing capacity. Based on the determination result, the image quality instruction unit 54 determines whether or not to allow the image quality of the attention area to be high, or to reduce the image quality of the area other than the attention area (hereinafter referred to as “normal area”). Is output to the ROI mask generator 56. Details of this processing will be described later with reference to FIG. 11 or FIG.

  The ROI mask generation unit 56 generates an ROI mask for specifying a portion corresponding to the region of interest among the wavelet transform coefficients based on the ROI position information from the position information generation unit 50. The generated ROI mask is used by the low-order bit zero replacement unit 58 to adjust the number of low-order bits to be replaced with a zero value in the bit string of the wavelet transform coefficient. By performing inverse wavelet transform on this, it is possible to obtain an image in which the region of interest has a high image quality. This will be described later.

  Here, a method for generating an ROI mask based on the ROI position information by the ROI mask generation unit 56 will be described with reference to FIGS. As illustrated in FIG. 8A, it is assumed that the attention area 90 is designated by the user on the image 80 decoded and displayed by the image processing apparatus 300. The ROI mask generation unit 56 specifies a wavelet transform coefficient necessary for restoring the attention area 90 selected on the image 80 in each subband.

FIG. 8B shows a converted image 82 of the first layer obtained by performing wavelet transform on the image 80 only once. The converted image 82 in the first layer is composed of four subbands LL ₁ , HL ₁ , LH ₁ , and HH _{1 of} the first level. The ROI mask generation unit 56 outputs wavelet transform coefficients (hereinafter referred to as “ROI transform coefficients”) 91 to 94 on the first-level transformed image 82 necessary for restoring the attention area 90 of the image 80 to the first. It is specified in each sub-band LL ₁ , HL ₁ , LH ₁ , HH ₁ of one level.

FIG. 8C shows a second-layer converted image 84 obtained by further wavelet transforming the subband LL ₁ of the converted image 82 of FIG. 8B. As shown in the drawing, the second-level converted image 84 includes the first level three subbands HL ₁ , LH ₁ , HH ₁ and the second level four subbands LL ₂ , HL ₂ , LH _2. , HH ₂ . The ROI mask generation unit 56 uses the wavelet transform coefficients on the second-layer transform image 84 necessary for restoring the ROI transform coefficients 91 in the subband LL ₁ of the first-layer transform image 82, that is, ROI transform coefficients 95˜ 98 is identified in each of the second level subbands LL ₂ , HL ₂ , LH ₂ , HH ₂ .

  Similarly, the ROI transform coefficient corresponding to the attention area 90 is recursively specified in each layer by the number of wavelet transforms, so that the ROI necessary for restoring the attention area 90 in the converted image of the final hierarchy is obtained. All conversion coefficients can be specified. The ROI mask generation unit 56 generates an ROI mask for specifying the position of the finally specified ROI conversion coefficient on the converted image of the final hierarchy. For example, when the wavelet transform is performed only twice, ROI masks that can specify the positions of the seven ROI transform coefficients 92 to 98 indicated by diagonal lines in FIG. 8C are generated.

  Next, with reference to FIGS. 9 and 10, a method for improving the image quality of the attention area will be described. Here, as shown in FIG. 9A, it is assumed that the encoded image data CI is composed of 5-bit planes from MSB (Most Significant Bit) to LSB (Least Significant Bit).

  The image processing apparatus 300 performs simple reproduction in which the lower bit plane of the wavelet transform coefficient is appropriately discarded and reproduced in order to reduce the processing load during normal times when the attention area is not designated by the user. The image quality at this time is referred to as “medium image quality”. In this case, as shown in FIG. 9B, the lower bit zero replacement unit 58 replaces, for example, the lower 2 bits of the bit plane decoded by the bit plane decoding unit 14 with zero, and only the 3 bit plane. To be decrypted. In order to achieve high image quality only in the attention area from this state, it is only necessary to decode a larger number of bit planes only in the attention area.

  FIGS. 10A to 10C show an example of processing for improving the image quality of a region of interest. As shown in FIG. 10A, at the time of simple reproduction, the lower 2 bits are zero-substituted from the LSB side by the lower bit zero replacing unit 58. When the attention area is designated by the user, the ROI mask generation unit 56 generates an ROI mask corresponding to the attention area. This is indicated by the oblique lines in FIG. Then, as shown in FIG. 10C, the low-order bit zero replacement unit 58 generates a wavelet transform coefficient by referring to the ROI mask and replacing only the low-order 2 bits of the non-ROI portion that is not ROI masked with zero. .

  The inverse quantization unit 18 inversely quantizes the generated wavelet transform coefficient, and the wavelet inverse transform unit 20 inversely transforms the inversely quantized wavelet transform coefficient. Thereby, image data in which only the attention area ROI is improved in image quality is obtained.

  Next, processing in the determination unit 52 will be described with reference to the flowchart in FIG. As a premise, it is assumed that the moving image is displayed with the above-described medium image quality at the normal time when the attention area is not designated by the user.

  First, the determination unit 52 receives ROI position information of a region of interest from the position information creation unit 50 (S30). Next, the area (or the number of pixels) of the attention area is calculated from the ROI position information, and the entire decoding processing amount P of the image processing apparatus 300 is calculated (S32).

Here, the decoding processing amount P can be expressed as a sum of (processing amount of each image quality) × (area of each image quality). When the processing amount per unit area at the time of low image quality is l _L , the processing amount per unit area at the time of medium image quality is l _M , the processing amount per unit area at the time of high image quality is l _H, and the area of the entire image is expressed as S, The normal decryption processing amount is
P = 1 _M · S (1)
It becomes.

The area of the attention area designated by the user is s _H, and the decoding processing amount P when the attention area is improved in image quality is:
P = l _H · s _H + l _M (S−s _H ) (2)
Can be calculated with

The determination unit 52 determines whether or not the decoding processing amount P calculated by Expression (2) exceeds the upper limit processing capability P _max that the image processing apparatus 300 can process in one frame period (S34). If the decoding processing amount P is less than or equal to the upper limit processing capacity P _max (NO in S34), the image quality instruction unit 54 permits the image quality of the attention area to be improved (S36). If the decoding processing amount P exceeds the upper limit processing capability P _max (YES in S34), the processing capability sufficient for high-quality decoding does not remain in the image processing apparatus 300. Is not permitted to improve the image quality (S38).

FIGS. 12A and 12B schematically show the state of the screen when it is determined in S34 of the flowchart of FIG. 11 that the decoding processing amount P is equal to or less than the upper limit processing capability _Pmax . In the figure, “L” indicates a low image quality region, “M” indicates a medium image quality region, and “H” indicates a high image quality region. As shown in FIG. 12A, when the user designates a region of interest on the screen while decoding the entire image with medium quality, only the region of interest has high image quality as shown in FIG. (H), and other normal areas remain medium image quality (M).

  As described above, according to the image processing apparatus of the present embodiment, when the user designates a region of interest that the user wants to reproduce with high image quality in the decoded and displayed image, there is a margin in the decoding processing capability of the image processing apparatus. Sometimes, the attention area is improved in image quality, and when the decoding processing capacity is not sufficient, the attention area is not improved in image quality.

  As described above, when the attention area is designated, only the attention area can be reproduced with higher quality while the normal area has the same quality as the simple reproduction. This is particularly useful when a high quality is not required during normal times and a point of interest is to be reproduced with high quality only when an abnormality occurs, as in a surveillance video.

  Next, another example of processing when the processing capability of the image processing apparatus 300 is insufficient when the attention area is improved in image quality will be described with reference to FIGS.

  As shown in FIG. 13A, at the time of simple reproduction, it is assumed that the lower 2 bits are zero-substituted from the LSB side by the lower bit zero replacing unit 58. When the attention area is designated by the user, the ROI mask generation unit 56 generates an ROI mask corresponding to the attention area. This is indicated by the oblique lines in FIG. This time, the area of the attention area is larger than that in the case of FIG. 10B, and if the attention area is improved in image quality, the processing capability of the image processing apparatus 300 becomes insufficient. In this case, as shown in FIG. 13C, the lower bit zero replacement unit 58 refers to the ROI mask and replaces the lower 3 bits instead of the lower 2 bits with zero for the non-ROI portion that is not ROI masked. Generate wavelet transform coefficients.

  Then, the inverse quantization unit 18 inversely quantizes the generated wavelet transform coefficient, and the wavelet inverse transform unit 20 inversely transforms the inversely quantized wavelet transform coefficient. As a result, image data in which the attention area ROI has a high image quality and the normal area has a low image quality can be obtained. As described above, when the ROI masked region of interest is improved in image quality, that is, when the number of bit planes is increased, if the processing capability of the image processing apparatus is insufficient, the number of bit planes in the normal region that is not ROI masked is reduced. Thus, the processing amount of the entire image processing apparatus is balanced.

  Next, with reference to the flowchart of FIG. 14, processing in the determination unit 52 when the processing capability of the image processing apparatus 300 is insufficient when the attention area is improved in image quality will be described. As in the case of FIG. 11, it is assumed that the moving image is displayed with the above-described medium image quality in the normal time when the attention area is not designated by the user.

Receiving the attention area (S50) and calculating the decoding processing amount P of the entire image processing apparatus 300 (S52) are the same as S30 and S32 in FIG. The determination unit 52 determines whether or not the decoding processing amount P calculated in S52 exceeds the upper limit processing capacity _Pmax that the image processing apparatus 300 can process in one frame period (S54). If the decoding processing amount P is less than or equal to the upper limit processing capability P _max (NO in S54), the image quality instruction unit 54 permits the image quality of the attention area to be improved (S64).

When the decryption processing amount P exceeds the upper limit processing capacity P _max , the determination unit 52
P = l _H · s _H + l _L (S−s _H ) (3)
The processing amount l _L satisfying the above is calculated, and the image quality of the normal area is determined (S56). Then, the image quality instruction unit 54 displays a warning screen on the display device as to whether or not to allow the user to deteriorate the image quality of the normal area other than the attention area (S58). When the user makes an intention not to accept this with an input device (not shown) (NO in S60), the image quality instruction unit 54 does not allow the image quality of the attention area to be improved (S66). When the intention display is accepted by the user (YES in S60), the image quality instruction unit 54 issues an instruction to improve the image quality of the attention area and reduce the image quality of the normal area (S62). As a result, the decoding processing amount P is set to be equal to or less than the upper limit processing capability P _max .

  FIGS. 15A and 15B are diagrams schematically showing a screen when the user accepts that the image quality of the normal area other than the attention area is deteriorated in S60 of the flowchart of FIG. As shown in FIG. 15A, when the user designates an attention area on the screen while decoding the entire image with medium image quality (M), the attention area is displayed as shown in FIG. The image quality is improved (H), and other normal areas are also reduced in image quality (L).

  In this embodiment, when the user designates an attention area that he / she wants to reproduce with high image quality, the decoding processing amount of the attention area is increased to improve the image quality, and the decoding processing amount of the normal area other than the attention area is reduced accordingly. As a result, the processing amount of the entire image processing apparatus is reduced to the upper limit processing capacity or less. As a result, it is possible to reproduce the region of interest to the user with high image quality without increasing the processing amount of the image processing apparatus. Further, it is possible to avoid the occurrence of frame dropping that occurs when the decoding processing amount exceeds the capability of the image processing apparatus.

  As yet another embodiment, when the attention area is designated by the user, the image quality of the attention area may be maintained at the medium image quality, and the normal area other than the attention area may be reduced to a low image quality. In this case, the low-order bit zero replacement unit 58 can perform decoding by relatively replacing the low-order bits of the wavelet transform coefficient corresponding to the non-ROI region with the image quality of the region of interest higher than that of the normal region. This is shown in FIG. As shown in FIG. 16A, when the user designates an attention area on the screen while decoding the entire image with medium image quality (M), the attention area is displayed as shown in FIG. The normal area is reduced in image quality (L) while maintaining the medium image quality. As a result, a clear display screen can be obtained only in the attention area, and the subjective quality of the user is increased.

  In the above description, the image quality is described as three stages of high, medium, and low, but there may be more image quality stages depending on the number of lower bits to be replaced with zero.

  The user may specify a plurality of attention areas. When two regions of interest are designated, the image quality instruction unit 54 may increase the image quality of one region of interest and the image quality of the other region of interest as it is according to the decoding processing amount. Instead of being designated by the user, the attention area may be set by automatically extracting an important area such as an area in which a person or a character is captured by the position information creation unit 50.

When the decoding processing amount P exceeds the upper limit processing capacity P _max in the determination when the attention area has high image quality, the image quality instruction unit 54 reduces the frame rate of the moving image to be output to the decoding unit 310. May be commanded as follows. As a result, the decoding processing amount per unit time of the entire image processing apparatus is reduced, so that although the time resolution is reduced, it is possible to achieve high image quality of the attention area.
Fourth embodiment.
FIG. 17 is a configuration diagram of an image display apparatus 400 according to the fourth embodiment. The image display device 400 displays a moving image on a display device such as a display, and corresponds to a display control unit such as a television receiver or a surveillance camera as an example.

  An image decoder 412 in the processing block 410 cooperates with the CPU 414 and the memory 416 to continuously decode the input encoded image data stream. The image decoder 412 has the configuration of the image processing apparatus 300 of the third embodiment. Note that the processing block 410 may acquire the encoded image data stream via a communication interface of a wireless or wired network, and acquire the encoded image data stream via a reception block that receives broadcast radio waves. May be.

  The display circuit 418 receives the decoded image from the processing block 410 and outputs it to the display device 420. The decoded image frames are continuously displayed on the display device 420, and a moving image is reproduced.

  The user designates the attention area in the image displayed on the display device 420 using the input device 424 such as a pointing device, or designates the attention area using a contact-type display device such as a touch panel. Information on the region of interest is input to the processing block 410 via the interface 422. The processing block 410 receives the information on the attention area and generates a decoded image in which the image quality of the attention area is different.

According to the image display device 400, it is possible to display only a region selected by the user in high quality for a movie or a video from a surveillance camera.
Fifth embodiment.
The fifth embodiment of the present invention is a display device that receives an encoded image data stream layered according to resolution, continuously decodes each encoded frame by a decoding unit, and displays a low-resolution moving image. And an image display device that provides moving image data to both of the display devices that display a high-resolution moving image, and in either one of the display devices, the user instructs to improve the image quality of a partial area of the image. When this is the case, the image display apparatus executes an improvement in image quality for both the low-resolution moving image and the high-resolution moving image.

  FIG. 18 shows a configuration of an image display system 500 according to the fifth embodiment. Since the display circuits 218 and 220, the first display device 222, and the second display device 224 are the same as those in the second embodiment, they are denoted by the same reference numerals. The decoding unit 512 and the area specifying unit 514 have the same configuration as the decoding unit 310 and the area specifying unit 320 in the third embodiment shown in FIG.

  The decoding unit 512 of the image processing apparatus 510 continuously decodes the input encoded image data stream. Then, the high resolution image data is output to the first display device 222 that displays the high resolution moving image via the frame buffer 516 and the display circuit 218, and the low resolution image data is output to the frame buffer 518 and the display circuit 220. Via the second display device 224 for displaying a low-resolution moving image. This process is executed according to the procedure shown in the first embodiment. Then, the first display device 222 and the second display device 224 continuously display the decoded image data at a given frame rate, and a moving image is reproduced. The image processing apparatus 510 may acquire the encoded image data stream via a communication interface of a wireless or wired network, or may acquire the encoded image data stream via a reception block that receives broadcast radio waves. Also good.

  A user designates a region of interest in an image displayed on the first display device 222 or the second display device 224 using an input device 524 such as a pointing device, or a touch-type display device such as a touch panel. Specify the region of interest using. Information on the attention area is input to the image processing apparatus 510 via the interface 522. The area designation unit 514 receives information on the attention area, determines whether or not the attention area should be improved in image quality, and informs the decoding unit 512 of the result. The decoding unit 512 generates image data with a difference in image quality between the attention area and the other normal area for each of the high resolution image data and the low resolution image data according to the determination result. This process is executed according to the procedure shown in the third embodiment. Thereafter, a moving image is reproduced on the first display device 222 and the second display device 224 in the same manner as described above.

  According to this embodiment, when moving images having different resolutions are displayed on a plurality of display devices at the same time, if the attention area is designated on any one of the display devices, the image quality of the attention area is improved also on the other display devices. be able to. For example, in a presentation, when moving images are reproduced on both a large screen projected by a projector and a PC screen, it is possible to produce an effect that appeals to the participant about the image to be emphasized. Further, in the surveillance camera system, when the same surveillance image stream is displayed on the displays of a plurality of surveillance rooms, an image area that should be noted by other surveillance personnel can be aroused.

  Needless to say, the image display system 500 may include three or more display devices that display moving images having different resolutions according to the purpose.

  The present invention has been described based on some embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  In any of the embodiments, the wavelet transform has been described as spatial filtering for image coding, but other spatial frequency transforms may be used. For example, even in the case of the discrete cosine transform used in the JPEG standard, by substituting the lower bits of the transform coefficient of the normal area with zero by the same method, the image quality of the attention area is relatively improved while the image quality of the normal area is improved. By sacrificing, the processing amount of the entire image processing apparatus can be reduced.

  10 stream analysis unit, 12 arithmetic decoding unit, 14 bit plane decoding unit, 18 inverse quantization unit, 20 wavelet inverse transformation unit, 22 memory control unit, 30 low resolution frame buffer, 32 low resolution display circuit, 34, 44 resolution conversion 36, low-resolution display device, 40 high-resolution frame buffer, 42 high-resolution display circuit, 46 high-resolution display device, 50 position information creation unit, 52 determination unit, 54 image quality instruction unit, 56 ROI mask generation unit, 58 low-order bit zero Replacement unit, 60 display circuit, 62 display device, 100, 300 image processing device, 150, 310 decoding unit, 320 region designation unit, 200, 400 image display device, 500 image display system.

Claims (5)

A decoding unit that decodes the encoded image data to generate a moving image;
An area designating unit for designating a region of interest on the screen,
The decoding unit decodes the image with medium image quality when the attention area is not designated, and when the attention area is designated, the image quality of the attention area is set to be higher than the medium image quality, and other normal areas An image processing apparatus that decodes an image with a low image quality lower than a medium image quality and outputs moving image data.   2. The image processing according to claim 1, wherein the decoding unit reduces the image quality of the other normal areas to a low image quality in accordance with an amount of decoding processing increased to increase the image quality of the attention area. apparatus. The decoding unit decodes the encoded image data to generate two moving images having different resolutions,
A first display circuit for generating a display signal for displaying one of the two moving images on a display device;
2. A second display circuit that generates a display signal for displaying the other moving image of the two moving images on a display device different from the display device. 2. The image processing apparatus according to 2. In the area designating unit, when receiving an instruction to decode the attention area of either one of the two moving images with high image quality, the attention area of either one of the moving images is received.
The decoding unit decodes both of the two moving images with a high image quality of the attention area and a low image quality of the other normal areas, and outputs moving image data. The image processing apparatus according to any one of claims 1 to 3.   The image processing apparatus according to claim 1, wherein the encoded image data is multiplexed according to resolution.

Images