JP4201780B2 - Image processing apparatus, image display apparatus and method - Google Patents

Description

  The present invention relates to an image processing device, an image display device, and a method for decoding an encoded image data stream having a hierarchical structure.

In recent years, with the spread of infrastructure such as DVD media, digital broadcasting, and on-demand communication via a network, the use of digital contents has been expanded. Since digital content does not deteriorate in quality due to copying, copyright management becomes more important. The demand for legitimate protection of copyrighted works such as digital contents is expected to increase further in the future. Patent Document 1 discloses a method of prohibiting duplication according to copy restriction information of a stream.
JP 2001-86444 A

  However, such a method is a uniform process of permitting or prohibiting, and it is difficult to satisfy various levels of requirements on both the image providing side and the user side. For example, the image provider may lose business opportunities by simply prohibiting copying. The user side also has various orientations from full-scale to low-priced.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing apparatus and an image display apparatus capable of providing a flexible image that is beneficial to both the image providing side and the user side using the image. And to provide a method.

In order to solve the above problems, an image processing apparatus according to an aspect of the present invention acquires encoded image data multiplexed in a state that can be decoded with a plurality of image quality, and at least two images having different image quality are obtained. A plurality of decoded image data based on copyright management information that regulates permission and prohibition of use according to a combination of image quality and usage of the image data; A copyright management unit that performs copyright management . “Image quality” may include resolution. The “copyright management information” may define the permitted image quality level when using the encoded image data, or may define a combination of the use of the encoded image data and the permitted image quality level. There may be.

According to this aspect, it is possible to provide a flexible image that is beneficial to both the image providing side and the user side using the image. In addition, a plurality of images with different image quality can be processed in real time. Furthermore, flexible copyright management becomes possible. For example, it is possible to provide image quality for which copyright management is performed and image quality for which copyright management is not performed.

Another embodiment of the present invention is also an image processing apparatus. The apparatus acquires encoded image data multiplexed in a state that can be decoded with a plurality of image quality, decodes the encoded image data with a plurality of decodable image quality, and the image quality and use of the image data And a copyright management unit that performs copyright management on a plurality of decrypted image data based on copyright management information that regulates permission and prohibition of the use according to the combination .

According to this aspect, it is possible to provide a flexible image that is beneficial to both the image providing side and the user side using the image. In addition, a plurality of images with different image quality can be processed in real time. Furthermore, flexible copyright management becomes possible. For example, it is possible to provide image quality for which copyright management is performed and image quality for which copyright management is not performed.

  The decoding unit may decode the encoded image data with all decodable image quality. According to this aspect, it is possible to create a situation in which the most flexible image can be provided from the encoded image data.

The copyright management unit may add a noise component to at least one of the plurality of decrypted image data . According to this aspect, flexible copyright management is possible instead of copyright management that prohibits uniform use.

You may further provide the transmission part which transmits the image data selected among several decoded image data to a predetermined | prescribed terminal. The transmission unit may transmit image data having different image quality to each of the plurality of terminals substantially simultaneously. The transmission unit may transmit image data selected by a user operation, or may transmit image data selected based on spec information of a predetermined terminal. This spec information may include the resolution of the display unit of the terminal. According to this aspect, flexible image transmission becomes possible.

  Another aspect of the present invention is an image display device. This apparatus includes the image processing apparatus according to any one of the above-described aspects, and includes a display unit that displays a plurality of images having different image quality.

  According to this aspect, it is possible to perform flexible image display that is beneficial to both the image providing side and the user side using the image. In addition, a plurality of images with different image quality can be displayed in real time.

  The display unit may display a plurality of images with different image quality while individually managing the copyright for each image according to the set copyright management information. According to this aspect, flexible copyright management is possible.

  The display unit may display a plurality of images having different image quality in a multi-window so that the user can select an image having any image quality. According to this aspect, user operability can be improved.

  It should be noted that any combination of the above-described components, and those obtained by replacing the components and expressions of the present invention with each other among methods, apparatuses, systems, recording media, programs, etc. are also effective as an aspect of the present invention. .

  According to the present invention, it is possible to provide a flexible image that is beneficial to both the image providing side and the user side using the image.

  The present invention can use a technique for generating still images and moving images having different image quality from a single encoded image data stream. As an example of this, a method for encoding a moving image by the Motion-JPEG2000 system will be briefly described with reference to FIG. An image encoding device (not shown) continuously encodes each frame of a moving image in units of frames to generate an encoded image data stream of the moving image. At the start of the encoding process, an original image OI (Original Image) 102 corresponding to one frame of a moving image is read into the frame buffer. The original image OI read into the frame buffer is hierarchized by a wavelet transformer.

  The wavelet transformer in JPEG2000 uses a Daubechies filter. This filter acts as a high-pass filter and a low-pass filter simultaneously in the x and y directions of the image, and divides one image into four frequency subbands. These subbands include an LL subband having a low frequency component in both the x and y directions, an HL subband having a low frequency component in one of the x and y directions and a high frequency component in the other direction, and An LH subband and an HH subband having a high frequency component in both the x and y directions. This filter also has an action of reducing the number of pixels by half in both the x and y directions. That is, the number of vertical and horizontal pixels of each subband is ½ of the image before processing, and a subband image with a resolution, that is, an image size of ¼ can be obtained by one filtering. In this specification, an image that has undergone wavelet transform once with respect to the original image OI is referred to as a first layer image WI1, and hereinafter referred to as an nth layer image WIn depending on the number of times of undergoing wavelet transform. To.

  As schematically shown in FIG. 1, four subbands LL1, HL1, LH1, and HH1 are generated in the first layer image WI1 104. The first layer image WI1 104 is subjected to wavelet transform to generate a second layer image WI2 106. Here, the second and subsequent wavelet transforms are performed only on the LL subband component in the immediately preceding layer image. Accordingly, in the second layer image WI2 106, the LL1 subband of the first layer image WI1 is decomposed into four subbands LL2, HL2, LH2, and HH2. The wavelet transformer executes this filtering a predetermined number of times and outputs wavelet transform coefficients for each subband. Thereafter, the image encoding device performs quantization and other processes, and finally outputs encoded image data CI (Coded Image).

  For simplicity of explanation, in this example, it is assumed that the image encoding device performs wavelet transformation three times on the original image OI. Thus, for example, if the original image OI 102 is 1440 × 960 pixels, the size of the LL1 subband of the first layer image WI1 104 is 720 × 480, and the LL2 subband of the second layer image WI2 106 is 360 ×. 240, the size of the LL3 subband of the third layer image WI3 108 is 180 × 120.

  It should be noted about the hierarchized image that the low frequency component in the original image OI appears in the upper left in FIG. In the case of FIG. 1, the LL3 subband at the upper left corner of the third layer image WI3 has the lowest frequency. Conversely, if the LL3 subband can be obtained, the most basic property of the original image OI is obtained. Can be reproduced.

  In addition to Motion-JPEG2000, the encoded image data stream may be, for example, an SVC (Scalable Video Codec) having a high-quality HD stream and a low-quality SD stream in one stream, or Motion-JPEG. But you can. In JPEG, there is a method in which each frame is transmitted from a low-order term coefficient among DCT (Discrete Cosine Transform) coefficients obtained by discrete cosine transform, and depending on which order term coefficient is used for decoding. The image quality can be selected. According to these standards, spatial resolution can be encoded hierarchically.

  The above example is an example of hierarchization by frequency division coding. In this respect, hierarchization may be performed by approximation-enhanced coding. That is, it is possible to perform decoding with different image quality by adjusting the number of decoding bits from the upper bits of the DCT coefficient or wavelet coefficient. In MPEG2, in addition to the spatial resolution, temporal resolution by adjusting the number of frames can be hierarchically encoded.

  FIG. 2 shows an example of the data structure of the encoded image data stream. The header in FIG. 2 indicates a stream header. The sign side can add copyright level information to be described later to a free area of the header area. Each frame is also provided with a frame header (not shown), which describes information necessary for decoding. Note that the copyright level information may be added to the frame header. For example, it can be added to any frame in the frame group, such as every few frames.

  As described above, when hierarchical encoded image data CI is generated by frequency division, data is described in the order of low frequency components, medium frequency components, and high frequency components from the head of the data area of each frame. . In addition, when the system which divides | segments to a DC component is employ | adopted, the data of a DC component are described before a low frequency component. Even if only the first half of each frame is used, an image whose contents can be grasped can be reproduced, but a low-quality image is obtained. As the second half of the video is played back, the quality gradually becomes higher. As described above, the hierarchical encoded image data stream includes an image data area necessary for reproducing a high-quality image and an image data area necessary for reproducing a low-quality image in a form overlapping the area. Have both. Accordingly, it is possible to reproduce a plurality of images having different image quality from one encoded image data stream.

  FIG. 3 shows an example of copyright level information. The copyright level information is information for managing the copyright level granted to the user who uses the encoded image data CI. The use of the encoded image data CI is not an alternative management of whether to permit or prohibit the entire use, but the usage, image quality, combinations thereof, and the like are individually managed for each usage mode. In FIG. 3, copyright level information to be added to the encoded image data CI is represented by a table. The horizontal axis indicates the usage of the encoded image data CI, and the vertical axis indicates the licensed image quality level corresponding to each usage.

  Display that is one aspect of the application is allowed without limitation at low image quality. For medium and high image quality, noise such as scrambled display is added to prevent viewing. Therefore, display with medium and high image quality is substantially limited. Note that this noise is not superimposed when a predetermined condition such as payment by a user is satisfied. The initial display indicates a function of displaying an image that should be difficult to view due to noise superimposed for a certain period from the start of display as a normal screen to which noise is not added. In FIG. 3, initial display is performed for 1 minute for high image quality and for 30 seconds for medium image quality. After that, the image is superimposed with noise.

  Transmission indicates a function of transmitting from the device in which the encoded image data CI is decoded to another device. In FIG. 3, unlimited transmission is permitted for all image quality. Duplication is prohibited for high image quality, and unlimited for low and medium image quality. Editing is prohibited for medium and high image quality, and unlimited for low image quality. This editing may include editing that cuts a commercial portion from encoded image data received by digital television broadcasting. In this case, the encoded image data can be duplicated with medium image quality, but the commercial portion cannot be cut and duplicated.

  The image providing side can manage such a tabulated profile for each user and appropriately update each item of the profile. When the encoded image data CI is provided to the user for a fee, different profiles can be assigned depending on the fee paid by the user, the course setting, and the like. If the profile is different, the usage that can be used and the licensed image quality level corresponding to the usage are different. The profile may be generated not for each user but for each region or each time. For example, when a profile for each time is generated, a time limit can be set for the use of the user's image content. If the user's device is configured to switch profiles after a certain period, it is not a process that disables the use of all content after that period. can do.

  Next, the image processing apparatus 200 that decodes the hierarchical encoded image data stream as described above will be described. FIG. 4 shows a configuration of the image processing apparatus 200 according to the embodiment of the present invention. The configuration of the image processing apparatus 200 can be realized in hardware by a CPU, memory, or other LSI of an arbitrary computer, and is realized in software by a program having a decoding function loaded in the memory. Here, the functional blocks realized by the cooperation are depicted. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The image processing apparatus 200 includes a decoding block 250. The decoding block 250 includes a stream analysis unit 252, an arithmetic decoding unit 254, a bit plane decoding unit 256, an inverse quantization unit 258, a wavelet inverse transform unit 260, and a frame buffer 262.

  The stream analysis unit 252 acquires the encoded image data stream CI from the outside, and analyzes the data stream. The encoded image data stream CI may be acquired by downloading from a network, may be acquired from a broadcast radio wave, or may be acquired from a recording medium on which it is recorded. If copyright level information is added to the header of the encoded image data CI or the like, it is read out and passed to the copyright management unit 30.

The arithmetic decoding unit 254 performs arithmetic decoding on a data string to be decoded that is found as a result of stream analysis. The bit plane decoding unit 256 decodes the data obtained as a result of arithmetic decoding in the form of a bit plane for each color component. The inverse quantization unit 258 performs inverse quantization on the decoded quantized data. The wavelet inverse transform unit 260 uses the frame buffer 262 to perform wavelet inverse transform on the _n- th layer image WI _n obtained as a result of inverse quantization. Each time wavelet inverse transform is performed on the encoded image data CI, an image of a higher layer is obtained, and when decoding is performed up to the highest layer, decoded image data DI (Decoded Image) can be obtained.

  The decoding block 250 outputs the image data expanded in the frame buffer 262 to the storage unit 34 for a plurality of hierarchies. For example, the expanded image data may be output for all layers from the first layer to the highest layer. In this case, if the image data in the frame buffer 262 is output to the storage unit 34 every time wavelet inverse transformation is performed, the image data of all layers can be output. Further, image data of two or more hierarchies from the first hierarchy to the highest hierarchy may be output to the storage unit 34 according to a predetermined setting. For example, it may be output every other layer from the first layer to the highest layer.

  The copyright management unit 30 manages the copyright of the encoded image data CI used in the image processing apparatus 200. It manages copyright level information passed from the stream analysis unit 252 or copyright level information CLI acquired from a server on the image providing side via a network such as the Internet. The copyright level information may include information that permits use that has been limited until then, depending on the payment of fees. For example, when display with high image quality is restricted, information for permitting the display may be included. When the user uses the encoded image data CI, the copyright management unit 30 specifies the permitted image quality according to the use, and manages a plurality of image data with different image quality stored in the storage unit 34.

  The copyright management unit 30 may include a noise superimposing unit 32. The noise superimposing unit 32 superimposes a predetermined noise component on the decoded image data in order to generate a display for preventing viewing such as a scramble display. A noise component may be superimposed during the inverse wavelet transform in the frame buffer 262 in the decoding block 250, or when the image data stored in the storage unit 34 is further output to the transmission unit 36 or the display unit 38. You may superimpose. Note that the noise superimposing unit 32 does not need to be provided when copyright display simply prohibits image display with a certain image quality. If the image processing apparatus 200 does not perform copyright management, the copyright management unit 30 need not be provided.

  The storage unit 34 stores the image data output from the decoding block 250. In the present embodiment, it is possible to store a plurality of pieces of image data having different image qualities and simultaneously provide images having the same contents having different image qualities to the transmission unit 36 and the display unit 38.

  FIG. 5 shows a data structure of the storage unit 34 in the present embodiment. The storage unit 34 stores a plurality of pieces of image data with different image quality decoded from a single encoded image data CI. In FIG. 5, high resolution image data 322, medium resolution image data 324, and low resolution image data 326 are stored in independent forms. In the case of a moving image, the image data of each area is sequentially updated, but substantially the same scene image data is held.

  The transmission unit 36 transmits image data stored in the storage unit 34 to a terminal such as a TV, a PC, a mobile phone, or a PDA (Personal Digital Assistance) by wired or wireless communication. The display unit 38 displays the image data stored in the storage unit 34. The image processing apparatus 200 only needs to include at least one of the transmission unit 36 and the display unit 38.

  The selection unit 40 selects image data to be output from a plurality of image data in the storage unit 34 when outputting image data from the storage unit 34 to the transmission unit 36 or the display unit 38. The selection unit 40 may be selected in response to a user instruction from an operation unit (not shown), or may be selected based on predetermined spec information. The specification information may include the resolution of the display unit 38, the display resolution of the transmission destination, the calculation speed of the decoding block 250, the memory capacity, and the like. For example, the selection unit 40 may acquire the resolution of the display of the transmission destination terminal in advance and automatically select image data having the highest resolution within the resolution.

  Next, an operation example of the image processing apparatus 200 in this embodiment will be described. Since the image processing apparatus 200 holds a plurality of pieces of image data with different image quality in the storage unit 34, various applications as described below can be realized.

  First, the image processing apparatus 200 can display images of the same content with different image quality substantially simultaneously. FIG. 6 shows a screen on which images of the same content with different image quality are displayed in a multi-window. In FIG. 6, three high-resolution images 342 generated from the high-resolution image data 322, the medium-resolution image data 324, and the low-resolution image data 326, the medium-resolution image 344, and the low-resolution image 346 are superimposed and displayed on one screen. Has been. The user can select to display one of a plurality of images with different image quality displayed on the display unit 38. The selection unit 40 controls to output only the image data for displaying the image selected by the user from the storage unit 34 to the display unit 38.

  A size changing unit (not shown) may enlarge or reduce the selected image data in accordance with a user instruction. However, enlargement of the originally low-resolution image data does not improve the image quality only by generating block noise. Note that the image selection screen as shown in FIG. 6 may be changed at any time by a user operation.

  In such a configuration, it is possible to perform processing such as switching to a high-resolution image so that a user who was viewing a medium-resolution image can view it on a more powerful screen. It can also be used for applications that collect different fees depending on the image quality to be displayed. If the charge for display per unit time is set differently depending on the image quality, for example, if the display with low image quality is set to a low charge or free, and the display with high image quality is set to a relatively high price, the user It is possible to take a viewing method in which a low quality image is selected in a scene to be followed and a high quality image is selected in the climax. In order to realize this, a time monitoring unit may be provided. A time monitoring unit (not shown) monitors the selection unit 40 to record how long the user has viewed an image of which image quality as viewing time information. Then, if the viewing time information is transmitted to the server on the image providing side after the end of viewing, the image providing side can calculate the fee to be charged to the user based on the information.

  In addition, when displaying the same content image with different image quality substantially simultaneously, the image processing apparatus 200 can superimpose noise for preventing viewing from one or more images. FIG. 7 shows a screen on which images of the same content with different image quality, including an image on which noise is superimposed, are displayed in multiple windows. In FIG. 7, the high resolution image 342 generated from the high resolution image data 322 is scrambled. As described above, the copyright management unit 30 can superimpose noise on an image of an image quality that the image provider performs copyright management according to the acquired copyright level information. Also, it is possible to perform control such as releasing the scramble on the screen for a limited time. For example, it is possible to perform control such that an image having an image quality to be managed is displayed as usual for several tens of seconds from the start of display, and then an image on which scramble is superimposed is displayed. When the user wants to remove the noise from the image on which the noise is superimposed and views the image, the user can remove the noise by selecting the image and performing a predetermined procedure such as payment of a fee.

  In addition, the image processing apparatus 200 can transmit images having the same content to a plurality of terminals substantially simultaneously. With such a usage method, the image processing apparatus 200 can function as a home server. At that time, images of different image quality can be transmitted for each terminal. In addition, an image on which noise is superimposed can be transmitted to one or more of a plurality of terminals. It is possible to designate to which terminal an image of which image quality is to be transmitted by a user operation. Further, the selection unit 40 may automatically transmit an image with an optimum image quality based on the specification information of the transmission destination terminal. For example, low-resolution image data may be automatically transmitted to a terminal having a narrow display area such as a portable terminal. The user can easily transmit the image data from the image processing apparatus 200 to the portable terminal, and can view the image even when going out. For example, the continuation of the content that was viewed on the home TV can be viewed outside using a portable terminal.

  As described above, according to the present embodiment, it is possible to realize flexible image provision that is beneficial to both the image providing side and the user side using the image. That is, since image data having the same content with different image quality is stored in the storage unit, various processes can be realized using the image data in real time. For example, a plurality of images with different image quality can be displayed in a multi-window, or images with different image quality can be transmitted to different terminals substantially simultaneously. The image providing side can use various sales techniques and sales promotion techniques using such functions.

  In addition, by performing copyright management for each image quality and usage, it is not an alternative choice of whether the use of the image is permitted or prohibited, but a request for copyright protection on the image provider side and the user It is possible to flexibly balance the usage request on the side.

  The present invention has been described based on some embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  In the embodiment, the frequency division type hierarchization has been described. In this regard, hierarchization with improved approximation accuracy may be used. In that case, images of various image quality levels can be decoded by discarding a predetermined number of lower bits in the bit string of wavelet transform coefficients. That is, the image quality level can be adjusted by designating the number of bits that the copyright management unit 30 converts to the bit plane decoding unit 256.

  When this hierarchization method is used, a difference in image quality can be provided for each image part. In that case, each profile of the table managing the copyright level information shown in FIG. 3 is not only described with two parameters of usage and image quality level, but also with three parameters of usage, image quality, and image portion. Can be described. It can also be described by two parameters of image quality and image portion. In this case, the image providing side can manage the copyright more flexibly than the above-described embodiment.

  Different image quality can be realized not only by resolution but also by layering the compression rate and the number of colors. The compression rate can be hierarchized by using the JPEG2000 layer function. A color image is represented by a luminance component and a color difference component, but the above-described hierarchization method can also be used for the color difference component.

It is a figure which shows the procedure of an image encoding process. It is a figure which shows an example of the data structure of an encoding image data stream. It is a figure which shows an example of copyright level information. It is a figure which shows the structure of the image processing apparatus in embodiment of this invention. It is a figure which shows the data structure of the memory | storage part in this embodiment. It is a figure which shows the screen which displayed the image of the same content from which image quality differs with a multiwindow. It is a figure which shows the screen which displayed the image of the same content from which the image quality differs including the image on which the noise was superimposed by the multi-window.

Explanation of symbols

  30 copyright management unit, 32 noise superimposing unit, 34 storage unit, 36 transmission unit, 38 display unit, 40 selection unit, 200 image processing device, 250 decoding block, 252 stream analysis unit, 254 arithmetic decoding unit, 256 bit plane decoding 258 inverse quantization unit 260 wavelet inverse transform unit 262 frame buffer.

Claims (9)

A decoding unit that obtains encoded image data multiplexed in a decodable state with a plurality of image quality and decodes the encoded image data so that at least two images having different image quality are generated ;
A copyright management unit that performs copyright management on a plurality of decrypted image data based on copyright management information that regulates permission and prohibition of use according to the combination of image data quality and usage An image processing apparatus. A decoding unit that obtains encoded image data multiplexed in a decodable state with a plurality of image quality and decodes the encoded image data with a plurality of decodable image quality ;
A copyright management unit that performs copyright management on a plurality of decrypted image data based on copyright management information that regulates permission and prohibition of use according to the combination of image data quality and usage An image processing apparatus.   The image processing apparatus according to claim 1, wherein the decoding unit decodes the encoded image data with all decodable image quality. The image processing apparatus according to claim 3 , wherein the copyright management unit adds a noise component to at least one of the plurality of decrypted image data . Of the decoded plurality of image data, the image processing apparatus according to claim 1, further comprising a transmitter for transmitting image data selected in a predetermined terminal 4. The image processing apparatus according to claim 5 , wherein the transmission unit transmits image data having different image quality to each of a plurality of terminals in parallel. The image processing apparatus according to any one of claims 1 to 6 , comprising:
An image display device comprising a display unit for displaying a plurality of images each having different image quality. The image display device according to claim 7 , wherein the display unit displays a plurality of images having different image quality while individually managing the copyright for each image according to the set copyright management information. . Wherein the display unit, in order to select an image of any quality to the user, the image display apparatus according to claim 7 or 8, wherein the displaying the plurality of images having different image quality with multi-window.

Images