JP4697249B2 - Information processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an information processing apparatus and method, and a program, and more particularly, to an information processing apparatus and method, and a program that can more easily edit a code stream that is encoded in a scalable manner.

  In recent years, in the production, distribution, and screening of movies, the spread of digital cinema using digital data instead of film has been promoted, and standardization such as DCI (Digital Cinema Initiative) composed of seven major US movie distribution companies The specification of the standard is being developed by the organization.

  For example, in the DCI standard established by DCI, two image resolutions of 4K (4,096 × 2,160 pixels) and 2K (2,048 × 1,080 pixels) are defined, and video compression can be decoded at a plurality of resolutions. JPEG2000 (Joint Photographic Experts Group 2000) for encoding (scalably) in this way is determined to be used (see, for example, Patent Document 1).

  In digital cinema production workflows, images scanned from film are captured in image formats such as Tiff (Tagged Image File Format) and Dpx (Digital Picture Exchange), and uncompressed (baseband) digital data (baseband data ). This baseband data is managed as master data and subjected to editing processing such as noise removal.

  FIG. 1 is a diagram illustrating an example of a state of editing processing.

  As shown in FIG. 1, the frame image 10 obtained by scanning a motion picture film may contain a noise component due to, for example, dust or dirt attached to the scanned film, or scratches on the film. FIG. 11 is an enlarged view of the range 10A including the noise component of the frame image 10. As shown in the enlarged view 11, in the example of FIG. 1, noise 12 due to dust attached to the film is generated in the frame image 10. In a digital cinema production workflow, such noise component removal processing (editing processing) is performed.

  However, if the resolution is very high, such as 4K, the amount of data becomes very large. Therefore, if the baseband data is master data, the storage capacity required for storage, the network bandwidth required for transfer, and various Processing load, processing time, etc. will increase.

  Therefore, in order to reduce the amount of data, a method of using compressed data (code stream) compressed by a predetermined compression method such as JPEG 2000 as master data can be considered. As described above, JPEG2000 is one of the scalable encoding schemes, and therefore supports a plurality of resolutions such as 4K and 2K.

JP 2006-311327 A

  However, also in this case, editing processing such as noise removal is performed after the master data is decoded at the highest resolution. For example, when the resolution of each frame image is 4K, master data that is digital data (JPEG2000 data) compressed by the JPEG2000 system is decoded at a resolution of 4K, each frame image at a resolution of 4K is restored, noise removal, etc. The editing process has been performed on each frame image of the 4K resolution baseband.

  As described above, the scalable-coded data is subjected to the editing process after being decoded at the highest resolution as in the case of the data encoded by the non-scalable encoding method. Therefore, similarly to the case of the data encoded by the non-scalable encoding method, the data of the scalable encoding increases the data amount due to the increase in the resolution, so that the load and the processing time of the decoding process greatly increase. As a result, the load and processing time of the editing process may increase significantly.

  The present invention is for solving such a problem, and specifies an edit part using low-resolution baseband data, and edits the specified edit part, thereby enabling scalable encoding. The code stream editing process can be performed more easily.

One aspect of the present invention is an information processing apparatus that processes a code stream of the first resolution, in which data of a first resolution is encoded in a scalable manner, and the code stream of the first resolution is Decoding at a second resolution lower than the first resolution, setting means for setting an editing area for editing using the obtained baseband data of the second resolution, and the first resolution creating a code stream, a baseband data portion of said first of said first resolution decoded by resolution of the editing area set by the setting means, the partial baseband data of the first resolution first and creation means, an editing means for performing editing processing with respect to the first of said first created by creating means the resolution of the partial baseband data, said editing means Partial baseband data of the edited first resolution and scalably encoded and a second generating means for generating partial code stream is a code stream of the editing area, said first resolution codestream An information processing apparatus comprising: a code stream replacing unit that replaces a part of an editing area with the partial code stream created by the second creating unit .

  The first creation means includes a first extraction means for extracting a portion of the editing area from the baseband data of the second resolution, and a portion of the editing area from the code stream of the first resolution. A second extracting means for extracting; a decoding means for decoding a portion of the editing area of the code stream of the first resolution extracted by the second extracting means; and a second extracting means for extracting by the first extracting means Each of the first resolution obtained by decoding the portion of the editing area of the baseband data of the second resolution and the portion of the editing area of the code stream of the first resolution by the decoding means. The inverse wavelet transform that creates the partial baseband data of the first resolution by performing inverse wavelet transform on the portion of the edit region of the subband of the high frequency component. Can and means.

  The editing means can perform editing to remove a noise component included in the partial baseband data of the first resolution.

The second creation means is obtained by wavelet transform means for wavelet transforming the edited partial baseband data of the first resolution at the first resolution, and wavelet transform processing by the wavelet transform means. A first encoding unit that encodes a portion of the edit region of each high-frequency component subband having a resolution of 1, and a portion of the edit region of the baseband data of the second resolution obtained by the setting unit; Is replaced with the part of the editing area of the baseband data of the second resolution included in the subband of the low frequency component of the first resolution obtained by the wavelet transform processing by the wavelet transform means Substitution means and baseband data of the second resolution in which the portion of the editing area is replaced It may include a second coding means for coding.

  The second creation means is obtained by wavelet transform means for wavelet transforming the edited partial baseband data of the first resolution at the first resolution, and wavelet transform processing by the wavelet transform means. Baseband data of the second resolution obtained by the first encoding means for encoding the portion of the edit area of the subband of each high-frequency component of the resolution of 1 and the wavelet transform processing by the wavelet transform means And a second encoding means for encoding the part of the editing area.

One aspect of the present invention is also an information processing method for an information processing apparatus that processes a code stream having the first resolution, in which data having the first resolution is encoded in a scalable manner. Is decoded at a second resolution lower than the first resolution, and an editing area for editing is set using the obtained baseband data at the second resolution, 1 resolution codestream a baseband data of the set portion of the editing area decoded by the first resolution first resolution, to create a partial baseband data of the first resolution performs an editing process on the partial baseband data of the created first resolution, encoded in scalable partial baseband data edited the first resolution, before Create a partial code stream is a code stream of the editing area, a part of the editing area of the first resolution code stream, an information processing method comprising the step of replacing the partial code stream created.

According to another aspect of the present invention, in the program for processing the first resolution code stream in which the first resolution data is scalable and encoded, the first resolution code stream is converted into the first resolution code stream. A setting step of setting an editing area to be edited using the baseband data of the second resolution obtained by decoding at a second resolution lower than the resolution of the first resolution, and the code of the first resolution creating a stream, the setting is baseband data set portion of the editing area by the processing of the first said decoded by the resolution of the first resolution step, the partial baseband data of the first resolution first generating step and the editing process to the first of said first created by the processing steps for creating resolution partial baseband data An editing step of performing a second creation step of encoding the partial baseband data of the first resolution which has been edited by the processing of the editing step to a scalable, to create a partial code stream is a code stream of the editing area A code stream replacement step of causing a computer to execute a code stream replacement step of replacing a part of the editing area of the code stream of the first resolution with the partial code stream created by the processing of the second creation step .

Another aspect of the present invention is an information processing apparatus that processes a code stream of the first resolution, in which data of the first resolution is encoded in a scalable manner, and the code stream of the first resolution is Decoding means at a second resolution lower than the first resolution, and using the obtained baseband data at the second resolution, setting means for setting an edit area for editing; and the first included in the resolution of the code stream, from the code stream of the sub-band of each high-frequency component of the first resolution, and extracting means for extracting a portion of the editing area set by the setting unit, extracted by the extraction means In addition, the editing area portion of the sub-band code stream of each high-frequency component of the first resolution, and the editing of the baseband data of the second resolution And editing means for performing editing processing with respect to each portion of the band, and coding means for coding scalably baseband data of the portion of the editing area is edited second resolution by said editing means, said A portion of the second resolution code stream of the first resolution code stream and the second resolution baseband data obtained by editing the portion of the editing area are encoded in a scalable manner by the encoding means. The second resolution code stream is replaced, and the edit area portion of the sub-band code stream of each high frequency component of the first resolution included in the first resolution code stream is further replaced. the knitting of the code stream of the sub-band of each high-frequency component of the first resolution the editing process is performed by said editing means An information processing apparatus and a replacement means for replacing the portion of the region.

  The editing means, for each of the editing area portion of the substream code stream of each high-frequency component of the first resolution, and each of the editing area portion of the baseband data of the second resolution, Editing that removes noise components can be performed.

  The editing means can remove the noise component by replacing a part of the editing area of the substream code stream of each high-frequency component of the first resolution with a predetermined other code stream.

Another aspect of the present invention is also an information processing method of an information processing apparatus for processing a code stream of the first resolution, in which data of the first resolution is encoded in a scalable manner. Decoding the resolution code stream at a second resolution lower than the first resolution, and using the obtained baseband data at the second resolution to set an editing area to be edited, A portion of the set editing area is extracted from the substream code stream of each high frequency component of the first resolution included in the code stream of the first resolution, and each of the extracted first resolutions is extracted. Edit processing is performed on each of the edit region portion of the high-frequency component subband code stream and the edit region portion of the baseband data of the second resolution. The baseband data of the second resolution in which the part of the editing area is edited is scalablely encoded, and the part of the code stream of the second resolution of the code stream of the first resolution is the editing area. The baseband data of the second resolution, which is edited from the second resolution, is replaced with the codestream of the second resolution that is encoded in a scalable manner, and further included in the codestream of the first resolution. The part of the edit area of the subband code stream of each high frequency component of 1 resolution is the part of the edit area of the code stream of the subband of each high frequency component of the first resolution subjected to the editing process. It is the information processing method including the step to replace with.

According to another aspect of the present invention, in the program for processing the code stream of the first resolution in which the data of the first resolution is encoded in a scalable manner, the code stream of the first resolution is Decoding at a second resolution lower than the first resolution, and using the obtained baseband data at the second resolution, a setting step for setting an editing area to be edited; An extraction step for extracting a portion of the editing area set by the processing of the setting step from a substream code stream of each high-frequency component of the first resolution included in the code stream; and by the processing of the extraction step The portion of the editing area of the extracted sub-band codestream of each high-frequency component of the first resolution, and the second solution Scalable and editing step, the baseband data processed by the editing area the second resolution part is edited in the editing step for editing processing on the respective part of the editing area of the baseband data degrees The second resolution base stream data obtained by editing the portion of the editing area, the encoding step of encoding the first resolution code stream, and the second resolution code stream portion of the first resolution code stream. Substitution of the second resolution code stream encoded in a scalable manner by the process of the encoding step , and further, sub-frequency of each high-frequency component of the first resolution included in the first resolution code stream. band code stream, a portion of the editing area, the editing process is performed by processing in the editing step And a first resolution program for executing the substitution step to the computer to replace the portion of the editing area of the code stream of the sub-band of the radio frequency components.

In one aspect of the present invention, a code stream having a first resolution is decoded at a second resolution lower than the first resolution, and editing is performed using the obtained baseband data having the second resolution. set editing area for, the first resolution code stream, a baseband data of the first resolution is part of the set editing area decoded by the first resolution, the portion of the first resolution Baseband data is created, editing processing is performed on the created partial baseband data of the first resolution, the edited partial baseband data of the first resolution is scalable and encoded, A partial code stream that is a code stream is created, and the portion of the editing area of the first resolution code stream is replaced with the created partial code stream.

In another aspect of the present invention, a first resolution code stream is decoded at a second resolution lower than the first resolution, and the obtained second resolution baseband data is used. An editing area to be edited is set, and a portion of the set editing area is extracted and extracted from the substream code stream of each high frequency component of the first resolution included in the code stream of the first resolution. In addition, an editing process is performed on each of the editing region portion of the subband code stream of each high-frequency component of the first resolution and the editing region portion of the baseband data of the second resolution, The baseband data of the second resolution whose portion is edited is encoded in a scalable manner, and the codestream of the second resolution of the codestream of the first resolution is encoded. The second resolution baseband data in which the portion of the editing area is edited is replaced with the second resolution code stream encoded in a scalable manner, and further included in the first resolution code stream. The edit region portion of the substream code stream of each high frequency component of the first resolution is replaced with the edit region portion of the substream code stream of each high frequency component of the first resolution subjected to the editing process. The

  The network is a mechanism in which at least two devices are connected and information can be transmitted from one device to another device. The devices that communicate via the network may be independent devices, or may be internal blocks that constitute one device.

  The communication is not only wireless communication and wired communication, but also communication in which wireless communication and wired communication are mixed, that is, wireless communication is performed in a certain section and wired communication is performed in another section. May be. Further, communication from one device to another device may be performed by wired communication, and communication from another device to one device may be performed by wireless communication.

  According to the present invention, information can be processed. In particular, the editing processing of a code stream encoded in a scalable manner can be performed more easily.

  FIG. 2 is a block diagram showing a main configuration example of an image processing apparatus to which the present invention is applied. An image processing apparatus 100 shown in FIG. 2 is an apparatus that removes noise from each frame image of image data.

  The image data to be processed is moving image data in which each image with a resolution of 4K (4,096 × 2,160 pixels) obtained by scanning each frame of a movie film in a digital cinema production workflow is a frame image. It is compressed in a scalable manner. In other words, this image data (code stream) is compressed in a scalable manner so that a baseband image can be obtained with a resolution of at least 4K or 2K (2,048 × 1,080 pixels) by decoding (select resolution). be able to). In the JPEG2000 system, wavelet transform is adopted, and the frequency component of the image is recursively decomposed into a high frequency component and a low frequency component, so that at the time of decoding, a baseband image is obtained at an arbitrary resolution from a plurality of stages. be able to.

  The image processing apparatus 100 uses, for this scalable encoded code stream, from the image of each frame, as described with reference to FIG. 1, due to dust attached to the film during scanning, film scratches, or the like. Remove noise components. At this time, the image processing apparatus 100 performs each process such as a decoding process, a noise removing process, and a re-encoding process by performing removal of the noise component on a frame image (low-resolution image) decoded at a resolution of 2K. Reduces the processing load.

  The image processing apparatus 100 includes a 2K scalable decoding unit 101, a noise removal unit 102, a 2K re-encoding unit 103, and a 2K code stream replacement unit 104 as main components.

  The 2K scalable decoding unit 101 decodes the code stream input to the image processing apparatus 100, and obtains a baseband frame image with a resolution of 2K. In the JPEG2000 system, as described above, low-frequency components of image data are recursively separated by wavelet transform, and are encoded for each frequency band. Accordingly, the code stream 150A for one frame image input to the 2K scalable decoding unit 101 is obtained by performing wavelet transform on an image with a resolution of 4K once each in the vertical direction and the horizontal direction, as shown in FIG. 3A. 2KS151A, which is a partial code stream in which a low-frequency component subband (2K (4KLL)) is encoded in the horizontal direction and the vertical direction, a high-frequency component in the horizontal direction, and a low-frequency component subband in the vertical direction 4KSHL152 which is a partial code stream in which (4KHL) is encoded, 4KSLH153 which is a partial code stream in which a low-frequency component in the horizontal direction and a subband (4KLH) in the high-frequency component are encoded in the vertical direction, and A partial code stream in which high-frequency component subbands (4KHH) are encoded in the horizontal and vertical directions. It consists of a certain 4KSHH154.

  In the wavelet transform, the image data is separated into a high frequency component and a low frequency component and then down-sampled by half. That is, a baseband (2K) image obtained by decoding 2KS151A is an image with a resolution of 2K. The 2K scalable decoding unit 101 extracts 2KS 151A from the code stream 150A, decodes the 2KS 151A, and performs inverse wavelet transform to obtain baseband data of an image with a resolution of 2K.

  In a normal case, the wavelet transform is recursively repeated a plurality of times, so that the wavelet coefficients corresponding to 2KS151A are further separated into a plurality of subbands and encoded. That is, the 2K scalable decoding unit 101 extracts and decodes all of these subband code streams, and performs inverse wavelet transform.

  The noise removing unit 102 removes noise from the baseband frame image having a resolution of 2K obtained by the 2K scalable decoding unit 101. The noise removing unit 102 is supplied with 2 KBB 151B, which is baseband data of a 2K resolution frame image, as shown in FIG. 3B, from the 2K scalable decoding unit 101. The noise removal unit 102 detects a noise component from the 2KBB 151B, for example, by edge detection or comparison with images at the same position of the preceding and following frame images, and sets an editing region for performing removal processing so as to include the noise component. . Further, the noise removing unit 102 removes noise by performing, for example, an editing process such as a filtering process or replacement of an image at the same position in the previous and subsequent frame images with another image on the editing area.

  When the noise removal editing process ends, the noise removal unit 102 supplies the 2K re-encoding unit 103 with the baseband data of the 2K resolution frame image from which the noise has been removed. That is, as shown in FIG. 3C, the partial image 155A set in the editing area in FIG. 3B is edited (noise removal), and the image 2KBB 151C that is the partial image 155B is transferred from the noise removal unit 102 to the 2K re-encoding unit 103. Supplied.

  The 2K re-encoding unit 103 re-encodes 2 KBB 151C, which is baseband data having a resolution of 2K, from which noise has been removed by the noise removing unit 102, using the JPEG2000 method. The 2K code stream replacement unit 104 replaces the 2K resolution portion of the code stream input to the image processing apparatus 100 with the code stream obtained by the 2K re-encoding unit 103, and the code from which noise has been removed at the 2K resolution. A stream is generated and output to the outside of the image processing apparatus 100.

  That is, as shown in FIG. 3D, the 2K code stream replacement unit 104 converts the 2KS 151A of the code stream 150A shown in FIG. 3A into an image with a resolution of 2K from which noise has been removed, supplied from the 2K re-encoding unit 103. A code stream 150B from which noise has been removed at a resolution of 2K is generated and output by replacing with 2KS151D which is a code stream obtained by encoding with the JPEG2000 system.

  An example of the flow of image processing for removing such noise components from the frame image will be described with reference to the flowchart of FIG.

  When image processing is started, in step S101, the 2K scalable decoding unit 101 extracts a 2K resolution code stream from the input code stream and decodes it. In step S102, the noise removing unit 102 performs an editing process for removing noise components with 2K resolution baseband data. In step S103, the 2K re-encoding unit 103 re-encodes 2K resolution baseband data from which noise components have been removed, in a scalable manner using the JPEG2000 method, and generates 2KS 151D encoded in the same manner as 2KS 151A. In step S104, the 2K code stream replacement unit 104 replaces the 2K resolution code stream of the code stream input to the image processing apparatus 100 with one from which the noise component has been removed, and the code from which the noise component has been removed at the resolution 2K. The stream 150B is output and the image processing ends.

  This image processing is performed for each frame image of the image data input to the image processing apparatus 100.

  As described above, the image processing apparatus 100 performs editing processing for removing noise components at a resolution 2K lower than the original resolution 4K of the frame image. As a result, the image processing apparatus 100 decodes the code stream with a resolution of 4K, removes the noise component, and re-encodes the code stream, removes the noise component, and performs the base associated with the editing process. It is possible to reduce the load of each process such as re-encoding of band data, the processing time, and the memory capacity necessary for data retention. That is, the image processing apparatus 100 can perform editing processing more easily than when editing processing is performed at a resolution of 4K.

  In the above description, the noise component is removed at the resolution 2K. However, the noise component included in the resolution 4K may also be removed.

  FIG. 5 is a block diagram showing another configuration example of the image processing apparatus to which the present invention is applied. An image processing apparatus 200 shown in FIG. 5 is an apparatus that removes noise from each frame image of image data, similarly to the image processing apparatus 100 of FIG. However, noise components (images such as film dust and scratches) included in a frame image created at a resolution of 4K are generally subbands of each high-frequency component when the frame image is wavelet transformed at a resolution of 4K. Is also included. Therefore, the image processing apparatus 200 further removes noise components included in the subbands of the respective high frequency components having a resolution of 4K. At this time, the image processing apparatus 200 performs noise detection on the image of resolution 2K to set an editing area, restores a baseband image corresponding to resolution 4K only in the editing area, and removes noise components. .

  As illustrated in FIG. 5, the image processing apparatus 200 includes an editing area setting unit 201, a partial baseband data creation unit 202, a noise removal unit 203, a partial code stream creation unit 204, and a code stream replacement unit 205.

  As in the case of the image processing apparatus 100, the editing area setting unit 201 generates baseband data with a resolution of 2K, detects a noise component in the baseband data, and edits the editing area so as to include the detected noise component. Set up. The editing area setting unit 201 includes a 2K scalable decoding unit 211 and an editing area specifying unit 212.

  As shown in FIG. 6A, a code stream 250A input to the image processing apparatus 200 is obtained by encoding a 4K resolution frame image in the JPEG2000 system, and the 4K resolution frame image in the vertical and horizontal directions. 2KS251A, which is a partial code stream in which a subband (2K (4KLL)) of a low frequency component is encoded in the horizontal direction and the vertical direction, each obtained by performing wavelet transform once, is a high frequency component in the horizontal direction, and 4KSHL252A, which is a partial code stream in which a low-frequency component subband (4KHL) is encoded in the vertical direction, a low-frequency component in the horizontal direction, and a high-frequency component subband (4KLH) in the vertical direction is encoded Partial code stream 4KSLH253A and high-frequency component subbands (4KHH) in the horizontal and vertical directions Constituted by 4KSHH254A is encoded partial code stream.

  In general, since wavelet transform is recursively repeated, 2KS 251A is further separated into a plurality of subbands and encoded.

  Similar to the 2K scalable decoding unit 101 in FIG. 2, the 2K scalable decoding unit 211 extracts and decodes 2KS 251A from the code stream 250A input to the image processing apparatus 200, performs inverse wavelet transform, and performs the process shown in FIG. 6B. As described above, 2 KBB 251 B (that is, a low-frequency component subband obtained by wavelet transform with a resolution of 4K) that is baseband data of a frame image with a resolution of 2K is generated.

  The edit area specifying unit 212 detects a noise component in the baseband data of the frame image with the resolution of 2K, for example, by a predetermined process such as edge detection or comparison with the previous and next frame images, and the detected noise component Set the edit area to include FIG. 6B shows an example in which a partial image 255A of 2 KBB 251B is set in the editing area.

  The partial baseband data creation unit 202 creates baseband data (partial baseband data) of an image in the editing area set by the editing area setting unit 201 in a frame image with a resolution of 4K. The partial baseband data creation unit 202 includes a 2K editing region extraction unit 221, a 4K editing region extraction unit 222, a decoding unit 223, and an IDWT (Inverse Discrete Wavelet Transform) unit 224.

  Based on the coordinate information indicating the editing area acquired from the editing area specifying unit 212, the 2K editing area extracting unit 221 uses the editing area based on the baseband data of the 2K resolution frame image generated by the 2K scalable decoding unit 211. Extract the data.

  In the following, partial baseband data extracted from baseband data of a frame image with a resolution of 2K (a subband of a low-frequency component obtained by wavelet transform with a resolution of 4K) is referred to as partial baseband data with a resolution of 2K. A code stream of partial baseband data is referred to as a partial code stream having a resolution of 2K.

  Based on the coordinate information indicating the editing area acquired from the editing area specifying unit 212, the 4K editing area extracting unit 222 uses each subband of the high-frequency component with resolution 4K included in the code stream input to the image processing device 200. Edit area data is extracted from the code stream. At this time, the 4K editing area extraction unit 222 performs data extraction in units of code blocks that are units of encoding and decoding. That is, the 4K editing area extraction unit 222 extracts code block data including the editing area. Since the length of the code string of each code block is described in the packet header, the 4K editing area extraction unit 222 can access the code string in code block units by analyzing the packet header.

  At this time, the 4K editing area extraction unit 222 may take into account the overlap of the image boundary of the wavelet transform, and may expand the extraction range from the editing area, for example, as shown in FIG. In the case of the example in FIG. 7, the 4K editing area extracting unit 222 includes the code block 272-1 to code block 272-8 including the editing area 271 and the surrounding code blocks 273-1 to 273-12. A code block adjacent in the direction corresponding to the subband (that is, a code block that affects any of the code blocks 272-1 to 272-8 in the wavelet transform in the subband) is also extracted as appropriate. . In this case, the 2K editing area extraction unit 221 also expands the extraction range according to the 4K editing area extraction unit 222. Of course, this extraction range can be expanded to an arbitrary size, and the 4K editing area extraction unit 222 may be expanded to a range of two or more surrounding code blocks.

  Hereinafter, the baseband data of a partial image of a frame image with a resolution of 4K is referred to as partial baseband data with a resolution of 4K, and the code stream of the partial baseband data is referred to as a partial codestream of a resolution of 4K.

  In the following, partial baseband data extracted from each subband of a high-frequency component obtained by wavelet transform of a frame image with a resolution of 4K is referred to as partial baseband data of a subband with a resolution of 4K. The code stream is referred to as a subband partial code stream having a resolution of 4K.

  The decoding unit 223 decodes the 4K resolution subband partial code streams extracted by the 4K editing region extraction unit 222, respectively.

  That is, as illustrated in FIG. 6C, the 2K editing area extraction unit 221 extracts the data (wavelet coefficient) of the partial image 255A from 2KBB 251B. 6C, the 4K editing area extraction unit 222 extracts a partial code stream spatially corresponding to the partial image 255A from the 4KSHL 252A in units of code blocks, and the decoding unit 223 extracts the code stream. The partial wavelet coefficient 256A is obtained by decoding. Similarly, as shown in FIG. 6C, the partial code stream spatially corresponding to the partial image 255A is extracted from the 4KSLH 253A by the 4K editing region extraction unit 222 and the decoding unit 223, and is decoded and decoded. 257A is obtained, and a partial code stream spatially corresponding to the partial image 255A is extracted from the 4KSHH 254A in units of code blocks and decoded to obtain a partial wavelet coefficient 258A.

  The IDWT unit 224 performs inverse wavelet transform on the partial image 255A and the partial wavelet coefficient 256A to the partial wavelet coefficient 258A (partial wavelet coefficient 255A to partial wavelet coefficient 258A), and partial baseband data 261A having a resolution of 4K in the editing area. (FIG. 6D) is created.

  The noise removing unit 203 performs an editing process for removing a noise component from the partial baseband data 261A by a predetermined method in the same manner as the noise removing unit 102 in FIG. 2, and the partial baseband data 261B ( FIG. 6E) is obtained.

  The partial code stream creation unit 204 encodes the partial baseband data with a resolution of 4K from which the noise component has been removed by the JPEG2000 system, and creates a partial code stream with a resolution of 4K. The partial code stream creation unit 204 includes a 4KDWT unit 231, a 4K re-encoding unit 232, a 2K baseband replacement unit 233, and a 2K re-encoding unit 234.

  A 4KDWT (4K Discrete Wavelet Transform) unit 231 performs wavelet transform on the partial baseband data 261B having a resolution of 4K, from which the noise component generated by the noise removing unit 203 has been removed, once in the vertical and horizontal directions. The 4K re-encoding unit 232 re-encodes each of the high-frequency component subbands obtained by the wavelet transform. That is, the 4K re-encoding unit 232 generates the partial code stream 256B, the partial code stream 257B, and the partial code stream 258B of the 4K resolution subband illustrated in FIG. 6F.

  The 4K re-encoding unit 232 divides the code block at the time of encoding at the same position as the code stream of the original 4K frame image. In addition, the code block decoded for the overlap of the wavelet transform is not included in the editing area, and the data does not change before and after the noise removal, so that it is not necessary to encode again. When rate control is necessary for lossy encoding, the 4K re-encoding unit 232 sets the number of coding passes included in the code block to be the same as the code block of the code stream input to the image processing apparatus 200. The number of coding passes included in the code block of the code stream input to the image processing apparatus 200 can be obtained from the packet header of the code stream.

  The 2K baseband replacement unit 233 uses the vertical and horizontal data obtained in the 4KDWT unit 231 as the data of the partial image 255A in the editing area of the baseband data 251B (FIG. 6B) with the resolution of 2K generated by the 2K scalable decoding unit 211. Replace with wavelet coefficients 255B (FIG. 6F) of low frequency components in the direction. The 2K re-encoding unit 234 re-encodes the baseband data of the 2K resolution frame image using the JPEG2000 method (recursively so as to have the same configuration as the code stream input to the image processing apparatus 200). Wavelet transform is performed, and the obtained coefficients are encoded by a predetermined method), and a code stream 251C (FIG. 6F) of a 2K resolution frame image from which noise components are removed is generated. At this time, if rate control is necessary for irreversible encoding, the 2K re-encoding unit 234 performs the number of coding passes in the packet header of the code stream input to the image processing apparatus 200 as in the case of the 4K re-encoding unit 232. Refer to and adjust the values.

  The code stream replacement unit 205 replaces the code stream 251A of the 2K resolution frame image of the code stream 250A input to the image processing apparatus 200 with the code stream 251C generated by the 2K re-encoding unit 234. Further, the code stream replacement unit 205 replaces the partial code stream in the edit area of 4KSHL 252A of the code stream 250A with the partial code stream 256B. Further, the code stream replacement unit 205 replaces the partial code stream in the edit area of 4KSLH 253A of the code stream 250A with the partial code stream 257B. Further, the code stream replacement unit 205 replaces the partial code stream in the edit area of 4KSHH 254A of the code stream 250A with the partial code stream 258B. That is, the code stream replacement unit 205 generates a code stream 250B from which noise components are removed as illustrated in FIG. 6G and outputs the code stream 250B to the outside of the image processing apparatus 200.

  At this time, in the replaced portion of the code stream, the code length of each code block and the number of zero bit planes may change. That is, the information may change before and after the replacement. In that case, the code stream replacement unit 205 also rewrites the information of the code length and the number of zero bit planes in the packet header of the replaced part.

  Here, the number of zero bit planes represents the number of bit planes in which all coefficient bit values are 0 on the most significant bit (MSB) side. The wavelet coefficient obtained by the wavelet transform is encoded for each code block. At the time of encoding, each coefficient of the code block is divided for each bit, that is, for each position, and for each position. Grouped as a bit plane. That is, the bit plane is a set of bits (coefficient bits) at the same position of each coefficient of a certain code block. In this way, in the bit plane formed at each position of each coefficient, a bit plane in which the values of all coefficient bits higher than that position are 0 is specifically referred to as a zero bit plane. That is, the number of zero bit planes indicates the number of such zero bit planes continuing from the most significant bit of the coefficient.

  In the replaced code block, the value of each coefficient may change before and after the replacement, so that the position of the most significant bit having a value of 1 may change. That is, the number of zero bit planes of the code block may change due to the replacement. Since the information on the number of zero bit planes is described in the packet header of the code stream, it is necessary to update the information in the packet header of the code stream at least when the number of zero bit planes changes in this way. Of course, regardless of whether or not the number of zero bit planes has changed, the number of zero bit planes may be calculated for the replaced code block, and the information in the packet header may be updated.

  An example of the flow of image processing for removing such noise components from the frame image will be described with reference to the flowchart of FIG.

  When image processing is started, in step S201, the editing area setting unit 201 performs editing area setting processing and sets an editing area at 2K resolution. In step S202, the partial baseband data creation unit 202 performs partial baseband data creation processing to create baseband data in the editing area (baseband data of a code block including the editing area). In step S203, the noise removing unit 203 performs an editing process for removing noise components on the created partial baseband data. In step S204, the partial code stream creation unit 204 performs a partial code stream creation process, encodes the partial baseband data from which the noise component has been removed, using the JPEG2000 method, and creates a partial code stream.

  In step S205, the code stream replacement unit 205 identifies a portion corresponding to the partial code stream supplied from the 4K re-encoding unit 232 for each subband having a resolution of 4K of the code stream input to the image processing apparatus 200. The partial code stream (that is, the code stream including the noise component) is replaced with the partial code stream (that is, the code stream from which the noise component is removed) supplied from the 4K re-encoding unit 232. In step S <b> 206, the code stream replacement unit 205 converts the code stream of each subband with a resolution of 2K or less (that is, a code stream including a noise component) of the code stream input to the image processing apparatus 200 into the 2K re-encoding unit 234. It replaces with the code stream supplied more (that is, the code stream from which the noise component is removed). Further, in step S207, the code stream replacement unit 205 updates information regarding the code length and the number of zero bit planes in the header of the packet including the replaced code block.

  When the process of step S207 ends, the image processing ends. This image processing is performed for each frame image of the image data input to the image processing apparatus 200.

  Next, an example of the flow of the edit area setting process executed in step S201 in FIG. 8 will be described with reference to the flowchart in FIG.

  When the editing area setting process is started, in step S211, the 2K scalable decoding unit 211 decodes the 2K resolution code stream, performs inverse wavelet transform, and obtains baseband data of the 2K resolution frame image. In step S212, the edit area specifying unit 212 detects noise from the baseband data of the frame image having a resolution of 2K. In step S213, the edit area specifying unit 212 specifies an edit area based on the noise detection result, and the coordinate information is 2K edited. This is supplied to the area extraction unit 221 and the 4K editing area extraction unit 222. When the process of step S213 ends, the editing area setting process ends, the process returns to step S201 in FIG. 8, and the processes after step S202 are executed.

  Next, an example of the flow of partial baseband data creation processing executed in step S202 of FIG. 8 will be described with reference to the flowchart of FIG.

  When the partial baseband data creation processing is started, in step S221, the 2K editing area extraction unit 221 selects code block data (partial baseband with 2K resolution) including the editing area from the baseband data of the 2K resolution frame image. Data). In step S222, the 4K editing area extracting unit 222 extracts code block data (subband partial code stream of 4K resolution) including the editing area from the code stream of the 4K resolution frame image. In step S223, the decoding unit 223 decodes the subcode partial code stream of resolution 4K extracted in the process of step S222. In step S224, the IDWT unit 224 uses the partial baseband data of resolution 2K extracted by the processing of step S221 and the partial baseband data of subband of resolution 4K obtained by decoding by the processing of step S223. And inverse wavelet transform to generate partial baseband data with a resolution of 4K. When the process of step S224 ends, the partial baseband creation process ends, the process returns to step S202 of FIG. 8, and the processes after step S203 are executed.

  Next, an example of the flow of the partial code stream creation process executed in step S204 of FIG. 8 will be described with reference to the flowchart of FIG.

  When the partial code stream creation processing is started, in step S231, the 4KDWT unit 231 performs wavelet transform on the partial baseband data with a resolution of 4K from which noise components have been removed once in the vertical direction and in the horizontal direction. In step S232, the 4K re-encoding unit 232 encodes the partial baseband data of the 4K resolution subband generated by the process of step S231. In step S233, the 2K baseband replacement unit 233 obtains the partial baseband data in the editing area of the baseband data of the frame image having the resolution of 2K obtained in the 2K scalable decoding unit 211 by using the resolution obtained by the processing in step S231. Replacement with 2K partial baseband data (baseband data from which noise components have been removed). In step S234, the 2K re-encoding unit 234 encodes the baseband data of the frame image with the resolution of 2K that has been replaced in step S233. When the process of step S234 ends, the partial code stream creation process ends, the process returns to step S204 in FIG. 8, and the processes after step S205 are executed.

  As described above, the image processing apparatus 200 detects a noise component at a resolution 2K lower than the original resolution 4K, decodes only a portion including the noise component at a resolution 4K, and performs an editing process. As a result, the image processing apparatus 200 decodes the code stream of the entire frame image at a resolution of 4K, removes the noise component, and re-encodes the code stream associated with this editing process, compared with the case of re-encoding. It is possible to reduce the load of each process such as the removal and re-encoding of the baseband data, the processing time, the memory capacity necessary for data holding, and the like. That is, the image processing apparatus 200 can perform the editing process more easily than when the entire frame image is decoded at a resolution of 4K and the editing process is performed.

  In the above description, the subband (2K) having a resolution of 2K has been described as being replaced as a whole. However, the present invention is not limited to this. ) May only be replaced.

  FIG. 12 is a block diagram showing still another configuration example of the image processing apparatus to which the present invention is applied. In FIG. 12, blocks that perform the same processing as in FIG. An image processing apparatus 300 shown in FIG. 12 is an apparatus that removes noise from each frame image of image data, like the image processing apparatus 100 of FIG. 2, and is basically the same as the image processing apparatus 200 of FIG. Although having the configuration and performing the same processing, a partial code stream creating unit 304 and a code stream replacing unit 305 are provided instead of the partial code stream creating unit 204 and the code stream replacing unit 205 of the image processing apparatus 200.

  The partial code stream creation unit 204 in FIG. 5 generates a code stream of a frame image with a resolution of 2K from which noise has been removed, whereas the partial code stream creation unit 304 has a code block that includes an editing area with a resolution of 2K. Are generated and supplied to the code stream replacement unit 305.

  The partial code stream creation unit 304 includes a 2K re-encoding unit 334 in addition to the 4KDWT unit 231 and the 4K re-encoding unit 232. The 2K re-encoding unit 334 is similar to the case of the code stream input to the image processing apparatus 300, with the 4KDWT unit 231 performing partial waveband conversion on the partial baseband data having the resolution of 4K. The wavelet transform is further recursively performed so as to have a subband configuration, and the obtained coefficients are encoded by a predetermined method (encoded by the JPEG2000 method) to create a partial code stream having a resolution of 2K. That is, the partial code stream creation unit 304 supplies the code stream replacement unit 305 with a partial code stream of each subband having a resolution of 4K and a partial code stream having a resolution of 2K.

  The detailed configuration and processing of the 2K re-encoding unit 334 will be described later.

  The code stream replacement unit 305 supplies the part including the noise component of the code stream input to the image processing apparatus 300 to each subband of resolution 4K from which the noise component has been removed supplied from the partial code stream generation unit 304. The code stream is replaced with a partial code stream having a resolution of 2K, and is output to the outside of the image processing apparatus 300.

  As shown in FIG. 13A, the code stream 350A input to the image processing apparatus 300 includes 2KS351A that is a 2K resolution code stream, 4KSHL352A, 4KSLH353A, and 4KSHH354A that are 4K resolution subband codestreams. . Then, as shown in FIG. 13B, 2KBB 351B, which is baseband data of a 2K resolution frame image, is created by the 2K scalable decoding unit 211, and an editing region including a noise component is set by the editing region specifying unit 212 (FIG. 13B). 13B partial image 355A).

  As shown in FIG. 13C, the 2K editing area extraction unit 221 extracts a partial image 355A with a resolution of 2K, and the 4K editing area extraction unit 222 extracts each of the 4K resolutions corresponding spatially to the partial image 355A with a resolution of 2K. The subband partial code streams are extracted and decoded by the decoding unit 223, and the partial wavelet coefficient 356A, the partial wavelet coefficient 357A, and the partial wavelet coefficient 358A of each subband having a resolution of 4K are generated. The IDWT unit 224 performs inverse wavelet transform on the partial baseband data to generate partial baseband data 361A (FIG. 13D) with a resolution of 4K.

  The noise removing unit 203 removes a noise component from the partial baseband data 361A having a resolution of 4K (partial baseband data 361B having a resolution of 4K in FIG. 13E).

  The 4KDWT unit 231 performs wavelet transform on the partial baseband data 361B having a resolution of 4K once in the vertical direction and once in the horizontal direction. The 4K re-encoding unit 232 encodes the partial wavelet coefficients of each subband of 4K resolution obtained by the wavelet transformation, and the 2K re-encoding unit 334 obtains the partial baseband data of resolution 2K obtained by the wavelet transformation. Then, encoding is performed according to the JPEG2000 system so as to have the same subband configuration as that of the code stream input to the image processing apparatus 300. By these processes, as shown in FIG. 13F, a partial code stream 355B having a resolution of 2K, a partial code stream 356B having a resolution of 4K, a partial code stream 357B having a resolution of 4K, and a partial code stream 358B having a resolution of 4K are generated. . In practice, partial code stream 355B is divided into subbands by wavelet transform.

  Based on the coordinate information, the code stream replacement unit 305 replaces these partial code streams with code streams at spatially corresponding positions, and as shown in FIG. A code stream 350B replaced with the partial code stream from which noise is removed is generated and output.

  Note that the code stream replacement unit 305 updates information regarding the code length and the number of zero bit planes included in the packet header of the data replaced part as necessary.

  FIG. 14 is a block diagram illustrating a detailed configuration example of the 2K re-encoding unit 334 in FIG. As illustrated in FIG. 14, the 2K re-encoding unit 334 includes a replacement code block specifying unit 371, a baseband region extracting unit 372, a DWT (Discrete Wavelet Transform) unit 373, and an encoding unit 374.

  Based on the partial baseband data supplied from the 4KDWT unit 231, the replacement code block specifying unit 371 specifies the part to be replaced in the code stream input to the image processing apparatus 300 in units of code blocks.

  For example, an image 380 shown in FIG. 15A is a 2K resolution image output from the 2K scalable decoding unit 211, and an area 381 indicated by a hatched pattern in the upper right and lower left is a portion where image processing such as noise removal has been performed. . The 2K re-encoding unit 334 performs wavelet transform on such baseband data with a resolution of 2K, and encodes the obtained coefficients in units of code blocks for each subband.

  When the partial baseband data in the region 381 is subjected to wavelet transform, for example, wavelet coefficients for each subband are obtained like coefficients 382-1 to 382-10 indicated by the hatched pattern in the upper right and lower left of FIG. When the range of the coefficients for each subband is narrower than the code block unit, the surrounding coefficients are also required for encoding the coefficients 382-1 to 382-10. For example, in FIG. 15B, the code block is larger than the coefficient 382-1 to the coefficient 382-4 in the lowermost subband, such as the code block 383-1 to the code block 383-4 indicated by the hatched pattern in the upper right and lower left. And

  As described above, encoding and decoding are performed in units of this code block. In this case, all the coefficients included in the range of the code block 383-1 to the code block 383-4 are included in the coefficients 382-1 to 382. 4 changes will be affected. That is, at the baseband level, as shown in FIG. 15A, the data in the region 384 indicated by the hatched pattern at the upper right and lower left is affected.

  That is, in order to replace the data in the area 381, it is necessary to replace all the data in the area 384. As described above, due to the restriction of the processing data unit (code block) at the time of encoding / decoding, the restriction of the processing data unit also occurs in the replacement process of the code stream of each subband.

  Therefore, first, the replacement code block specifying unit 371 performs coefficient (wavelet transform) on the partial baseband data (region 381 in the example of FIG. 15) supplied from the 4KDWT unit 231 by encoding in each subband. In the example of FIG. 15B, a code block (replacement code block) affected by the coefficients 382-1 to 382-10) is specified.

  That is, the replacement code block specifying unit 371 specifies a code block including a coefficient obtained by wavelet transforming the partial baseband data supplied from the 4KDWT unit 231 in the lowest subband, and further, the lowest code. A code block corresponding to the code block (a code block subjected to inverse wavelet transform using the lowest code block) is specified in each upper subband.

  The replacement code block specifying unit 371 supplies information of the specified code block to the baseband region extracting unit 372. The baseband region extraction unit 372 corresponds to the replacement code block of the baseband data of 2K resolution supplied from the 2K scalable decoding unit 211 based on the information of the replacement code block supplied from the replacement code block specifying unit 371. The area to be identified is specified, and the data of the area is extracted. The baseband region extraction unit 372 supplies the extracted partial baseband data with a resolution of 2K to the DWT unit 373. The DWT unit 373 performs wavelet transform on the partial baseband data with a resolution of 2K supplied from the baseband region extraction unit 372, and supplies the obtained coefficient to the encoding unit 374. The encoding unit 374 encodes the coefficient data supplied from the DWT unit 373, generates a partial code stream with a resolution of 2K, and supplies it to the code stream replacement unit 305 (FIG. 12).

  An example of the flow of image processing for removing noise components from the frame image as described above will be described with reference to the flowchart of FIG.

  Each process of step S301 thru | or step S303 is performed similarly to each process of step S201 thru | or step S203 of FIG. In step S304, the partial code stream creation unit 304 performs a partial code stream creation process, encodes the partial baseband data from which noise components have been removed, using the JPEG2000 method, and creates a partial code stream.

  In step S305, the code stream replacement unit 305 performs space for each subband of the 4K resolution generated by the process of step S304 for each subband of the 4K resolution input to the image processing apparatus 200. The corresponding part (that is, the code stream including the noise component) is identified as the partial code stream (that is, the code stream from which the noise component has been removed) of each subband having the resolution of 4K generated by the process of step S304. ). Further, in step S306, the code stream replacement unit 305, for each subband having a resolution of 2K or less of the code stream input to the image processing apparatus 200, a portion of each subband having a resolution of 2K or less generated by the process of step S304. A portion spatially corresponding to the code stream is specified, and the portion (that is, the code stream including the noise component) is determined as the partial code stream (that is, the noise component is included in each subband having a resolution of 2K or less generated by the process of step S304. (Removed code stream). Further, in step S307, the code stream replacement unit 305 updates information regarding the code length and the number of zero bit planes in the header of the packet including the replaced code block.

  When the process of step S307 ends, the image processing ends. This image processing is performed for each frame image of the image data input to the image processing apparatus 300.

  Next, an example of the flow of the partial code stream creation process executed in step S304 in FIG. 16 will be described with reference to the flowchart in FIG.

  When the partial code stream creation processing is started, in step S311, the 4KDWT unit 231 performs wavelet transform once on the partial baseband data with a resolution of 4K from which noise components have been removed in the vertical direction and the horizontal direction. In step S312, the 4K re-encoding unit 232 encodes the partial baseband data of each subband having a resolution of 4K generated by the process of step S311. In step S313, the 2K re-encoding unit 334 further sets the partial baseband data with the resolution of 2K generated by the process of step S311 to have the same configuration as that of the code stream input to the image processing apparatus 300. The wavelet transform is recursively performed, and the obtained coefficient is encoded by a predetermined method. When the process of step S313 ends, the partial code stream creation process ends, the process returns to step S304 of FIG. 16, and the processes after step S305 are executed.

  Next, an example of the flow of 2K re-encoding processing executed in step S313 in FIG. 17 will be described with reference to the flowchart in FIG.

  When the 2K re-encoding process is started, in step S321, the replacement code block specifying unit 371 determines the position of the partial baseband data with the resolution 2K generated by the process at step S311 in the baseband of the frame image with the resolution 2K. In each subband when the wavelet transform is recursively performed a predetermined number of times. That is, the replacement code block specifying unit 371 specifies a region affected by editing of each subband when the baseband of the frame image of 2K resolution is recursively wavelet transformed a predetermined number of times. If the conditions of wavelet transform and encoding are constant, the positional relationship between each data of baseband data and each wavelet coefficient obtained by wavelet transforming the baseband data is constant. Therefore, in that case, the replacement code block specifying unit 371 prepares rule information such as a conversion table and a conversion formula in advance, and uses the rule information to thereby determine the partial baseband data in the baseband of the frame image. From the position (that is, the position of the editing area), the area of the coefficient of each subband corresponding to the editing area can be easily obtained without actually performing wavelet transform.

  In step S322, the replacement code block specifying unit 371 further specifies a replacement code block affected by editing based on the coefficient region of each subband corresponding to the editing region obtained in step S321. That is, the replacement code block specifying unit 371 specifies the code block including the coefficient obtained in step S321 in the lowest subband, and specifies the code block corresponding to the code block in each subband. If the position of the coefficient of each subband is specified in step S321, the position of the target replacement code block is specified by the position information. Therefore, the replacement code block specifying unit 371 can prepare rule information such as a conversion table and a conversion formula in advance, and can easily obtain a replacement code block based on the rule information.

  As described above, the replacement code block specifying unit 371 fixes the wavelet transform and encoding conditions, thereby replacing the replacement code block corresponding to the editing area (the code block affected by editing) based on a predetermined rule. ) Can be specified. Therefore, the replacement code block specifying unit 371 prepares in advance rule information such as a conversion table and a conversion formula for specifying a replacement code block from the position of the partial baseband data from which noise has been removed, and in steps S321 and S322. The processing may be performed with a single conversion.

  When the replacement code block is specified, the baseband region extraction unit 372 extracts a region including the specified replacement code block in the baseband data of 2K resolution in step S323. The baseband region extraction unit 372 obtains a range corresponding to the identified replacement code block in the baseband data with 2K resolution, and extracts data in the range from the baseband data with 2K resolution generated in the process of step S301. Extract.

  In step S324, the DWT unit 373 recursively wavelet transforms the partial baseband data extracted in step S323 a predetermined number of times. In step S325, the encoding unit 374 encodes the wavelet coefficient obtained in step S324 by a predetermined method.

  As described above, the image processing apparatus 300 detects a noise component at a resolution 2K lower than the original resolution 4K, decodes only a portion including the noise component at a resolution 4K, and performs an editing process. As a result, the image processing apparatus 300 decodes the code stream of the entire frame image at a resolution of 4K, removes the noise component, and re-encodes the code stream associated with the editing process and the noise component. It is possible to reduce the load of each process such as the removal and re-encoding of the baseband data, the processing time, the memory capacity necessary for data holding, and the like. That is, the image processing apparatus 300 can perform the editing process more easily than when the entire frame image is decoded at a resolution of 4K and the editing process is performed.

  In the case of this image processing apparatus 300, compared to the case of the image processing apparatus 200, the amount of data to be replaced can be reduced, and changes in the image caused by processing such as editing processing and replacement can be reduced. Depending on the amount of noise components included in the image, the load and processing time of each processing such as editing processing may be reduced or the amount of memory required for data retention may be reduced as compared with the case of the image processing apparatus 200. it can.

  Further, in the above description, in order to remove the noise component of the subband of 4K resolution, it has been described that the decoding is performed at the resolution of 4K. However, the present invention is not limited to this. For example, the noise component included in the subband of resolution 4K The code stream may be removed as it is.

  FIG. 19 is a block diagram showing still another configuration example of the image processing apparatus to which the present invention is applied. In FIG. 19, the same reference numerals are assigned to blocks that perform the same processing as in FIG. An image processing apparatus 400 shown in FIG. 19 is an apparatus that removes noise from each frame image of image data, like the image processing apparatus 100 of FIG. However, the image processing apparatus 400 detects a noise component using baseband data at a resolution of 2K, performs an editing process for removing the noise component, and converts a noise component included in a subband of a resolution of 4K into a code stream. Edit and remove.

  The image processing apparatus 400 includes a 4K editing area extracting unit 402, a noise removing unit 403, a 2K re-encoding unit 404, and a code stream replacing unit 405 in addition to the editing area setting unit 201 in FIG.

  As shown in FIG. 20A, the code stream 450A input to the image processing apparatus 400 includes 2KS451A that is a 2K resolution codestream, 4KSHL452A, 4KSLH453A, and 4KSHH454A that are 4K resolution subband codestreams. . Then, as shown in FIG. 17B, 2KBB 451B, which is baseband data of a 2K resolution frame image, is created by the 2K scalable decoding unit 211, and an editing region including a noise component is set by the editing region specifying unit 212 (FIG. 17B). 20B partial image 455A).

  The 4K editing area extraction unit 402 performs the same processing as the 4K editing area extraction unit 222 in FIG. 5, and includes a code including the editing area specified by the editing area specifying unit 212 from the code stream input to the image processing apparatus 400. The block is extracted and supplied to the 4K noise removing unit 412 of the noise removing unit 403. That is, in the case of the example of FIG. 20, as illustrated in FIG. 20C, the 4K editing area extraction unit 402 performs a partial code stream 456A of a subband of 4K resolution corresponding to a partial image 455A of 2K resolution, and a partial code. A stream 457A and a partial code stream 458A are extracted.

  In this image processing apparatus 400, since the noise component of the subband of 4K resolution is removed as it is in the code stream (since it is not subjected to inverse wavelet transform), as shown in FIG. No code block extraction is required.

  The noise removing unit 403 performs an editing process for removing the noise component detected by the editing area specifying unit 212. The noise removal unit 403 includes a 2K noise removal unit 411 and a 4K noise removal unit 412.

  The 2K noise removal unit 411 performs predetermined processing such as filter processing or replacement processing on the editing region specified by the editing region specifying unit 212 of the baseband data of the 2K resolution frame image generated by the 2K scalable decoding unit 211. The noise component is removed by the image editing process, and the baseband data (2KBB451C in FIG. 20D) of the 2K resolution frame image from which the noise component has been removed is supplied to the 2K re-encoding unit 404. That is, the partial image 455B included in 2KBB 451C in FIG. 20D is obtained by removing noise components from the partial image 455A.

  The 4K noise removing unit 412 removes noise components from the partial code stream of the subband of 4K resolution of the code block including the editing area extracted by the 4K editing area extracting unit 402 (partial code streams 456B to 456B in FIG. 20D). Partial code stream 458B). For example, the 4K noise removal unit 412 replaces the partial code stream supplied from the 4K editing area extraction unit 402 with a partial code stream spatially corresponding to the preceding and succeeding frame images, or duplicates from a surrounding code block. The noise component is removed by replacing the partial code stream. The noise component removal method is arbitrary as long as it can be performed as it is in the code stream.

  The 2K re-encoding unit 404 encodes the baseband data of the 2K resolution frame image from which the noise component has been removed by the 2K noise removing unit 411 according to the JPEG2000 system, and generates a code stream (2KS451D in FIG. 20E) of the 2K resolution frame image. This is supplied to the code stream replacement unit 405. By this encoding, the partial image 455B in FIG. 20D is converted into a partial code stream 455C in FIG. 20E. In practice, partial code stream 455C is divided into subbands by wavelet transform.

  The code stream replacement unit 405 replaces the code stream of the 2K resolution frame image of the code stream input to the image processing device 400 with the data supplied from the 2K re-encoding unit 404, and further inputs the code stream to the image processing device 400. In the substream code stream having a resolution of 4K, the code block including a portion spatially corresponding to the editing area is replaced with the data supplied from the 4K noise removing unit 412. Output to the outside.

  That is, the code stream 450B output from the code stream replacement unit 405 includes a code stream 451D of a 2K resolution frame image having a partial code stream 455C of 2K resolution from which noise components are removed, as shown in FIG. 20F. 4KSHL452B, which is a 4K resolution subband code stream 456B with a noise component removed, and 4KSHL 452B, a 4K resolution subband code stream 457B with noise component removed, 4KSLH 453B, which is a 4K resolution subband code stream, and 4KSHH 454B, which is a 4K resolution subband code stream having a sub code stream 458B of 4K resolution from which noise components have been removed. Ri made.

  Note that the code stream replacement unit 405 updates information regarding the code length and the number of zero bit planes included in the packet header of the part where the data is replaced, as necessary.

  An example of the flow of image processing for removing such noise components from the frame image will be described with reference to the flowchart of FIG.

  When image processing is started, in step S401, the editing area setting unit 201 performs editing area setting processing and sets an editing area at 2K resolution. The details of the editing area setting process are the same as those described with reference to the flowchart of FIG. In step S <b> 402, the 4K editing area extraction unit 402 extracts a partial code stream of a portion spatially corresponding to the editing area from a subband code stream with a resolution of 4K.

  In step S403, the noise removal unit 403 performs noise removal processing to remove noise components from the image.

  In step S404, the 2K re-encoding unit 404 encodes the baseband data of the 2K resolution frame image from which noise has been removed by the 2K noise removal unit 411, using the JPEG2000 method.

  In step S405, the code stream replacement unit 405 identifies a portion corresponding to the partial code stream from which noise has been removed by the processing in step S403, for each subband of resolution 4K of the code stream input to the image processing apparatus 400. The portion (that is, the code stream including the noise component) is replaced with the partial code stream from which noise has been removed by the process of step S403. In step S406, the code stream replacement unit 405 converts the code stream input to the image processing apparatus 400 to a substream code stream having a resolution of 2K or less (that is, a code stream including a noise component) by the process in step S404. It is replaced with a code stream of a frame image having a resolution of 2K generated by encoding (that is, a code stream from which noise components have been removed). Further, in step S407, the code stream replacement unit 405 appropriately updates information regarding the code length and the number of zero bit planes in the header of the packet including the replaced code block.

  When the process of step S407 ends, the image processing ends. This image processing is performed for each frame image of the image data input to the image processing apparatus 400.

  Next, an example of the flow of the noise removal process executed in step S403 in FIG. 21 will be described with reference to the flowchart in FIG.

  When the noise removal process is started, in step S411, the 2K noise removal unit 411 removes the noise component in the editing area set in the process in step S401 from the baseband data of the frame image with 2K resolution. In step S412, the 4K noise removal unit 412 removes the noise component of the partial code stream of each subband of resolution 4K extracted by the process in step S402 with other data, and removes the code stream as it is.

  When the process of step S412 is completed, the noise removal process is terminated, the process returns to step S403 in FIG. 21, and the processes after step S404 are executed.

  As described above, the image processing apparatus 400 detects a noise component at a resolution 2K lower than the original resolution 4K, and removes the noise component included in the subband of the resolution 4K as a code stream without decoding. Edit processing to be performed. As a result, the image processing apparatus 400 decodes the code stream of the entire frame image at a resolution of 4K, removes the noise component, and re-encodes the code stream associated with the editing process and the noise component. It is possible to reduce the load of each process such as the removal and re-encoding of the baseband data, the processing time, the memory capacity necessary for data holding, and the like. That is, the image processing apparatus 400 can perform the editing process more easily than when the entire frame image is decoded at a resolution of 4K and the editing process is performed.

  In the case of the image processing apparatus 400, as compared with the case of the image processing apparatus 200 and the image processing apparatus 300, the decoding process and the encoding process for the subband of 4K resolution can be omitted. As a result, the image processing apparatus 400 can reduce the load and processing time of each process such as editing processing and the amount of memory required for data retention, as compared with the image processing apparatus 200 and the image processing apparatus 300. be able to. However, since the image processing apparatus 400 removes the noise component of the subband having a resolution of 4K as the code stream, the image processing apparatus 200 and the image processing apparatus 300 can more accurately remove the noise component than the image processing apparatus 400. Can be removed.

  In the above, the case of removing a noise component has been described as an example of the editing process. However, the content of the editing process is not limited to this, and any process can be used as long as it is a process for editing scalable encoded data. There may be. For example, a process of combining a frame image with a composition image or characters may be used. That is, although the image processing apparatus has been described above, the present invention can be applied to apparatuses other than image processing apparatuses as long as they are apparatuses that process scalable encoded data.

  The data to be edited may be any data that is encoded in a scalable manner, and may be data other than image data such as audio data. Furthermore, in the above description, the data is compressed using the JPEG2000 method. However, if the data is encoded in a scalable manner, for example, the MPEG (Moving Picture Experts Group) method, the H.264 / AVC method, the wavelet transform Any coding / decoding method such as various user coding / decoding methods may be used.

  In the above, the case where there is one editing area has been described as an example, but this editing area can be set as many as the number of noise components, and a plurality of editing areas may be set in one frame image. Further, when a plurality of 1-frame image editing areas are set, each editing area may be processed one by one, or processing for a plurality of editing areas may be executed in parallel. For example, a processing result or intermediate data obtained in one editing area may be used for processing for another editing area.

  In the above description, 4K and 2K have been described as examples of the resolution. However, the resolution 4K is an example of high resolution, the resolution 2K is an example of low resolution, and each resolution is arbitrary. That is, the present invention only needs to be able to process high-resolution scalable encoded data at low resolution. For example, noise that can be decoded in three or more stages is detected at the minimum resolution, and the highest resolution portion is detected. You may make it perform a noise removal process with an image.

  Further, in the above description, the image processing is repeated for each frame image. However, the present invention is not limited to this, and the unit for repeating the image processing may be smaller than the frame image, or a plurality of frame images may be converted into one image. You may make it process by a process.

  In the above description, for convenience of explanation, the quantization process at the time of the encoding process and the inverse quantization process at the time of the decoding process have been omitted. However, the present invention is not limited to this. Of course, the quantization may be performed and the inverse quantization may be performed at the time of decoding.

  The series of processes described above can be executed by hardware or can be executed by software. In this case, for example, it may be configured as a personal computer as shown in FIG.

  In FIG. 23, a CPU (Central Processing Unit) 501 of the personal computer 500 performs various processes according to a program stored in a ROM (Read Only Memory) 502 or a program loaded from a storage unit 513 to a RAM (Random Access Memory) 503. Execute the process. The RAM 503 also appropriately stores data necessary for the CPU 501 to execute various processes.

  The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input / output interface 510 is also connected to the bus 504.

  The input / output interface 510 includes an input unit 511 including a keyboard and a mouse, a display including a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal Display), an output unit 512 including a speaker, and a hard disk. A communication unit 514 including a storage unit 513 and a modem is connected. The communication unit 514 performs communication processing via a network including the Internet.

  A drive 515 is connected to the input / output interface 510 as necessary, and a removable medium 521 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted, and a computer program read from them is It is installed in the storage unit 513 as necessary.

  When the above-described series of processing is executed by software, a program constituting the software is installed from a network or a recording medium.

  For example, as shown in FIG. 23, the recording medium is distributed to distribute a program to a user separately from the apparatus main body, and includes a magnetic disk (including a flexible disk) on which a program is recorded, an optical disk ( It only consists of removable media 521 consisting of CD-ROM (compact disc-read only memory), DVD (including digital versatile disc), magneto-optical disc (including MD (mini disc)), or semiconductor memory. Rather, it is composed of a ROM 502 on which a program is recorded and a hard disk included in the storage unit 513, which is distributed to the user in a state of being pre-installed in the apparatus main body.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. It also includes processes that are executed individually.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of devices (apparatuses).

  In the above, the configuration described as one device may be divided and configured as a plurality of devices. Conversely, the configurations described above as a plurality of devices may be combined into a single device. Of course, configurations other than those described above may be added to the configuration of each device. Furthermore, if the configuration and operation of the entire system are substantially the same, a part of the configuration of a certain device may be included in the configuration of another device. That is, the embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a figure which shows the example of the mode of an edit process. It is a block diagram which shows the main structural examples of the image processing apparatus to which this invention is applied. It is a figure explaining the example of the mode of the data in an edit process. It is a flowchart explaining the example of the flow of an image process. It is a block diagram which shows the other structural example of the image processing apparatus to which this invention is applied. It is a figure explaining the other example of the mode of the data in an edit process. It is a figure explaining the example of the range of the code block to extract. It is a flowchart explaining the other example of the flow of an image process. It is a flowchart explaining the example of the flow of an edit area | region setting process. It is a flowchart explaining the example of the flow of a partial baseband data creation process. It is a flowchart explaining the example of the flow of a partial code stream creation process. FIG. 12 is a block diagram illustrating still another configuration example of an image processing apparatus to which the present invention has been applied. It is a figure explaining the further another example of the mode of the data in an edit process. It is a block diagram which shows the detailed structural example of the 2K re-encoding part of FIG. It is a figure explaining the example of the mode of a code block. It is a flowchart explaining the further another example of the flow of an image process. It is a flowchart explaining the other example of the flow of a partial code stream creation process. It is a flowchart explaining the example of the flow of 2K re-encoding process. FIG. 12 is a block diagram illustrating still another configuration example of an image processing apparatus to which the present invention has been applied. It is a figure explaining the further another example of the mode of the data in an edit process. It is a flowchart explaining the further another example of the flow of an image process. It is a flowchart explaining the example of the flow of a noise removal process. It is a block diagram which shows the structural example of the personal computer to which this invention is applied.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 Image processing apparatus, 101 2K scalable decoding part, 102 Noise removal part, 103 2K re-encoding part, 104 2K code stream replacement part, 200 Image processing apparatus, 201 Editing area setting part, 202 Partial baseband data preparation part, 203 Noise Removing unit, 204 partial code stream creating unit, 205 code stream replacing unit, 211 2K scalable decoding unit, 212 editing region specifying unit, 221 2K editing region extracting unit, 222 4K editing region extracting unit, 223 decoding unit, 224 IDWT unit, 231 4KDWT unit, 232 4K re-encoding unit, 233 2K baseband replacement unit, 234 2K re-encoding unit, 300 image processing device, 304 partial code stream creation unit, 305 code stream Replacement unit, 334 2K re-encoding unit, 371 replacement code block identification unit, 372 baseband region extraction unit, 373 DWT unit, 374 encoding unit, 400 image processing device, 402 4K editing region extraction unit, 403 noise removal unit, 404 2K re-encoding unit, 405 code stream replacement unit, 411 2K noise removal unit, 412 4K noise removal unit

Claims (12)

An information processing apparatus for processing a code stream of the first resolution in which data of a first resolution is encoded in a scalable manner,
An editing area in which the code stream of the first resolution is decoded at a second resolution lower than the first resolution and editing is performed using the obtained baseband data of the second resolution. Setting means for setting;
A partial baseband data having a first resolution is generated, in which a portion of the editing area set by the setting unit of the code stream having the first resolution is baseband data decoded at the first resolution. A first creation means;
Editing means for performing editing processing on the partial baseband data of the first resolution created by the first creating means ;
Second creation means for scalablely coding the partial baseband data of the first resolution edited by the editing means and creating a partial code stream that is a code stream of the editing area;
An information processing apparatus comprising: a code stream replacing unit that replaces a portion of the editing area of the code stream of the first resolution with the partial code stream created by the second creating unit . The first creation means includes:
First extraction means for extracting a portion of the editing area from the baseband data of the second resolution;
Second extraction means for extracting a portion of the editing area from the code stream of the first resolution;
Decoding means for decoding the portion of the editing area of the code stream of the first resolution extracted by the second extraction means;
The part of the editing area of the baseband data of the second resolution extracted by the first extracting unit and the part of the editing area of the code stream of the first resolution are decoded by the decoding unit. Inverse wavelet transform means for creating partial baseband data of the first resolution by performing inverse wavelet transform on the portion of the editing region of the subband of each high-frequency component of the first resolution obtained. Item 4. The information processing apparatus according to Item 1. The information processing apparatus according to claim 1, wherein the editing unit performs editing to remove a noise component included in the partial baseband data having the first resolution. The second creation means includes:
Wavelet transforming means for wavelet transforming the edited partial baseband data of the first resolution at the first resolution;
First encoding means for encoding a portion of the editing area of a subband of each high-frequency component of the first resolution obtained by wavelet transform processing by the wavelet transform means;
The portion of the editing area of the baseband data of the second resolution obtained by the setting means is included in the subband of the low frequency component of the first resolution obtained by the wavelet transform processing by the wavelet transform means. Baseband data replacement means for replacing the portion of the editing area of the baseband data of the second resolution to be
The information processing apparatus according to claim 1, further comprising: a second encoding unit that encodes the baseband data of the second resolution in which the part of the editing area is replaced. The second creation means includes:
Wavelet transforming means for wavelet transforming the edited partial baseband data of the first resolution at the first resolution;
First encoding means for encoding a portion of the editing area of a subband of each high-frequency component of the first resolution obtained by wavelet transform processing by the wavelet transform means;
2. The information processing apparatus according to claim 1, further comprising: a second encoding unit that encodes a portion of the editing area of the baseband data of the second resolution obtained by the wavelet transform process by the wavelet transform unit. . An information processing method of an information processing apparatus for processing a code stream of the first resolution in which data of a first resolution is encoded in a scalable manner,
An editing area in which the code stream of the first resolution is decoded at a second resolution lower than the first resolution and editing is performed using the obtained baseband data of the second resolution. Set,
Of the first resolution code stream, a baseband data of the set portion of the editing area decoded by the first resolution a first resolution, the partial baseband data of the first resolution make,
An editing process is performed on the generated partial baseband data of the first resolution,
The edited partial baseband data of the first resolution is encoded in a scalable manner to create a partial code stream that is a code stream of the editing area,
An information processing method including a step of replacing a portion of the editing area of the code stream of the first resolution with the generated partial code stream. In a program for processing a code stream of the first resolution in which data of the first resolution is encoded in a scalable manner,
An editing area in which the code stream of the first resolution is decoded at a second resolution lower than the first resolution and editing is performed using the obtained baseband data of the second resolution. A setting step to set;
Said first resolution code stream, a baseband data portion of said first of said first decoded by a resolution of the editing area set by the processing of said setting step, the first resolution A first creation step of creating partial baseband data of
An editing step of performing an editing process on the partial baseband data of the first resolution created by the process of the first creating step ;
A second creation step of scalablely coding the partial baseband data of the first resolution edited by the processing of the editing step and creating a partial code stream that is a code stream of the editing area;
A program that causes a computer to execute a code stream replacement step of replacing a portion of the edit area of the code stream of the first resolution with the partial code stream created by the processing of the second creation step . An information processing apparatus for processing a code stream of the first resolution in which data of a first resolution is encoded in a scalable manner,
An editing area in which the code stream of the first resolution is decoded at a second resolution lower than the first resolution and editing is performed using the obtained baseband data of the second resolution. Setting means for setting;
Extraction means for extracting a portion of the editing area set by the setting means from a code stream of a subband of each high-frequency component of the first resolution included in the code stream of the first resolution;
For each of the editing region portion of the subband codestream of each high-frequency component of the first resolution extracted by the extracting means and the editing region portion of the baseband data of the second resolution. Editing means for performing editing processing,
Encoding means for encoding in a scalable baseband data of the portion of the editing area is edited second resolution by said editing means,
The second resolution baseband data obtained by editing the edit area portion of the code stream portion of the second resolution of the first resolution code stream is encoded by the encoding means in a scalable manner. A portion of the editing area of the substream code stream of each high-frequency component of the first resolution included in the code stream of the first resolution. An information processing apparatus comprising: a replacing unit that replaces a part of the code stream of the subband of each high-frequency component of the first resolution subjected to the editing process by the editing unit. The editing means, for each of the editing area portion of the substream code stream of each high-frequency component of the first resolution, and each of the editing area portion of the baseband data of the second resolution, The information processing apparatus according to claim 8, wherein editing is performed to remove a noise component. The said editing means removes the said noise component by replacing the part of the said edit area | region of the code stream of the subband of each high frequency component of the said 1st resolution with a predetermined other code stream. The information processing apparatus described. An information processing method of an information processing apparatus for processing a code stream of the first resolution in which data of a first resolution is encoded in a scalable manner,
An editing area in which the code stream of the first resolution is decoded at a second resolution lower than the first resolution and editing is performed using the obtained baseband data of the second resolution. Set,
Extracting the set edit area from the subband codestream of each high frequency component of the first resolution included in the codestream of the first resolution,
Editing processing is performed on each of the editing area portion of the extracted subband code stream of each high-frequency component of the first resolution and the editing area portion of the baseband data of the second resolution. Done
The baseband data of the second resolution in which a part of the editing area is edited is encoded in a scalable manner,
The second resolution codestream portion of the first resolution codestream portion, the second resolution baseband data in which the edit area portion is edited, and the second resolution baseband data are scalable. And the editing region portion of the substream code stream of each high-frequency component of the first resolution included in the code stream of the first resolution. The information processing method includes a step of replacing the editing region portion of the substream code stream of each high-frequency component of the first resolution that has been performed. In a program for processing a code stream of the first resolution in which data of the first resolution is encoded in a scalable manner,
An editing area in which the code stream of the first resolution is decoded at a second resolution lower than the first resolution and editing is performed using the obtained baseband data of the second resolution. A setting step to set;
An extraction step of extracting a portion of the editing area set by the setting step from a substream code stream of each high-frequency component of the first resolution included in the code stream of the first resolution;
Each of the editing area portion of the subband code stream of each high frequency component of the first resolution extracted by the processing of the extraction step, and each of the editing area portion of the baseband data of the second resolution. An editing step for performing an editing process on
An encoding step for scalablely encoding the baseband data of the second resolution in which a portion of the editing area is edited by the processing of the editing step ;
The code stream portion of the second resolution of the code stream of the first resolution and the baseband data of the second resolution obtained by editing the portion of the editing area are made scalable by the processing of the encoding step. The editing region of the substream code stream of each high-frequency component of the first resolution, which is replaced with the encoded code stream of the second resolution and further included in the code stream of the first resolution And a replacement step of causing the computer to execute a replacement step of replacing the portion of the editing region with the portion of the editing region of the subband codestream of each high-frequency component of the first resolution that has been subjected to the editing processing by the processing of the editing step. .

Images