JP2005101728A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of lessening burden on the software and shortening the processing time of motion compensation processing in MPEG4 decoding. <P>SOLUTION: The image processing apparatus performing motion compensation processing comprises a memory 10 for storing image data, and an address generating section 11 for comparing a reference image address obtained from motion vectors mvx and mvy with an image boundary address, and accessing the memory with the image boundary address to read out image data from the memory if the reference image address deviates from the range of the image boundary address. When the image data read out at the address generating section 11 has a plurality of bits width, a data output selector 15 outputs the image data thus read out in units of pixel while extending the pixel data at the image boundary. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus, and more particularly to a technique effective when applied to motion compensation processing of an MPEG4 decoder (decoder).

  As techniques studied by the present inventor, for example, the following techniques are conceivable in motion compensation processing of an MPEG (Moving Picture Experts Group) decoder (decoder).

  In a conventional decoder such as MPEG2, only restoration of an image processed by a specific encoder (encoder) is assumed, so that motion compensation designating outside the image range is not performed. Therefore, data outside the image range is not specified at the time of decoding (decoding), and a function for generating data outside the image range is not necessary.

  On the other hand, in a moving image compression standard such as ISO / IEC 14496-2 (MPEG4), the motion vector may point outside the image range. In other words, motion compensation can be performed with reference to an area outside the image range. When the region specified by the motion vector is outside the image range, a pixel on the boundary is used as a predicted value (see, for example, Non-Patent Document 1).

  Since MPEG4 so far has a circuit configuration of only encoding (encoding), MPEG4 was supported by prohibiting designation outside the image range by software during image compression.

  However, when the image data is compressed by designating the outside of the image range with another general-purpose encoder, it is necessary to restore the image data using the boundary image according to the standard. In the general-purpose MPEG4 decoder, data encoded by another general-purpose encoder needs to be decoded, and it has become impossible to cope with the above method.

  The following methods can be considered as countermeasures examined by the present inventors.

  (1) Based on the image data at the boundary on the memory, the boundary data is written in advance in a region outside the image range that would be predicted.

(2) Capture boundary data from the memory and generate data outside the image range with software.
Shinichi Miki, “All about MPEG-4”, first edition, Industrial Research Committee, September 30, 1998, p. 45

  By the way, as a result of the study of the motion compensation processing technique at the time of MPEG4 decoding as described above, the following has been clarified.

  For example, in the method (1), since a region outside the image range that would be predicted is not determined by the general-purpose encoder, it cannot be predicted in advance. Even if the range is known in advance and can be realized, it takes too much transfer time to embed data in the memory, and the hardware configuration becomes considerably complicated, which is not realistic.

  In the method (2), since control for generating data outside the image range is performed only by software, the time required for image processing increases, and the VGA (Video Graphic Array) size is processed at 30 frames / second. It may be impossible.

  Accordingly, an object of the present invention is to provide an image processing apparatus capable of reducing the burden of software and shortening the processing time in motion compensation processing during MPEG4 decoding.

  The above and other objects and novel features of the present invention will be apparent from the description of this specification and the accompanying drawings.

  Of the inventions disclosed in the present application, the outline of typical ones will be briefly described as follows.

  (1) An image processing apparatus according to the present invention is an image processing apparatus that performs motion compensation processing, and compares a memory that stores image data with a reference image address obtained from a motion vector and an image boundary address. And a second means for accessing the memory by the image boundary address and reading the image data from the memory when the reference image address is outside the range of the image boundary address as a result of the comparison in the first means. Is.

  (2) In the image processing apparatus according to (1), when the image data read by the second means has a plurality of bit widths, the read image data is expanded on a pixel basis while extending the pixel data at the image boundary. And a third means for shifting and outputting.

  (3) In the image processing apparatuses (1) and (2), the first means, the second means, and the third means are constituted by hardware.

  (4) In the image processing apparatuses according to (1) to (3), the motion compensation processing is motion compensation processing in the MPEG4 standard.

  Of the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows.

  (1) When an area outside the image range is designated, a motion vector can be designated even for an infinite area.

  (2) It is possible to cope with restoration of data compressed by a general-purpose encoder without increasing the memory transfer time of hardware or the amount of processing of software.

  (3) Even when motion compensation points out of the image range in the restoration process of the image compressed by another encoder, it is not necessary to determine and control whether the image is out of the image range or within the range by software. The burden on software is reduced and the processing time of the entire system is shortened.

  (4) The VGA size can be processed at 30 frames / second.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.

  FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention. In the following, although not limited to this, a case where one word has a 32-bit configuration will be described as an example. Further, the data for one pixel has an 8-bit (1 byte) structure, and the data for 32 bits (1 word) corresponds to 4 pixels.

  First, an example of the configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. The image processing apparatus according to this embodiment is, for example, an MPEG4 decoder (decoder), and includes a memory 10, an address generation unit 11, buffers 12, 13, and 14, a data output selector 15, and the like. Motion vectors mvx and mvy are input, the vertical direction control signal 16 is input from the address generation unit 11 to the memory 10, and the horizontal direction control signal 17 is input from the address generation unit 11 to the data output selector 15. Image data is input to the buffers 12, 13, and 14 via a 32-bit bus, and 32-bit image data vctlt00 [31: 0] and vctlt01 [ 31: 0], vctlt02 [31: 0] are input, and the data output selector Each 32-bit image data vctltsel00 from motor 15 [31: 0], vctltsel01 [31: 0], vctltsel02 [31: 0] is output.

  The memory 10 is a memory for storing image data, and has, for example, a 32-bit width data output.

  The address generator (first means, second means) 11 generates a reference image address from the motion vectors mvx and mvy, compares it with the image boundary address, determines underflow or overflow, and determines the memory 10 The memory 10 and the data output selector 15 are controlled by the vertical direction control signal 16 and the horizontal direction control signal 17. If the reference image address is outside the range of the image boundary address, the memory 10 is accessed using the image boundary address, and the image data is read out.

  The buffers 12, 13, and 14 temporarily hold the image data read from the memory 10, and are configured by flip-flops or the like that hold 32-bit data, for example.

  The data output selector (third means) 15 applies the 32-bit image data vctlt00 [31: 0], vctlt01 [31: 0], and vctlt02 [31: 0] output from the buffers 12, 13, and 14, respectively. Thus, pixel data at the image boundary is expanded (stretched) while being shifted in units of pixels, and image data vctltsel00 [31: 0], vctltsel01 [31: 0], and vctltsel02 [31: 0] are output. The shift amount of the image data is determined by the address generation unit 11 according to the value of the motion vector mvx, and the data is extended / shifted by the horizontal direction control signal 17.

  Next, the operation of the image processing apparatus according to the present embodiment will be described. In the following, although not limited to this, it is assumed that the address values indicating the coordinate positions such as the image range, the reference image, and the motion vector are small in the upper left and large in the lower right.

  FIG. 2 is an explanatory diagram showing motion compensation processing in the case where the motion vector specifies outside the image range in the MPEG4 standard. In the following description, each cell represents one pixel, and the reference range of one macroblock is composed of 8 pixels × 8 pixels.

  In the MPEG4 standard, the motion vector may point outside the image range 20. In other words, motion compensation can be performed with reference to an area outside the image range 20. When the region (reference range 21) specified by the motion vector is outside the image range 20, pixels on the boundary of the image range 20 are used as predicted values as shown in FIG.

  The image processing apparatus according to the present embodiment specifies pixel data (a, b, c, d, B in FIG. 2) when data is taken in from the memory 10 with the motion vector designating outside the image range 20. , C, D), and has a function of extending by hardware. That is, the address generation unit 11 compares the reference image address obtained from the motion vectors mvx and mvy with the image boundary address. If the reference image address is outside the image range 20, the image boundary data is captured. Hand it over to the software.

  FIG. 3 is an explanatory diagram showing processing outside the upper image range (vertical underflow) in the image processing apparatus according to the present embodiment.

  The processing outside the upper image range (vertical underflow) is performed by comparing the upper boundary address (upper boundary image address 30) with the reference image address 32 obtained from the motion vector 31, and using the larger value. The memory 10 is accessed. That is, when the reference picture address 32 <the upper border picture address 30, the memory 10 is accessed with the upper border picture address 30, and when the reference picture address 32 ≧ the upper border picture address 30, the memory 10 is accessed with the reference picture address 32. . With this method, when the area of the reference image address 32 is outside the upward image range (undefined value), the pixel data on the upper boundary is read. As described above, by manipulating the vertical address, the function of putting the upper boundary pixel in the area outside the upward image range is realized.

  FIG. 4 is an explanatory diagram showing processing outside the downward image range (vertical overflow) in the image processing apparatus according to the present embodiment.

  For processing outside the lower image range (vertical overflow), the lower boundary address (lower boundary image address 40) is compared with the reference image address 42 obtained from the motion vector 41, and the smaller value is used. The memory 10 is accessed. That is, when the reference picture address 42> the lower border picture address 40, the memory 10 is accessed with the lower border picture address 40, and when the reference picture address 42 ≦ the lower border picture address 40, the memory 10 is accessed with the reference picture address 42. . With this method, when the area of the reference image address 42 is outside the downward image range (indefinite value), the pixel data of the lower boundary is read. As described above, by manipulating the address in the vertical direction, the function of putting the lower boundary pixel in an area outside the downward image range is realized.

  FIG. 5 is an explanatory diagram showing processing outside the left image range (horizontal underflow) in the image processing apparatus according to the present embodiment.

  For processing outside the image range in the left direction (horizontal underflow), the left boundary address (left boundary image address 50) is compared with the reference image address 52 obtained from the motion vector 51, and the larger value is used. Then, the memory 10 is accessed. That is, when the reference image address 52 <the left border image address 50, the memory 10 is accessed at the left border image address 50, and when the reference image address 52 ≧ the left border image address 50, the memory 10 at the reference image address 52 is accessed. To access. Further, when the memory 10 cannot be accessed in byte units (one pixel unit), the horizontal direction control signal 17 from the address generation unit 11 expands the leftmost 1-byte data in byte units in the right direction. shift. With this method, when the area of the reference image address 52 is outside the left direction range, the pixel data on the left boundary is read while being expanded. As described above, the function of putting the left boundary pixel in the area outside the left image range is realized by operating the horizontal address.

  FIG. 6 is an explanatory diagram showing processing outside the right image range (horizontal overflow) in the image processing apparatus according to the present embodiment.

  For processing outside the image range in the right direction (horizontal overflow), the right boundary address (right boundary image address 60) is compared with the reference image address 62 obtained from the motion vector 61, and the smaller value is used. The memory 10 is accessed. That is, when the reference image address 62> the right border image address 60, the memory 10 is accessed with the right border image address 60. When the reference image address 62 ≦ the right border image address 60, the memory 10 with the reference image address 62 is accessed. To access. The control method in the case where the memory 10 cannot be accessed in byte units (one pixel unit) is as follows. The horizontal direction control signal 17 from the address generation unit 11 expands the rightmost 1-byte data in byte units in the left direction. shift. With this method, when the area of the reference image address 62 is outside the right direction range, the pixel data on the right boundary is read while being expanded. As described above, by manipulating the address in the horizontal direction, the function of putting the right boundary pixel in the area outside the right image range is realized.

  When the vertical direction and the horizontal direction are simultaneously outside the image range, the above functions are combined.

  FIG. 7 is an explanatory diagram showing processing outside the upper left image range (vertical underflow / horizontal underflow) in the image processing apparatus according to the present embodiment.

  Processing outside the image range in the upper left direction (vertical underflow / horizontal underflow) is performed when the reference image address 72 <the upper boundary image address 30 and the reference image address 72 <the left boundary image address 50, the upper boundary image address 30. And the left border image address 50 are combined to access the memory 10. As described above, by combining the address operations in the vertical direction and the horizontal direction, a function of putting a boundary pixel in a region outside the upper left image range is realized.

  FIG. 8 is an explanatory diagram showing processing outside the upper right image range (vertical underflow / horizontal overflow) in the image processing apparatus according to the present embodiment.

  Processing outside the image range in the upper right direction (vertical underflow / horizontal overflow) is performed when the reference image address 82 <the upper boundary image address 30 and the reference image address 82> the right boundary image address 60. The right border image address 60 is synthesized and the memory 10 is accessed. As described above, by combining the address operations in the vertical direction and the horizontal direction, a function of putting a boundary pixel in an area outside the upper right direction image range is realized.

  FIG. 9 is an explanatory diagram showing processing outside the lower left image range (vertical overflow / horizontal underflow) in the image processing apparatus according to the present embodiment.

  The processing outside the image range in the lower left direction (vertical overflow / horizontal underflow) is performed when the reference picture address 92> the lower border picture address 40 and the reference picture address 92 <the left border picture address 50, The left border image address 50 is synthesized and the memory 10 is accessed. As described above, by combining the address operations in the vertical direction and the horizontal direction, a function of putting a boundary pixel in an area outside the lower left direction image range is realized.

  FIG. 10 is an explanatory diagram showing processing outside the image range in the lower right direction (vertical overflow / horizontal overflow) in the image processing apparatus according to the present embodiment.

  The processing outside the image range in the lower right direction (vertical overflow / horizontal overflow) is performed when the reference picture address 102> the lower border picture address 40 and the reference picture address 102> the right border picture address 60, The right border image address 60 is synthesized and the memory 10 is accessed. As described above, by combining the address operations in the vertical direction and the horizontal direction, a function of putting a boundary pixel in an area outside the lower right image range is realized.

  Therefore, according to the image processing apparatus of the present embodiment, when the outside of the image range is designated, a motion vector can be designated even for an infinite region, and data capture control outside the wide image range is possible. It becomes.

  Further, it is possible to cope with restoration of data compressed by a general-purpose encoder without increasing the memory transfer time of hardware or the processing amount of software.

  That is, even when motion compensation points outside the image range in the restoration processing of an image compressed by another encoder, it is not necessary to determine whether the image is out of the image range or within the range and control by software. As a result, the decoder The burden of software is reduced in the reference image reading process at the time of image restoration by the method, and the processing time of the entire system is shortened. Therefore, the VGA size can be processed at 30 frames / second, and processing in real time is possible.

  The image processing apparatus according to the present embodiment refers to an image based on boundary pixel data in accordance with a standard when referring to pixel data outside the image range by a motion vector in moving image restoration represented by MPEG (MPEG decoding). It can be applied to products that need a function to generate and transfer out-of-range pixel data (digital video cameras, video decks, information terminals, etc.).

  FIG. 11 is an explanatory diagram showing an application example in which the image processing apparatus according to the embodiment of the present invention is applied to the out-of-image-range processing at the time of reference image comparison for MPEG-encoded image compression processing. FIG. 11 shows processing outside the upper left image range (vertical underflow / horizontal underflow), but the other three ends (upper right, lower left, lower right) can be similarly applied.

  By applying the processing outside the image range shown in the above-described embodiment, as shown in FIG. 11, the image compression processing by the MPEG encoder can be performed outside the image range when the reference image for comparison is captured. Data processing outside the image range in this case is the same as the method shown in FIGS.

  Therefore, even in moving image compression represented by MPEG (MPEG encoding), by applying this function at the time of capturing the previous image / rear image for motion detection, software for comparison image reading processing at the time of moving image compression by the encoder is applied. The burden can be reduced, and the entire processing time of the encoder can be shortened.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

  The image processing apparatus according to the present invention is suitable for use in electronic equipment that performs moving picture compression / decompression, such as a digital video camera, a video deck, and an information terminal.

1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. In MPEG4 standard, it is explanatory drawing which shows the motion compensation process when a motion vector designates the outside of an image range. It is explanatory drawing which shows the process outside an upper image range (vertical underflow) in the image processing apparatus which is one embodiment of this invention. It is explanatory drawing which shows the process outside a downward image range (vertical overflow) in the image processing apparatus which is one embodiment of this invention. It is explanatory drawing which shows the process outside a left direction image range (horizontal underflow) in the image processing apparatus which is one embodiment of this invention. It is explanatory drawing which shows the process outside a right direction image range (horizontal overflow) in the image processing apparatus which is one embodiment of this invention. In the image processing apparatus which is one embodiment of this invention, it is explanatory drawing which shows the process of the upper left direction image range (vertical underflow / horizontal underflow). It is explanatory drawing which shows the process of the upper right direction image range (vertical underflow / horizontal overflow) in the image processing apparatus which is one embodiment of this invention. It is explanatory drawing which shows the process of the left lower direction image range (vertical overflow / horizontal underflow) in the image processing apparatus which is one embodiment of this invention. It is explanatory drawing which shows the process of the lower right direction image range (vertical overflow / horizontal overflow) in the image processing apparatus which is one embodiment of this invention. It is explanatory drawing which shows the application example to which the image processing apparatus which is one embodiment of this invention is applied about the image out-of-range process at the time of the comparison reference image capture of MPEG encoding image compression processing.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Memory 11 Address generation part 12, 13, 14 Buffer 15 Data output selector 16 Vertical direction control signal 17 Horizontal direction control signal 20 Image range 21 Reference range 30 Upper boundary image address 31,41,51,61,71,81,91 , 101 Motion vector 32, 42, 52, 62, 72, 82, 92, 102 Reference picture address 40 Lower border picture address 50 Left border picture address 60 Right border picture address

Claims (4)

An image processing apparatus that performs motion compensation processing,
A memory for storing image data;
A first means for comparing a reference image address obtained from a motion vector and an image boundary address;
A second means for accessing the memory by the image boundary address and reading the image data from the memory when the reference image address is outside the range of the image boundary address as a result of the comparison in the first means;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
When the image data read by the second means has a plurality of bit widths, third means for shifting and outputting the read image data in units of pixels while extending the pixel data at the image boundary An image processing apparatus comprising: The image processing apparatus according to claim 1 or 2,
The image processing apparatus, wherein the first means, the second means, and the third means are configured by hardware. The image processing apparatus according to any one of claims 1 to 3,
The image processing apparatus, wherein the motion compensation processing is motion compensation processing in the MPEG4 standard.

Images