JP2000333131A - Image processor and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an image processor that delivers information as to a motion that a moving picture substantially has and can display a sense of the motion even when interleave processing is applied to a frame and only part of image data is used for a display object. SOLUTION: The image processor is provided with a motion generating section 103, which applies weight summing processing using a plurality of pixel values to the pixels being components of image data that are a processing object based on a motion vector corresponding to the image data to generate image data to which moving generation and imparting processing is applied and the resulting data are used for display and transmission or the like.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image processing method, and more particularly, to inputting image data and change information which is information indicating a temporal change of the image data, and displaying image data for display. The present invention relates to an image processing apparatus and an image processing method for generating an image.

[0002]

2. Description of the Related Art A technique for obtaining digital image data by digitizing an image is an important technique for displaying and transmitting image data in a broadcast, a videophone, a video conference, a computer network system and the like. And
For storage of digital image data, large-capacity storage media such as DVDs are becoming widespread.
Generally, encoded moving image data that has been compressed according to PEG1 or MPEG2 is stored.

[0003] For example, if the medium is a DVD, the data stored in such a medium is reproduced and used by a reproducing apparatus including a DVD player. And
In the case of reproduction using data transmitted in a network system or the like, compared to the case where normal reproduction for displaying an image similar to the original moving image is performed exclusively, in the case of reproduction with such a reproduction device, It often has various special reproduction functions for the convenience of the user. The special playback functions include reverse playback, which plays back in reverse chronological order from the original video, high-speed playback, which plays back faster than the original video, and high-speed, reverse playback. High-speed reverse playback, etc.

For example, in a reproducing apparatus including a DVD player, when reproducing compressed encoded data stored in a DVD, or when performing normal reproduction, the compressed encoded data is sequentially decompressed and decoded and output (display, etc.). ).
On the other hand, when performing high-speed reproduction, it is general to perform high-speed display or the like by outputting (displaying) only a part of the data. The compression-encoded video data is
Generally, processing is performed in units of one screen, that is, one frame. Therefore, in normal reproduction, each frame is sequentially processed and displayed, whereas in high-speed reproduction, some frames are processed. The frame (frame) dropping process of thinning out is performed. For example, if image data composed of 25 frames per second is processed and output only one frame out of 25 frames, the reproduction is performed at 25 times speed.

FIG. 15 is a diagram for explaining normal reproduction and high-speed reproduction of moving image data in units of frames. The illustrated F1 to F13 are images for each frame included in the output image of the normal reproduction, and these are to be reproduced in this order in a time series. When this data is normally reproduced by the reproducing apparatus, the data from F1 to F13 are transmitted in NTSC or P
The signals are sequentially output at a cycle according to a television signal standard such as AL.

[0006] On the other hand, the arrows shown in the figure indicate the order of frame output during quadruple-speed high-speed reproduction. In this case, the F5 frame is output after the F1 frame is output, and then the F9 and F13 frames are output.
By outputting only one frame out of four frames, 4
Double speed high speed reproduction is performed.

[0007] In the case of reverse reproduction, F13
From F1 to F1. In the case of high-speed reverse reproduction, F13, F1
By outputting one frame out of the four frames 9, 9, F5 and F1, high-speed reverse reproduction at a quadruple speed is performed.

[0008] Such a special reproduction function is generally used in a reproduction apparatus as described above. However, even when image data is transmitted, transmission at a low bit rate is required. As in the case of the above-described high-speed reproduction, transmission of data to be output by thinning out frames is also performed.

[0009]

In an image processing apparatus according to the prior art, high-speed reproduction is enabled by thinning out frames and outputting (displaying, transmitting, etc.) as described above. However, in the case of such processing, since the frames to be thinned out are not to be output at all, the information about the motion of the moving image included in the frames to be thinned out is completely lost. Was a problem.

For example, it is assumed that a certain object has a right-to-left movement in frames F2 to F7 of the image data shown in FIG. When the above-described quadruple-speed reproduction is performed on the image data, the right-to-left movement of the object is not transmitted at all depending on the information transmitted only by the frame F5. Therefore,
A user who uses the display result of the high-speed reproduction cannot obtain knowledge about what kind of movement the object has or whether the object has stopped without moving.

In the conventional image processing apparatus, when high-speed reproduction is performed by thinning out frames, the output frames are discontinuous, so that only a display that is uncomfortable to the user is displayed. The point that cannot be obtained was a problem.

Furthermore, in the conventional image processing apparatus, when performing high-speed reproduction by thinning out frames, the user can perform normal reproduction or high-speed reproduction based on only the display result. The problem is that it is not easy to obtain knowledge about
That is, not only when the degree of thinning is small (relatively low speed) but also when the degree of thinning is large (high speed),
Despite the strange display described above,
It has not been possible to easily know whether this is a high-speed reproduction or whether a moving image or the like that produces a special effect is normally reproduced. At the time of transmission at a low bit rate, even when data in which frames are thinned out is transmitted as described above, the same phenomenon occurs when displayed and used at a transmission destination, which is a problem. .

The present invention has been made in view of the above circumstances, and provides an image processing apparatus and an image processing method capable of transmitting information on the movement of an original image even when performing high-speed reproduction. The purpose is to do. Another object of the present invention is to provide an image processing apparatus and an image processing method capable of performing good display with less discomfort due to discontinuous display even when performing high-speed reproduction.

It is another object of the present invention to provide an image processing apparatus and an image processing method capable of easily knowing whether or not high-speed reproduction is being performed on a display screen or the like. Also, the present invention provides an image processing system that transmits motion information even when transmitting data in which frames are decimated as in the case of high-speed reproduction, and which does not cause a sense of incongruity and allows the user to easily know the reproduction state. It is an object to provide an apparatus and an image processing method.

[0015]

In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention includes: an image processing apparatus that converts image data and change information that is information indicating a temporal change of the image data; An image processing apparatus for inputting and generating display image data, comprising: a motion generation unit that performs a weighted addition process of pixels constituting the image data based on the change information to generate display image data. Things.

According to a second aspect of the present invention, there is provided an image processing apparatus comprising:
2. The image processing device according to claim 1, wherein the motion generation unit performs weighted addition processing of pixels constituting the unit image data using change information added to each unit image data in advance. 3. is there.

According to a third aspect of the present invention, there is provided an image processing apparatus comprising:
2. The image processing apparatus according to claim 1, wherein the motion generation unit is a unit of processing unit image data that is smaller than the unit image data, based on a plurality of pieces of change information previously assigned to each unit image data. 3. Is generated, and weighted addition processing is performed on the pixels constituting the processing unit image data using the change information corresponding to the generated processing unit image data.

Further, the image processing apparatus according to claim 4 is:
3. The image processing apparatus according to claim 2, wherein the motion generation unit determines a movement amount and a weighting factor for each pixel from the change information, and performs a processing based on the movement amount. A processing target pixel determining unit that determines a target pixel; and a weighting adding unit that weights and adds the pixel data of the processing target pixel using the weighting coefficient.

According to a fifth aspect of the present invention, there is provided an image processing apparatus comprising:
4. The image processing device according to claim 3, wherein the motion generator corresponds to processing unit image data that is a unit smaller than the unit image data, based on a plurality of pieces of change information attached to each unit image data. 5. A motion vector interpolating unit that generates change information to be processed, a moving amount weighting factor determining unit that determines a moving amount and a weighting factor for each pixel from the change information corresponding to the processing unit image data, and processes based on the moving amount. A processing target pixel determining unit that determines a target pixel; and a weighting adding unit that weights and adds the pixel data of the processing target pixel using the weighting coefficient.

Further, the image processing apparatus according to claim 6 is
4. The image processing apparatus according to claim 1, wherein the motion generation unit acquires change information transition information indicating a temporal change of the change information, and the acquired change information transition information. , And performs weighted addition processing of the pixels constituting the image data to generate display image data.

According to a seventh aspect of the present invention, there is provided an image processing apparatus comprising:
6. The image processing device according to claim 4, wherein the movement amount weight coefficient determination unit includes a change information storage unit that stores input change information, and is stored in the change information storage unit. The moving amount and the weighting factor are determined for each pixel based on a plurality of pieces of change information having different elapsed times.

Further, an image processing apparatus according to claim 8 is
4. The image processing device according to claim 1, further comprising: a reproduction information input unit configured to input reproduction information indicating a display state of the display image data, wherein the motion generation unit includes: The display image data is generated using the reproduction information.

Further, the image processing apparatus according to claim 9 is
4. The image processing apparatus according to claim 1, wherein the change information input is compared with a threshold value, and a threshold determination processing unit generates second change information based on the comparison result. 5. And the motion generation unit generates display image data using the second change information.

An image processing apparatus according to a tenth aspect is the image processing apparatus according to any one of the first to third aspects, wherein the value of the change information to be input is multiplied by a predetermined coefficient. A motion vector processing unit that generates the second change information, wherein the motion generation unit generates display image data using the second change information.

An image processing apparatus according to claim 11 is the image processing apparatus according to any one of claims 1 to 3, further comprising: an area determination unit that determines an area of image data to be processed. Further provisions are made.

According to a twelfth aspect of the present invention, there is provided an image processing method for inputting image data and change information which is information indicating a temporal change of the image data to generate display image data. Then, weighted addition processing of pixels constituting the image data is performed based on the change information to generate display image data.

An image processing method according to a thirteenth aspect is the image processing method according to the twelfth aspect, wherein
Using change information assigned to each unit image data, a weighted addition process is performed on the pixels constituting the unit data.

According to a fourteenth aspect of the present invention, there is provided the image processing method according to the twelfth aspect, wherein the unit image data is based on a plurality of pieces of change information previously assigned to each unit image data. Generating change information corresponding to the processing unit image data, which is a smaller unit, and performing weighted addition processing of the pixels constituting the processing unit image data using the change information corresponding to the generated processing unit image data It is.

According to a fifteenth aspect of the present invention, there is provided the image processing method according to any one of the twelfth to fourteenth aspects, wherein the change information transition information indicating a temporal change of the change information is acquired. Based on the acquired change information transition information, weighting and adding processing of pixels constituting image data is performed to generate display image data.

According to a sixteenth aspect of the present invention, there is provided the image processing method according to any one of the twelfth to fourteenth aspects, wherein reproduction information indicating a display state of the display image data is input. Then, display image data is generated using the reproduction information.

According to a seventeenth aspect of the present invention, in the image processing method according to the twelfth aspect, the change information to be input is compared with a threshold value, and the comparison result is obtained. Is used to generate display image data using the second change information generated based on.

An image processing method according to a twelfth aspect of the present invention is the image processing method according to any one of the twelfth to thirteenth aspects, wherein the image processing method is characterized in that a value of change information to be input is multiplied by a predetermined coefficient. The display image data is generated using the second change information thus obtained.

The image processing method according to the nineteenth aspect is the image processing method according to any one of the twelfth to fourteenth aspects, wherein the area of the image data to be processed is determined, and the determination result is obtained. Is used to generate display image data.

According to a twentieth aspect of the present invention, there is provided a recording medium having a program for causing a computer to execute a weighted addition process of pixels constituting image data based on change information which is information indicating a temporal change of image data. It is recorded.

[0035]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 The image processing device according to the first embodiment of the present invention generates motion information based on a motion vector for each macroblock and adds the motion information to image data. FIG. 1 is a block diagram illustrating an overall configuration of an image processing apparatus according to Embodiment 1 of the present invention. As illustrated, the image processing device according to the first embodiment includes an image decompression unit 101, a first frame memory 102, a motion generation unit 103, and a second frame memory 104.
Digital moving image data compressed and encoded from a storage medium such as a DVD is input as a device input of the image processing device, and display data is output to a display monitor or the like as a device output of the image processing device. Further, the input / output of the image processing apparatus may be transmitted via a network or the like.

The image decompression unit 101 generates decompressed image data by performing decompression decoding processing, which is the reverse processing of compression encoding processing, on coded image data input to the apparatus. The first frame memory 102 includes the image decompression unit 10
1 stores the expanded image data generated. As the first frame memory 102, a storage medium such as a DRAM memory can be used.

The motion generation unit 103 performs a motion generation / addition process for adding information (hereinafter, referred to as motion information) for transmitting the motion of the image to the decompressed image data stored in the first frame memory 102. Generate image data with motion information. The second frame memory 104 stores the motion generation unit 1
03 is a temporary storage medium used as a work area for performing a motion generation / assignment process, and stores image data with motion information generated by the motion generation unit 103 as display image data. As the second frame memory 104, similarly to the first frame memory 102, a storage medium such as a DRAM memory can be used.

The image processing apparatus according to the first embodiment uses a motion vector assigned to each macroblock (described later), which is unit image data that constitutes image data, and uses the motion vector assigned to each macroblock to form the macroblock. Is performed.

FIG. 2 is a block diagram showing the internal configuration of the motion generator 103 shown in FIG. As illustrated, the motion generation unit 103 includes a movement weight coefficient determination unit 201, a first address generation unit 202 and a second address generation unit 203 functioning as a processing target pixel determination unit 205, and a weighting addition unit 204. Weighting and adding section 204
A first multiplier 210, a second multiplier 211, a third multiplier 212, a fourth multiplier 213, and an adder 214 are provided.

The motion generation unit 103 includes an image decompression unit 101
(FIG. 1), motion vectors (X, Y) as change information,
And macroblock identification data corresponding to the motion vector.

The moving amount weight coefficient determining unit 201 receives a motion vector and, based on the motion vector, determines, for each pixel to be processed included in the decompressed image data, a moving amount (dx, dy) used for the motion generation process. , And weight coefficients (w0 to w3). The first address generation unit 202 receives the macroblock identification data, and generates a storage address (adr) for each pixel when the image data is stored in the second frame memory 104 based on the macroblock identification data. The second address generation unit 203 inputs the storage address generated by the first address generation unit 202 and the movement amount (dx, dy) generated by the movement amount weight coefficient determination unit 201, and converts the image data based on these. 4 when reading from the first frame memory 102
The storage addresses (adr0 to adr3) of one pixel are generated.

The weighting and adding unit 204 receives the weighting factors (w0 to w3) generated by the moving amount weighting factor determining unit 201 and reads out from the first frame memory 102 using the address generated by the second address generating unit 203. A weighted addition process is performed on the pixel data of the four pixels (pixels 0 to 3). The weighted addition processing is performed by the first to fourth multipliers 21 included in the weighted addition unit 204.
The weighting process includes a weighting process of multiplying pixel data by a weight amount by 0 to 213, and an adding process in an adder 214 for four weighting results obtained by the multiplication processes.

FIGS. 3 to 7 are diagrams for explaining processing in the image processing apparatus according to the first embodiment. Hereinafter, an operation of the image processing apparatus of the first embodiment when processing image data input from a storage medium or the like will be described with reference to FIGS. 1 and 2 and FIGS. I do.

The coded image data is input from the storage medium to the image decompression unit 101 provided in the image processing apparatus according to the first embodiment. Here, it is assumed that the encoded image data has been encoded according to MPEG1 or MPEG2 which is a general compression encoding standard. In such encoding,
The compression encoding process is performed using a macroblock including a plurality of pixels as a processing unit, and the intra-frame encoding process based on the correlation within one frame (one screen) of digital image data, that is, the spatial correlation is performed. By performing the inter-frame encoding process based on the correlation between the frames of the digital image data that are close in time series, that is, the temporal correlation, a high compression rate can be obtained.

That is, in the intra-frame processing, the macroblock (unit pixel area) to be processed is compared with the time-series adjacent frame in the intra-frame processing, while the pixel itself is processed in the intra-frame processing. In this method, a high compression ratio is realized by detecting how much motion has occurred and generating a motion vector indicating the motion.

According to a general compression encoding processing procedure,
Here, one macroblock is composed of six pixel blocks (consisting of fixed pixels such as 8 × 8 pixels), and four of the six pixel blocks that constitute the macroblock are luminance signals. And two indicate color difference signals. Hereinafter, only the luminance signal to be processed in the image processing apparatus according to the first embodiment will be described with reference to FIG. 3, but the same processing is performed for the color difference signal.

FIG. 3 is a diagram for explaining decompressed image data to be stored in the first frame memory 102 and motion vectors in the image processing apparatus according to the first embodiment. It should be noted that the motion vector here indicates a direction expected to move in the case of reproduction in the forward direction. The same applies to the following unless otherwise specified.

FIG. 3 is a conceptual diagram showing a part of the decompressed image data stored in the first frame memory 102. In this example, the first to fourth four macro blocks 301 to 30 are shown.
4 is stored. Each macro block has a configuration of 16 × 16 pixels.

These macroblocks are obtained by adding motion vectors detected between macroblocks adjacent in time series to each other.
314 is the first to fourth macro blocks 301 to 304
Shows the motion vector added to each.
The motion vectors 311 to 314 added to the first to fourth macroblocks 301 to 304 are extracted at the time of the expansion processing by the image expansion unit 101 (FIG. 1) in the first embodiment, and Is input to the movement amount weight coefficient determination unit 201 (FIG. 2) included in the motion generation unit 103.

Here, 311 to 314 shown in FIG. 3 are vectors indicating the direction of movement from the center of each macroblock 301 to 304 for convenience. Also, since these motion vectors 311 to 314 are in the reproduction time direction, each macroblock 301 to 304 is expected to show a motion in the direction of each motion vector 311 to 314 over time. Become.

The motion generator 103 shown in FIG. 1 targets the macroblocks 301 to 304 (stored in the first frame memory 102) shown in FIG. 3 based on the motion vectors 311 to 314 shown in FIG. , A weighted addition process is performed on the pixels included in each macroblock, and the result is stored in the second frame memory 104 (FIG. 1).

FIG. 4 is a diagram for explaining a method of determining the pixels used in the weight addition processing in the motion generation unit 103. Macro block 301 shown
And the motion vector 311 are equivalent to those in FIG. 3, and the macro block 301 is expected to move in the direction indicated by the motion vector 311. 4 shown in FIG.
01 is a reference pixel, and 402 to 404 are first to third moving pixels. The reference pixel 401 is a pixel to be subjected to the weighted addition processing, and can select an arbitrary pixel in the area of the macro block 301. Further, the position of the reference pixel is indicated by a coordinate position (x0, y0),
This coordinate position is stored in the first address generator 20 as described later.
2 is specified by the address generated.

As described with reference to FIG. 2, in the motion generating section 103, the moving amount weight coefficient determining section 201 uses the moving vector (X, Y) to determine the moving amount ( dx, dy). The moving amount weighting factor determination unit 201 according to the first embodiment generates an (dx, dy) = (X, Y) * k using an arbitrary number k. As shown in FIG. 2, the movement amount generated by the movement weight coefficient determination unit 201 is output to the second address generation unit 203.

On the other hand, as also shown in FIG. 2, macroblock identification data corresponding to the motion vector is input from the image decompression unit 101 to the first address generation unit 202, and the first address generation unit 202 Based on the macroblock identification data, a storage address (adr) of the reference pixel indicating the storage position of the reference pixel 401 in the first frame memory 102 is generated, and the storage address is output to the second address generation unit 203 and It is used to specify the storage position in the two-frame memory 104.

In the second address generation unit 203, the first address is stored based on the storage address (adr) of the reference pixel input from the first address generation unit 202 and the movement amount input from the movement amount weight coefficient determination unit 201. To the storage addresses (ad) of three pixels that specify the storage position of the third moving pixel
r1 to adr3) are generated. These and adr0 (same as adr) indicating the storage position of the reference pixel are storage addresses (adr) indicating the storage positions of the four pixels used in the processing.

The determination of the moving pixel in the motion generator 103 is performed as shown in FIG. 4 as follows. First, the first moving pixel 402 is determined as a pixel at a position moved by the amount indicated by the moving amount based on the coordinate position of the reference pixel 401. That is, from the coordinate position (x0, y0) of the reference pixel 401 and the moving amount (dx, dy), the coordinate position of the first moving pixel 402 is (x1, y1) = (x0, y0) + (dx , dy) * 1 Similarly, the second and third moving pixels 403 and 404 are calculated using the moving amount (dx, dy).
Is (x2, y2) = (x0, y0) + (dx, dy) * 2 (x3, y3) = (x0, y0) + (x2, y2) and (x3, y3), respectively. (dx, dy) * 3

The four storage addresses adr0 to adr3 generated by the second address generator 203 shown in FIG.
Are (x0, y0), (x1, y1), (x2, y2), and (x3, y3), respectively. In FIG. 1, the motion generation unit 103 accesses storage locations designated by these addresses from the decompressed image data stored in the first frame memory 102, and obtains pixel values (pixel0 to pixel3) of four pixels. These are input to the weighting and adding unit 204 as shown in FIG. In addition, as shown in FIG.
Since 04 exists outside the area of the macro block, in such a case, the pixel value of the corresponding pixel is obtained from another macro block.

As shown in FIG. 2, in the motion generation section 103, the movement amount weight coefficient determination section 201 determines weight coefficients (w0 to w3) indicating the contribution of each pixel data used in the processing. Output to the weighting addition section 204. Here, the moving amount weight coefficient determination unit 201 determines that the reference pixel 401
(FIG. 4), and first to third moving pixels 402 to 404
1/2, 1 /
4, 1/8 and 1/8 are generated and output. Here, the weighting factors w0 to w3 give weights by being multiplied by the pixel value (indicating the luminance of each pixel) of each pixel. In the first embodiment, the sum of these is 1 and By doing so, it is possible to prevent the luminance of the image from changing due to the processing.

In the weighting and adding section 204 shown in FIG. 2, the first to fourth multipliers 210 to 213 included therein have the reference pixel 401 (FIG. 4) and the first to third multipliers 210 to 213, respectively.
Weighting factors w0 to w for moving pixels 402 to 404
3 is input. Also, the first to fourth multipliers 210 to 21
3 have pixel values pixel0 to pixel pix of the reference pixel 401 and the first to third moving pixels 402 to 404, respectively.
el3 is input, and the first to fourth multipliers 210 to 213 are input.
Respectively perform a multiplication process on the input weight coefficient and the pixel value, and output the obtained multiplication result to the adder 214. The adder 214 performs an addition process on the four multiplication results, generates a pixel value (pixel) subjected to the motion generation / addition process, and outputs the pixel value to the second frame memory 104 (FIG. 1).

As described above, the address (a) generated by the first address generation unit 202 (FIG. 2) of the motion generation unit 103
Since dr) is also used to specify the storage position in the second frame memory 104, the storage positions (adr),
That is, the reference pixel 401 in the first frame memory
The processed pixel value (pixel) is stored in a storage position corresponding to the storage position of (FIG. 4).

The above processing is performed using all the pixels included in the macro block 301 shown in FIG. 3 as reference pixels, so that the processed pixels corresponding to all the pixels included in the macro block 301 become the second pixels. It will be stored in the frame memory 104 (FIG. 1). Further, FIG.
The same processing is performed on the other macroblocks 302 and the like shown in (1), and the motion-generated and added data corresponding to all the image data stored in the first frame memory 102 is stored in the second frame memory 104. Will be stored. The image data stored in the second frame memory 104 is displayed or the like as a device output of the image processing device. If a subsequent device input from a storage medium or the like is made to the image processing device, the image decompression unit The processing after step 101 is repeated.

FIG. 5 is a diagram for explaining the motion generation / assignment process performed by the image processing apparatus according to the first embodiment performed as described above, and FIG. 6 is a conceptual diagram showing the effect of the process. With reference to these, the motion generation / assignment processing according to the first embodiment will be further described.

FIG. 5 shows the coordinate positions (shown in two dimensions) shown in FIG. 4 as one-dimensional horizontal coordinates and the pixel values in the vertical coordinates in order to simplify the description of the weighted addition processing. FIG. 7A shows a state before the weighting addition processing, in which a processing target pixel having a coordinate position 520 (corresponding to the reference pixel 401 in FIG. 4) has a pixel value pv. In addition, a pixel having coordinate positions 521 to 523 (corresponding to moving pixels 402 to 404 in FIG. 4).
Have a pixel value of 0. The vector 501 in the figure is the motion vector 311 in FIGS.
, And is a vector that one-dimensionally indicates the direction of motion.

FIG. 5B shows the weighting factors w0 to w shown in FIG.
3 is shown. In the figure, horizontal coordinates are shown in FIG.
Indicates the same one-dimensional coordinate position, and the vertical coordinate indicates the value of the weight coefficient. As shown, a coordinate position 520 (corresponding to the reference pixel 401 in FIG. 4),
And 521 to 523 (moving pixels 402 to 404 in FIG. 4)
Is equivalent to ) Have a weight coefficient of 1
/ 2, 1/4, 1/8, 1/8.

The weighted addition process is performed on the pixel values of the pixels at the coordinate positions 520 to 523 shown in FIG.
Are multiplied by the weighting coefficients for the coordinate positions 520 to 523 shown in FIG. 7, and the sum of the multiplication results is the processed pixel value. By performing such processing for all the pixels indicated by the horizontal coordinates in FIG. 5A using the weighting coefficients in FIG. 5B, the motion generation / assignment processing in the first embodiment is performed. Done.

When the pixel 520 is used as a moving pixel in the processing of the pixels 551 and 552 in the figure, the weight set to the pixel value pv of the pixel 520 is used. For example, the pixel 551
And 552 both have a pixel value of 0, and when the moving pixels 521 to 523 other than the pixel 520 also have a pixel value of 0, in the process using the pixel 551 as a reference pixel,
If the pixel 520 is used as the second moving pixel, the pixel 551 has the pixel value pv / 4. If the pixel 520 is used as the first moving pixel in the process using the pixel 552 as the reference pixel, the pixel 552 becomes The pixel value is pv / 2.

FIGS. 5C and 5D are diagrams showing the results of such motion generation / addition processing. In these figures,
As in FIG. 3A, the horizontal coordinate indicates a one-dimensional coordinate position, and the vertical coordinate indicates a pixel value. FIG. 13C shows a state in which a pixel having a pixel value exists only in a narrow area denoted by reference numeral 750, and reference numeral 751 denotes a vector indicating the direction of movement as in FIG. For this state,
In the state shown in FIG. 11D obtained by performing the weighting addition processing (motion generation / assignment processing), the motion information is added in the direction indicated by the vector 751 as compared with FIG. I have.

FIG. 6 is a conceptual diagram showing the effect when displaying the device output of the image processing device of the first embodiment. FIG. 12A shows a display state when the motion generation / addition processing is not performed, and FIG. 6B shows a display state when the motion generation / addition processing is performed.In FIG. A display result equivalent to integrating the image in the time direction is obtained. Therefore, when the motion generation / addition processing is performed by the image processing apparatus according to the first embodiment, compared with the image processing apparatus according to the related art which can only display as shown in FIG. A display with a feeling of movement as shown in FIG.
It is possible to provide information about movement.

As described above, according to the image processing apparatus of the first embodiment, the image decompression unit 101 and the first frame memory 1
02, the motion generation unit 103, and the second frame memory 10
4, the motion generation unit 103 performs a motion generation and addition process on the decompressed image data stored in the first frame memory 102 using the motion vector acquired from the image decompression unit 101, and the result of the process The obtained image data is
Since the image is stored in the frame memory 104, the image displayed as a device output of the image processing device is accompanied by motion information, and a display result having a feeling of motion can be obtained. Therefore, it is possible to perform display with less discomfort than the image processing apparatus according to the related art, and it is possible for the user to easily know the reproduction state by adding the motion information. . Further, even when the device input in the image processing device of the first embodiment is image data transmitted at a low bit rate by thinning out frames, similar effects can be obtained by performing the same processing.

In the first embodiment, the motion generator 1
Regarding the process 03, as shown in FIG. 4, the determination of the moving pixel with respect to the reference pixel is performed by performing linear processing using the moving amount, that is, by multiplying the moving amount by 1, 2, and 3 to determine the moving pixel. But this is only an example,
It is also possible to use a numerical value other than these to perform by linear processing, or to perform by non-linear processing, and perform the same motion generation / assignment processing to obtain the above effects. .

FIG. 7 is a diagram for explaining a case where nonlinear processing is performed in determining a moving pixel. In the figure, a macro block 301 and a motion vector 311 are:
3 and FIG. 4 are shown. The reference pixel 401 is a pixel to be processed selected in the same manner as in the first embodiment shown in FIG.

In the case shown in FIG. 7, the first to third moving pixels 402 to 404 have coordinate positions (x1 ', y1') and (x2 ', y
2 ') and (x3', y3 '), which are (x1', y1 ') = (x0, y0) + (dx, dy) * 1 (x2', y2 ') = (x0, y0) + (dx, dy) * 4 (x3 ', y3') = (x0, y0) + (dx, dy) * 8 Even when a moving pixel determined by such non-linear processing is used, it is possible to perform the same motion generation / assignment processing as in the first embodiment.

In the first embodiment, the motion generation unit 1
03 (FIG. 1) includes a moving amount weighting factor determination unit 201 (FIG. 2) that determines the weighting factors w0 to w3 as 、, １ ／,
Although 1/8 and 1/8 are generated and output,
This numerical value is also an example, and processing using another coefficient can be performed by setting. Further, as described above, in the first embodiment, a change in luminance (similarly in the case of color difference) is prevented by making the sum of the weighting factors equal to 1, but
If the luminance is to be changed when a special effect is desired, it is possible to use a coefficient whose sum does not become 1 depending on the setting.

Embodiment 2 The image processing device according to the second embodiment of the present invention generates motion information based on a motion vector and adds the motion information to image data, as in the first embodiment. The motion information is generated based on the reproduction state and added to the image data.

FIG. 8 is a block diagram showing an overall configuration of an image processing apparatus according to the second embodiment of the present invention. As illustrated, the image processing apparatus according to the second embodiment includes an image decompression unit 101, a first frame memory 102, a motion generation unit 80
3, a second frame memory 104, and a reproduction information input unit 805. The reproduction information input unit 805 is added to the overall configuration of the image processing apparatus according to the first embodiment (FIG. 1). Then, similarly to the image processing apparatus according to the first embodiment, digital moving image data compression-encoded from a storage medium such as a DVD is input as an apparatus input of the image processing apparatus, and is output as an apparatus output of the image processing apparatus. The display data is output to a display monitor or the like.

The reproduction information input section 805 inputs reproduction information used by the image processing apparatus from the outside. Here, the reproduction information is information instructing a reproduction direction, a reproduction speed, and the like. Note that the image decompression unit 101, the first frame memory 102, and the second frame memory 104 are the same as those in the above-described first embodiment, and thus are assigned the same numbers and descriptions thereof are omitted.

The motion generation unit 803 has the internal configuration shown in FIG. 2 and performs the motion generation / addition processing, similarly to the case 103 in the first embodiment. In the generation unit 803, the configuration and operation of the moving amount weight coefficient determination unit 201 are different from those of the first embodiment.

FIG. 9 is a diagram showing an internal configuration of the moving amount weight coefficient determining unit 201 provided in the image processing apparatus according to the second embodiment. As shown, the moving amount weight coefficient determining unit 201
Are first to fourth motion vector storage units 901 to 904 functioning as a change information storage unit;
05. The first to fourth motion vector storage units 901 to 904 functioning as change information storage units store motion vectors as change information input from the image decompression unit 101 (FIG. 8). The first to fourth motion vector storage units 901 to 904 store motion vectors of image data corresponding to different times.

The motion vector determination unit 905 responds to the reproduction information input from the reproduction information input unit 805 (FIG. 8),
From the motion vectors stored in the first to fourth motion vector storage units 901 to 904, change information transition information indicating a temporal change of the motion vector is acquired, and the movement amount and the movement amount are determined based on the acquired change information transition information. Generate a weighting factor. That is, in the first embodiment, processing is performed based on a motion vector at a single time, whereas in the second embodiment, processing is performed based on a plurality of motion vectors at different times. It is.

As described above, in the second embodiment, the motion generating unit 803 generates display image data based on change information transition information that is a temporal change of a motion vector input as change information. . A reproduction information input unit 805 is provided as reproduction information input means for inputting reproduction information for designating a display state of image data, and generates display image data corresponding to the reproduction information, as described later.

FIGS. 10 to 12 are diagrams for explaining the processing in the image processing apparatus according to the second embodiment. Hereinafter, FIGS. 10 to 1 will be described with reference to FIGS. 8 and 9.
2, the operation when the image processing apparatus of the second embodiment processes image data input from a storage medium or the like will be described.

Encoded image data is input from a storage medium to an image decompression unit 101 provided in the image processing apparatus according to the second embodiment. It is assumed that the coded image data is generated by compression coding including general inter-frame processing as in the first embodiment, and is provided with a motion vector. The image decompression unit 101 according to the first embodiment
In the same manner as the image decompression unit 101, the decompression and decompression processing which is the reverse of the compression encoding processing is performed, and the obtained decompressed image data is output to the first frame memory 102, and the motion vector is output to the motion generation unit 803. I do.

The motion generator 803 stores the input motion vector in one of the motion vector storages. When the image decompression unit 101 processes the subsequent encoded data and outputs a motion vector to the motion generation unit 803, the motion vector is stored in a motion vector storage unit different from the one in which the motion vector was stored in the previous stage. Is stored. Accordingly, the first to fourth motion vector storage units 901 to 904 included in the motion generation unit 803 store motion vectors corresponding to image data at different times. Here, time (T-3) (T-2), (T-
1) and the motion vector of T are stored.

In the second embodiment, information indicating a change in a motion vector is acquired by holding a plurality of motion vectors at different times in this way, and motion information having a temporal width is added. . FIG. 10 is a diagram for explaining a change in a motion vector. In the figure, 10
01 to 1004 are time (T-3), (T-2), (T-1),
FIG. 6 shows the motion vectors of the specific macroblock and the magnitude (scalar value) of the motion vector over time. These times are arranged in chronological order in this order, and the motion vector takes a transition state as shown.

In the motion generator 803 of the second embodiment, these four motion vectors are stored in first to fourth motion vector storages 901 to 904. Then, the motion vector determination unit 905 calculates the motion vector from the motion vectors stored in the first to fourth motion vector storage units 901 to 904.
Change information transition information indicating a state of change of a motion vector along a time series is acquired, and a movement amount and a weight coefficient are generated in accordance with the acquired change information transition information.

FIG. 11 is a diagram for explaining selection of a weight coefficient corresponding to a change state of a motion vector according to the second embodiment. The figure shows the relationship between the amount of movement and the weighting factor. The horizontal coordinates in the figures (a) to (d) show the pixel coordinates (moving amount), and the vertical coordinates show the value of the weighting coefficient. . The motion vector determining unit 905 determines a weight coefficient using a function indicating one of the relationships shown in FIGS. 11A to 11D according to the state of change of the motion vector. The determination is made based on the magnitude of the motion vector and the magnitude of the change in the motion vector.

When the motion vector is large, it indicates that the motion of the image to be processed is large. Therefore, when generating motion information, it is desirable to use a wider range of pixels in the weighted addition process. Becomes Therefore, the determination is performed using a function indicating the relationship of FIG. 11B or FIG. 11D in which the weighting factor is set for a wide range of pixel coordinates. On the other hand, when the motion vector is small, that is, when the motion of the image is small, it is desirable to use a relatively narrow range of pixels for the weighting addition process in generating the motion information, It is assumed that the determination is performed using a function indicating the relationship between FIG. 11A and FIG. 11C in which a weight coefficient is set for the pixel coordinates.

When the change of the motion vector is gradual, it is desirable to make the influence of the pixels near the processing target (reference pixel) large, so that such a pixel is weighted as shown in FIG. Alternatively, the determination is made using a function indicating the relationship shown in FIG. On the other hand, when the change of the motion vector is drastic, it is desirable to consider the influence of the pixel at a position relatively distant from the reference pixel. The decision is made using a function indicating the relationship

As described above, when the motion vector is small and the change of the motion is drastic, the function indicating the relationship shown in FIG. 11A is used. When the motion vector is large and the change of the motion is drastic, the function shown in FIG.
If the motion vector is small and the change in motion is gradual, the function indicating the relationship in (b) is the function indicating the relationship in FIG. In this case, a function indicating the relationship shown in FIG. This makes it possible to perform weighted addition processing appropriately corresponding to the state of the image.

In the image processing apparatus according to the second embodiment, FIG.
From the reproduction information input unit 805, the reproduction information indicating the reproduction speed and the reproduction direction, that is, whether the reproduction in the forward direction or the reproduction in the reverse direction is performed, is included in the movement amount included in the motion generation unit 803. The reproduced information is input to the weight coefficient determining unit 201 (FIG. 2), and is input to the motion vector determining unit 905 inside the moving amount weight coefficient determining unit 201, and the moving amount and weight are added together with the information on the motion vector as described above. It is used to determine the coefficient.

Among the reproduction information, the information indicating the reproduction speed is handled in the same manner as the magnitude of the motion vector. That is, when the reproduction speed is high, as in the case where the motion vector is large, a wide range of pixels are used for processing, so that the functions indicating the relations of FIG. 11B and FIG. Used. On the other hand, when the reproduction speed is low, as in the case where the motion vector is small, pixels in a relatively narrow range are used for processing.
(a) and a function indicating the relationship of FIG. 11 (c) are used.

The information indicating the reproduction direction among the reproduction information is used as follows. FIG. 12 is similar to FIG.
It is a figure for explaining selection of a weight coefficient corresponding to a reproduction direction, and is a figure showing a relation between a movement amount and a weight coefficient. The horizontal and vertical coordinates are the same as in FIG. When the reproduction is performed in the forward direction, that is, when the reproduction is performed in a time series, a function indicating a relationship in which a weight coefficient is set in the forward direction (direction of motion) shown in FIG. 12A is used. On the other hand, when the reproduction is performed in the reverse direction, that is, when the reproduction is performed in the direction opposite to the time series, the function indicating the relationship in which the weight coefficient is set in the backward direction (the direction opposite to the motion) shown in FIG. Is used.

As described above, using the function selected adaptively, the moving amount and the weighting coefficient are calculated by the moving amount weight determining section 20.
1 (FIG. 2), the subsequent processing is executed in the same manner as in the first embodiment, and the second frame memory 10
4 (FIG. 8) stores the image data subjected to the motion generation / addition processing. Therefore, as in the case of the first embodiment, this image data accompanies motion information, and a display result having a feeling of movement can be obtained. The assigned motion information is more suitable for the state of the image than in the first embodiment.

As described above, according to the image processing apparatus of the second embodiment, the image decompression unit 101 and the first frame memory 1
02, the motion generator 803, the second frame memory 104,
And a reproduction information input unit 805, and the motion generation unit 803 performs a motion generation / addition process on the decompressed image data stored in the first frame memory 102 using the motion vector acquired from the image decompression unit 101. Since the image data obtained as a result of the processing is stored in the second frame memory 104, an image displayed as a device output of the image processing device is accompanied by motion information, and a display result having a feeling of movement can be obtained. It becomes possible. Then, the movement amount weight coefficient determination unit 201 included in the motion generation unit 803 includes first to fourth motion vector storage units 901 to 904, stores motion vectors at different times, and includes a motion vector determination unit 905. Thus, since the movement amount and the weight coefficient corresponding to the state of the motion vector and the reproduction information are determined, it is possible to add motion information more appropriately corresponding to the state of the image. Also, in the image processing apparatus according to the second embodiment, when image data transmitted at a low bit rate by thinning out frames is input to the apparatus, the same effect can be obtained by performing the same processing.

In the second embodiment, the moving amount weighting coefficient determining unit 201 includes the first to fourth motion vector storing units 90.
1 to 904, and the motion vector determination unit 905 performs processing based on the four motion vectors.
The present invention is not limited to this, and it is possible to reduce the circuit scale and simplify the processing as a smaller number, or to perform more appropriate processing as a larger number.

Embodiment 3 The image processing apparatus according to the third embodiment of the present invention generates motion information based on a motion vector and adds it to image data, as in the first embodiment. Is generated based on the motion information, and is added to the image data.

The overall configuration of the image processing apparatus according to the third embodiment is the same as that of the first embodiment, and FIG. 1 is used for the description. Further, the image processing apparatus according to the third embodiment also includes:
Like the image processing apparatus according to the first embodiment, the image processing apparatus includes the motion generation unit 103 having the internal configuration illustrated in FIG. 2 and performs the motion generation / assignment process. , A moving amount weighting coefficient determining unit 201, a first address generating unit 202 and a second address generating unit 203 functioning as a processing target pixel determining unit, and a weighting adding unit 20.
4 and a motion vector complementing unit 1301. Note that the moving amount weighting factor determining unit 201, the first address generating unit 202, the second address generating unit 203, and the weighting adding unit 204 are the same as those in the above-described first embodiment, and thus are assigned the same numbers. And the description is omitted.

FIG. 13 is a diagram showing the internal configuration of the motion generation unit 103 provided in the image processing apparatus according to the third embodiment.
As shown in the figure, the motion vector interpolation unit 1301
Fourth multipliers 1310 to 1313, fifth to eighth multipliers 1320 to 1323, and an adder 1330 are provided. Here, the fifth vector included in the motion vector interpolation unit 1301
To eighth multipliers 1320 to 1323 and adder 1330
Are a multiplier for vector operation and an adder, and can perform two sets of multiplication and addition processing.

The motion vector interpolation unit 1301 performs an interpolation process based on the four motion vectors and the four pieces of position information, and generates a motion vector corresponding to the processing unit image data to be processed. In the interpolation processing according to the third embodiment described below, a case will be described in which the processing unit image data is pixel data and a motion vector corresponding to a pixel to be processed is generated.

That is, the motion generation unit 103 included in the image processing apparatus according to the third embodiment determines a pixel corresponding to a processing unit image data to be processed based on a plurality of motion vectors input as change information. Then, a motion vector for each pixel as unit change information is generated, and display image data is generated based on the generated motion vector.

FIG. 14 is a diagram for explaining processing by the motion vector interpolation unit 1301 according to the third embodiment.
FIG. 11A shows the first frame memory 102 as in FIG.
FIG. 3 is a diagram showing decompressed image data stored in FIG. 1 and motion vectors in the image data;
Is a macro block composed of 16 × 16 pixels as in FIG. 3, and 311 to 314 are vectors indicating the direction of motion from the center of each of the macro blocks 301 to 304 as in FIG. Reference numeral 1401 in FIG. 14 denotes an interpolation target pixel, which is a processing target pixel for generating a motion vector by interpolation processing. 1402 denotes macro blocks 301 to 3
This area is obtained by surrounding each center point of No. 04 and includes an interpolation target pixel 1401, and is an interpolation area used for interpolation processing. Reference numeral 1403 denotes an interpolation reference pixel serving as a reference point set in the interpolation area 1402. Here, in the interpolation area 1402, a pixel closest to the origin shown in FIG. 1404 is
This is a motion vector generated for the interpolation target pixel by the interpolation processing.

FIG. 14B shows the state of the interpolation area 1402.
FIG. 9 is a diagram for describing pixel position information indicating a positional relationship between an interpolation target pixel 1401 and an interpolation reference pixel 1403. Here, the coordinates of the interpolation target pixel 1401 are (x, y), and the coordinates of the interpolation reference pixel 1403 are (x0, y0). The pixel position information (p, q, 1-p, 1-q) shown in FIG. 6 (b) is p = (x-x0) / 16 q = (y-y0) / 16 1-p = 1- ( x-x0) / 16 1-q = 1- (y-y0) / 16.

The motion vector interpolation unit 13 according to the third embodiment
01 is an interpolation process for generating a motion vector 1404 for the interpolation target pixel 1403 from the motion vector 311 of the macroblock and the motion vectors 312 to 314 of macroblocks located in the vicinity based on the pixel position information. Is what you do.

Here, the motion vectors 311 to 314 are
Assuming that (X0, Y0), (X1, Y1), (X2, Y2), (X3, Y3), the motion vector 14
04 (X, Y) is generated according to the following equation. (X, Y) = (1-p) * (1-q) * (X0, Y0) + (1-p) * q * (X
3, Y3) + p * (1-q) * (X1, Y1) + p * q * (X2,
Y2)

Therefore, the motion vector interpolation unit 1301
First to fourth multipliers 1310 to 1313 included therein
And the fifth to eighth multipliers 1320 to 1323 and the adder 1330 perform the above-mentioned arithmetic processing to generate a motion vector 1404 (X, Y).

First, in the motion vector interpolation section 1301, p, q of each term constituting the pixel position information (p, q, 1-p, 1-q) are obtained.
q, 1-p, 1-q are input to first to fourth multipliers 1310 to 1313 as shown in FIG. That is, the first
The term p is the first and second multipliers 1310 and 1310
11, the second term q is the first and third multiplier 131
0 and 1312, the third term 1-p is the third and fourth
Are input to the multipliers 1312 and 1313, and the fourth term 1-q is input to the second and fourth multipliers 1311 and 1313. The respective multipliers multiply the input terms. As a result, the first to fourth multipliers 1310 to 1310-1
At 313, p * q, p * (1-q), (1-p) * q, (1-p) * (1-
q) are generated, and the first to fourth multipliers 1310 to 131 are generated.
3 outputs these multiplication results to the fifth to eighth multipliers 1320 to 1323 as shown in FIG. Therefore, the fifth multiplier 1320 stores the multiplication result (1-p) * (1-q) in the sixth multiplier 1320.
Of the multiplication result (1-p) * q in the multiplier 1321, the multiplication result p * (1-q) in the seventh multiplier 1322, and the multiplication result p * q in the eighth multiplier 1323. Is entered.

On the other hand, motion vectors 311 to 314 are input to a motion vector interpolation unit 1301, and these motion vectors are input to fifth to eighth multipliers 1320 to 1323 as shown in FIG. . That is, the motion vector 3 of (X0, Y0) is added to the fifth multiplier 1320.
11, the sixth multiplier 1321 has a motion vector 312 of (X1, Y1), and the seventh multiplier 1322 has a motion vector 312 (X2, Y2).
, And the eighth multiplier 13
A motion vector 314 of (X3, Y3) is input to 23.

Fifth to eighth multipliers 1320 to 1323
Is a multiplier for vector operation as described above, and performs two sets of multiplication processing of the input multiplication result and the motion vector. For example, the fifth multiplier 1320 calculates (1-p) * (1
A multiplication process of -q) * X0 and a multiplication process of (1-p) * (1-q) * Y0 are performed. Therefore, the motion vector 1404 (X,
Y), the first to fourth terms of the arithmetic expression
To the eighth multipliers 1320 to 1323, respectively. The fifth to eighth multipliers 1320 to 1323 output the generated multiplication results to the adder 1330. Adder 1 as described above
Reference numeral 330 denotes an adder for vector operation, which performs two sets of addition processing. The addition processing of these multiplication results in the adder 1330 generates the motion vector (X, Y). As shown in FIG.

If the motion vector 1404 corresponding to the pixel as the processing unit image data, that is, the interpolation target pixel 1401 shown in FIG. 14, is output to the moving amount weight coefficient determining unit 201, the moving amount weight coefficient determining unit 201 A movement amount and a weight coefficient are generated based on the motion vector. Subsequent processing is executed in the same manner as in the first embodiment, motion information is generated, and the added image data is stored in the second frame memory 10.
4 (FIG. 1) and displayed as a device output of the image processing device.

In a compression encoding method such as MPEG1 or MPEG2, a motion vector is generally added in units of macroblocks as shown in the first and third embodiments. Also, in object coding that performs coding for each object, a motion vector may be given in units of objects. However, an encoding method that generates encoded data to which a motion vector is assigned for each pixel is not general. Therefore, in the first embodiment, when processing a pixel included in a certain macroblock, a motion generation / assignment process is performed using a motion vector assigned to the macroblock. In the case of the third embodiment, a motion vector for each pixel is generated using a motion vector assigned in units of macroblocks, and the motion generation / assignment process is performed based on the motion vector. Although accompanied, it is possible to provide more accurate motion information.

As described above, according to the image processing apparatus of the third embodiment, the image decompression unit 101 and the first frame memory 1
02, the motion generator 103, the second frame memory 104,
And a reproduction information input unit 105, wherein the motion generation unit 103 performs a motion generation / attachment process on the decompressed image data stored in the first frame memory 102 using the motion vector acquired from the image decompression unit 101. Since the image data obtained as a result of the processing is stored in the second frame memory 104, an image displayed as a device output of the image processing device is accompanied by motion information, and a display result having a feeling of movement can be obtained. It becomes possible. Then, the moving amount weighting factor determining unit 201 included in the motion generating unit 103 includes a motion vector interpolating unit 1301 to generate a motion vector by an interpolation process for each pixel, and based on the generated motion vector, a moving amount and a weight. Since the coefficient is determined, it is possible to add motion information more appropriately corresponding to the state of the image. Further, in the image processing apparatus according to the third embodiment, when image data transmitted at a low bit rate by thinning out frames is input to the apparatus, the same effect can be obtained by performing the same processing.

In the third embodiment, as described with reference to FIG. 14, a motion vector is generated by interpolation using motion vectors assigned to four macroblocks. However, the present invention is not limited to this. However, the accuracy can be further improved by using the motion vectors assigned to, for example, nine or sixteen macroblocks.

In the third embodiment, a plurality (four)
Although the motion vector of each pixel is generated by performing linear processing on the motion vector, the nonlinear interpolation process may be performed. For example, when the motion vectors assigned to a larger number of macroblocks are used as described above, some of the motion vectors are multiplied by a set coefficient in the motion vector interpolation unit 1301 so that adjacent macroblocks are multiplied. It is possible to increase the contribution degree of the motion vector.

Further, in the third embodiment of the present invention, the description has been given of the case where the processing unit image data is used as the pixel data to obtain and process the motion vector for each pixel. However, the unit is smaller than the image data to which the motion vector is added. Any motion vector may be obtained and processed for each processing unit image data.For example, if a motion vector is provided for each object, a motion vector for each macro block constituting the object is obtained. And motion information generated using the obtained motion vector.

In the third embodiment, the description has been given of the case where the motion vector is obtained and processed for each processing unit image data which is a unit smaller than the image data with the motion vector. The motion vector of the processing unit image data, which is a unit larger than the image data, may be obtained by inverse interpolation and processed. In this case, although the accuracy of the motion information is reduced, the processing load on the motion generation unit 103 can be reduced.

Embodiment 4 The image processing apparatus according to the fourth embodiment of the present invention generates motion information based on a motion vector and adds the generated motion information to image data, as in the first embodiment. The comparison is performed, motion information is generated based on the comparison result, and the motion information is added to the image data.

FIG. 16 is a block diagram showing the overall configuration of the image processing apparatus according to the fourth embodiment. As shown, the image processing apparatus according to the fourth embodiment includes an image decompression unit 10.
1. First frame memory 102, motion generation unit 103, second frame memory 104, and threshold determination processing unit 160
1 and has a configuration in which a threshold determination processing unit 1601 is added to the overall configuration of the image processing apparatus according to the first embodiment (FIG. 1). In the image processing apparatus according to the fourth embodiment, the image decompression unit 101 and the first frame memory 10
2, the motion generation unit 103, and the second frame memory 10
4 is the same as Embodiment 1 described above, and therefore, is denoted by the same reference numeral and description thereof is omitted.

The threshold value judgment processing section 1601 includes the image decompression section 1
01 and the macroblock identification data corresponding to the motion vector,
A second motion vector (x ′, y ′) for image data generated by the motion generation unit 103 and macroblock identification data corresponding to the motion vector are generated and output.

That is, when the motion vector (x, y) is input from the image decompression unit 101, the threshold value judgment processing unit 1601 according to the fourth embodiment receives the motion vector (x, y).
And a predetermined threshold value. At this time, if the motion vector (x, y) is equal to or smaller than the threshold value, it is determined that the current macroblock has no motion, and (x ′, y ′) = (0, 0) is output as the second motion vector. On the other hand, if the motion vector (x, y) is larger than the threshold value, it is assumed that there is a motion in the target macroblock, and (x ′, y ′) = (x, y) is output as the second motion vector.

Thus, the motion generation unit 103 uses the second motion vector (x ′, y ′) output from the threshold determination processing unit 1601 to convert the image to which the motion component has been added into the second
The image is generated in the frame memory 104, and the second frame memory 104 can generate an image in which a motion component is not added to a macroblock or an object having a small motion vector. In other words, if the image stored in the storage medium contains camera shake or the like,
Erroneous use of the camera shake image as image movement can be eliminated.

As described above, according to the fourth embodiment, by providing the threshold value determination processing unit 1602, when the size of the motion vector is equal to or smaller than the threshold value, the motion vector is not added to the image. Becomes possible. Therefore, in addition to the effects of the above-described first embodiment, when an image or an object having little camera shake or a motion vector having a very small motion vector is included,
Since no motion information is added, it is possible to prevent the camera shake image from being mistaken as a motion of the image.

As a result, it is possible to realize an image processing apparatus capable of transmitting an original motion of an image without giving unnecessary motion information to the image and obtaining a good image with less discomfort and discomfort. it can.

When (x ′, y ′) = (0,0) is input as the second motion vector to the motion generation unit 103, the motion generation unit 103 does not need to perform the weighting addition process. The pixels stored in the first frame memory 102 may be stored in the second frame memory 104 as they are. Therefore, the processing amount of the weighting and adding unit 204 that performs the weighting and adding process in the motion generating unit 103 can be reduced, and the processing speed can be improved.

The image processing apparatus according to the fourth embodiment of the present invention has been described by adding the threshold value determination processing unit 1601 to the image processing apparatus according to the first embodiment. The same effect can be obtained even when the image processing apparatus according to the third embodiment is added with the threshold value determination processing unit 1601.

Although the threshold value determination processing section 1601 in the image processing apparatus according to the fourth embodiment determines whether or not to add a motion component based on whether or not input change information is larger than the threshold value, it has been described. However, the present invention is not limited to this, and any method may be used as long as different processing is performed on the input motion vector based on the comparison result of whether or not the input change information is larger than the threshold.

Embodiment 5 FIG. The image processing apparatus according to the fifth embodiment of the present invention generates motion information based on a motion vector and adds the generated motion information to image data, as in the first embodiment. The motion information is generated by multiplying by a coefficient, and is added to the image data.

FIG. 17 is a block diagram showing the overall configuration of the image processing apparatus according to the fifth embodiment. As shown, the image processing apparatus according to the fifth embodiment includes an image
1, first frame memory 102, motion generation unit 103, second frame memory 104, and motion vector processing unit 1
The image processing apparatus according to the first embodiment has a configuration in which a motion vector processing unit 1701 is added to the entire configuration (FIG. 1) of the image processing apparatus according to the first embodiment. Note that the image decompression unit 101, the first frame memory 102, the motion generation unit 103, and the second frame memory 104 in the image processing device according to the fifth embodiment are the same as those in the first embodiment, and are therefore the same. And the description is omitted.

The motion vector processing section 1701 receives the motion vector (x, y) from the image decompression section 101 and the macroblock identification data corresponding to the motion vector as inputs, and outputs the image data generated by the motion generation section 103. And a second motion vector (x ', y') for generating the macro block identification data corresponding to the motion vector.

That is, the motion vector amount processing section 1701 according to the fifth embodiment multiplies an input motion vector (x, y) by a predetermined coefficient to obtain a second motion vector (x ′, y ′). Is output to the motion generation unit 103. The motion generation unit 103 generates the second motion vector (x ′,
y ′) is used to generate display image data.

FIG. 18A shows the relationship between the magnitude of the input motion vector (x, y) and the magnitude of the second motion vector (x ', y') multiplied by a constant coefficient. It is. In FIG. 18A, a broken line 1801 shows the relationship between the magnitudes of the input and output vectors when no coefficient is multiplied. A solid line 1802 shows the relationship between the input and output vectors when multiplied by a coefficient greater than 1.
A solid line 1803 shows the relationship between the magnitudes of the input and output vectors when multiplied by a coefficient smaller than one.

For example, when multiplied by a coefficient larger than 1 shown by a solid line 1802, when the coefficient is not multiplied (broken line 1)
801), the output second motion vector (x ′,
y ') increases. That is, the motion generation unit 103
Accordingly, when motion information is given to an image using the second motion vector (x ′, y ′), the image generated in the second frame memory 104 has a higher image quality than when no coefficient is multiplied. An image in which the movement is emphasized.

On the other hand, when multiplied by a coefficient smaller than 1 shown by a solid line 1803, when the coefficient is not multiplied (broken line 183).
01), the output second motion vector (x ′,
y ') becomes smaller. That is, the motion generation unit 103
Accordingly, when motion information is given to an image using the second motion vector (x ′, y ′), the image generated in the second frame memory 104 has a higher image quality than when no coefficient is multiplied. An image with reduced movement is obtained.

FIG. 18B shows an example of the relationship between the magnitude of the input and output motion vectors when the coefficient to be multiplied is changed according to the magnitude of the input motion vector (x, y). FIG.

In FIG. 18B, a broken line 1801 is
It shows the relationship between the magnitudes of input and output vectors when no coefficient is multiplied. A solid line 1804 indicates the relationship between the magnitudes of the input and output vectors when the coefficient multiplied by the magnitude of the input motion vector (x, y) is gradually increased from 1, and a solid line 1805 indicates the magnitude of the input motion vector (x, y). 3 shows the relationship between the magnitudes of the input and output vectors when the coefficient multiplied by the magnitude of the motion vector (x, y) is gradually reduced from 1.

For example, the coefficient to be multiplied by the magnitude of the input motion vector (x, y) shown by a solid line 1802 is 1
, The value of the output second motion vector (x ′, y ′) becomes larger than when the coefficient is not multiplied (broken line 1801). That is, the second motion vector (x ′, y ′) is generated by the motion generation unit 103.
When the motion information is given to the image by using, the image generated in the second frame memory 104 is an image in which the motion is emphasized as compared with a case where the coefficient is not multiplied.

On the other hand, when the coefficient to be multiplied by the magnitude of the input motion vector (x, y) is gradually reduced from 1 as shown by the solid line 1803, the output is smaller than when the coefficient is not multiplied (broken line 1801). The value of the second motion vector (x ′, y ′) to be obtained becomes smaller. That is, when motion information is given to an image using the second motion vector (x ′, y ′) by the motion generation unit 103, the image generated in the second frame memory 104 is multiplied by a coefficient. An image in which the movement is suppressed as compared with the case where there is no image.

As described above, according to the fifth embodiment, by providing motion vector processing section 1701, the magnitude of the motion vector input to motion generation section 103 can be changed, and the magnitude of the motion vector can be enhanced or suppressed. Adding motion information to an image can be realized.

Therefore, in addition to the effects of the first embodiment described above, it is possible to generate an image to which enhanced motion information is added, and conversely, to generate an image to which suppressed motion information is added. . Accordingly, it is possible to realize an image processing apparatus that gives a more dynamic presence or generates and displays an image to which motion information is added while suppressing the presence.

The image processing apparatus according to the fifth embodiment of the present invention has been described by adding the motion vector processing section 1701 to the image processing apparatus according to the first embodiment. And a motion vector processing unit 1701 in the image processing apparatus according to the third embodiment.
The same effect can be obtained by adding

Further, the image processing apparatus according to the fifth embodiment can be combined with the threshold determination processing section 1601 of the image processing apparatus according to the fourth embodiment described above. , It is possible to determine whether or not to multiply the motion vector by a predetermined coefficient, or to change the value of the coefficient by which the motion vector is multiplied.

Embodiment 6 FIG. The image processing apparatus according to the sixth embodiment of the present invention generates motion information based on a motion vector and adds the motion information to the image data as in the first embodiment, but determines the area of the image data to be processed. The motion information is generated based on the determination result and is added to the image data.

FIG. 19 is a block diagram showing the overall configuration of the image processing apparatus according to the sixth embodiment. As shown, the image processing apparatus according to the sixth embodiment includes an image
1. First frame memory 102, motion generation unit 103, second frame memory 104, and area determination processing unit 190
1 in which an area determination processing unit 1901 is added to the overall configuration of the image processing apparatus according to the first embodiment (FIG. 1). Note that, in the image processing apparatus according to the sixth embodiment, the image decompression unit 101 and the first frame memory 10
2, the motion generation unit 103, and the second frame memory 10
4 is the same as Embodiment 1 described above, and therefore, is denoted by the same reference numeral and description thereof is omitted.

The area determination processing unit 1901 is the
01 and the macroblock identification data corresponding to the motion vector,
A second motion vector (x ′, y ′) for image data generated by the motion generation unit 103 and macroblock identification data corresponding to the second motion vector are generated and output. That is, the area determination processing unit 1901
A region of image data to be processed is determined, and a second motion vector to be output is determined for each region to be processed.

FIG. 20 shows macro blocks constituting image data stored in the first frame memory 102. Note that the image data shown in FIG. 20 is composed of U macroblocks in the horizontal direction and V macroblocks in the vertical direction. It shall indicate the position of the block.

The area determination processing unit 1901 determines the position of the macroblock to be processed from the macroblock identification data which is the information (u, v) representing the position of the input macroblock. For example, when the motion information is not added to the peripheral portion of the display image data to be generated, the determination conditions of the region to be processed are 2 ≦ u <U−2 and 2 ≦ v <V−2, and For the motion vector corresponding to the macroblock satisfying the condition, the input motion vector (x, y)
And the output second motion vector (x ′, y ′) have the same value. On the other hand, for a motion vector corresponding to a macroblock that does not satisfy the condition, the output second motion vector (x ′, y ′) is set to (0, 0). Note that the above-described area determination condition can be determined based on, for example, the content (image and character) of the image data.

Thus, the motion generating unit 103 uses the second motion vector (x ′, y ′) output from the area determination processing unit 1901 to store the image with the motion component in the second frame memory 104. As a result, a generated image can be obtained in the second frame memory 104 in which no motion information is added to the peripheral portion of the image.

Therefore, according to the sixth embodiment, by providing the area determination processing unit 1901, it is possible to arbitrarily set the area to which the motion information is added. In addition, it is possible to separate a part that gives a dynamic sense of realism from a part that does not, according to the provided image data. For example, in the case of a movie having subtitles, an image is generated without adding motion information to the subtitle portion to make the subtitle portion easier to read, and motion information is provided to the video portion to generate a realistic image. I do.

When (x ′, y ′) = (0,0) is input as the second motion vector to the motion generation unit 103, the motion generation unit 103 does not need to perform the weighting addition process. The pixels stored in the first frame memory 102 may be stored in the second frame memory 104 as they are. Therefore, the processing amount of the weighting and adding unit 204 that performs the weighting and adding process in the motion generating unit 103 can be reduced, and the processing speed can be improved.

It should be noted that the region determination processing unit 1901 of the image processing apparatus according to the sixth embodiment has been described as one that determines whether a region is to be processed or not, but is not limited to this. It is only necessary that the determination processing unit 1901 determines an area to be processed and performs different processing for each of the determined areas.

The image processing apparatus according to the sixth embodiment of the present invention has been described by adding the area determination processing unit 1901 to the image processing apparatus according to the first embodiment. The same effect can be obtained even when the image processing apparatus according to the third embodiment is added with the area determination processing unit 1901.

Further, the image processing apparatus according to the sixth embodiment can be implemented by a combination with those described in the fourth and fifth embodiments. In addition to the effects, the effects of the image processing apparatuses according to the fourth and fifth embodiments described above can be obtained.

Further, in the first to sixth embodiments, the coded image data is assumed to be provided with a motion vector for each macroblock. However, for various processing units such as for each frame or for each object, The image processing described in Embodiments 1 to 6 can be performed using the motion vector assigned in this way, and the same effect can be obtained.

Also, when performing inter-frame coding on image data, intra-frame coding is performed on image data of some frames, and the frame (I-frame) on which the intra-frame coding has been performed is performed. In this case, the compression ratio is often increased by inter-frame encoding a frame (P frame or B frame) that is close in time series and processing only the difference. A motion vector is assigned to encoded data of a frame or a B frame, but no motion vector is assigned to an I frame. When processing such I-frame image data, similar effects can be obtained by processing using motion vectors given to P-frame or B-frame image data that are close in time series. Becomes This is because inter-frame coding is based on the correlation of image data that is close in time series, and it is possible to obtain good motion information even if a motion vector of highly correlated image data is used. Is possible.

[0154]

According to the image processing apparatus or the image processing method of the present invention, weighted addition processing of the pixels constituting the image data is performed based on the motion vector as the change information to generate display image data. Thus, the display image data accompanies the motion information, and a display result having a feeling of movement can be obtained. Therefore, it is possible to perform display with less discomfort than the image processing apparatus or the image processing method according to the related art, and the user can easily know the reproduction state by adding the motion information. It is what becomes.

Further, according to the image processing apparatus or the image processing method of the present invention, change information transition information indicating a temporal change of a motion vector, which is change information, is acquired, and based on the acquired change information transition information. By performing weighted addition processing of the pixels constituting the image data and generating the display image data, it is possible to obtain a display image more appropriately corresponding to the state of the image.

Further, according to the image processing apparatus or the image processing method of the present invention, by inputting reproduction information indicating a display state of display image data, and generating display image data using the reproduction information. Thus, it is possible to obtain a display image appropriately corresponding to the state of the image.

Further, according to the image processing apparatus or the image processing method of the present invention, a processing unit image which is a unit smaller than the unit image data based on a plurality of pieces of change information previously assigned to each unit image data. By generating change information corresponding to the data and performing weighted addition processing of the pixels constituting the processing unit image data using the change information corresponding to the generated processing unit image data, the change information corresponding to the image state can be more appropriately adjusted. Can be obtained.

Further, according to the image processing apparatus or the image processing method of the present invention, a motion vector, which is input change information, is compared with a threshold value, and second change information generated based on the comparison result is obtained. By generating the image data for display using, for example, if the image includes a camera shake such as a small motion vector or an image or an object having almost no motion vector, the motion information is not added. Therefore, it is possible to eliminate the possibility that the camera shake image is mistaken for the image motion, and to provide unnecessary motion information to the image, thereby conveying the original motion of the image and providing a good image with less discomfort and discomfort. Can be realized.

Further, according to the image processing apparatus or the image processing method of the present invention, the display is performed by using the second change information generated by multiplying the value of the motion information as the input change information by a predetermined coefficient. By generating image data for use, it is possible to generate an image to which enhanced motion information is added, and conversely, to generate an image to which suppressed motion information is added, thereby giving a more dynamic presence. Alternatively, it is possible to realize an image processing apparatus that generates and displays an image to which motion information has been added while suppressing a sense of reality.

Further, according to the image processing apparatus or the image processing method of the present invention, the area of the image data to be processed is determined, and the display image data is generated using the determination result, whereby the motion information is displayed. Can be set arbitrarily, and according to the provided image data,
It is possible to separate a part that gives a dynamic presence from a part that does not.

Further, according to the image processing apparatus or the image processing method of the present invention, even when transmitting an image at a low bit rate, the original motion of the image is transmitted, thereby achieving the same operation as in the above-described reproduction. The effect of is obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an overall configuration of an image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is a diagram for describing an internal configuration and a function of a motion generation unit included in the apparatus of the embodiment.

FIG. 3 is a diagram for explaining macroblocks and motion vectors constituting image data to be processed by the apparatus of the embodiment.

FIG. 4 is a diagram illustrating a motion generation / assignment process performed by a motion generation unit included in the apparatus according to the embodiment;

FIG. 5 is a diagram illustrating a weighted addition process performed by a motion generation unit included in the apparatus according to the embodiment;

FIG. 6 is a diagram for explaining a display state of image data on which motion generation / addition processing has been performed by the apparatus according to the embodiment;

FIG. 7 is a diagram for explaining a process in a case where a motion generation / assignment process performed by a motion generation unit included in the apparatus according to the embodiment is performed as a non-linear process;

FIG. 8 is a block diagram illustrating an overall configuration of an image processing apparatus according to a second embodiment of the present invention.

FIG. 9 is a diagram for describing an internal configuration and a function of a motion generation unit included in the apparatus of the embodiment.

FIG. 10 is a diagram for explaining a change state (transition state) of a motion vector to be processed by the apparatus of the embodiment.

FIG. 11 is a diagram for explaining a weighted addition process corresponding to the magnitude of the motion vector and the transition state by the apparatus according to the embodiment;

FIG. 12 is a diagram for explaining a weighted addition process corresponding to information indicating a reproduction direction by the device of the embodiment.

FIG. 13 is a diagram for explaining an internal configuration and functions of a motion generation unit included in an image processing device according to a third embodiment of the present invention.

FIG. 14 is a diagram illustrating a motion vector interpolation process performed by a vector interpolation unit included in the apparatus according to the embodiment.

FIG. 15 is a diagram for describing frames of image data and high-speed reproduction processing.

FIG. 16 is a block diagram illustrating an overall configuration of an image processing device according to a fourth embodiment of the present invention.

FIG. 17 is a block diagram illustrating an overall configuration of an image processing device according to a fifth embodiment of the present invention.

FIG. 18 is a diagram for describing processing in a motion vector processing unit.

FIG. 19 is a block diagram illustrating an overall configuration of an image processing device according to a sixth embodiment of the present invention.

FIG. 20 is a diagram showing macro blocks constituting image data stored in a first frame memory included in the apparatus of the embodiment.

[Explanation of symbols]

Reference Signs List 101 Image decompression unit 102 First frame memory 103 Motion generation unit 104 Second frame memory 201 Moving amount weight coefficient determination unit 202 First address generation unit 203 Second address generation unit 204 Weight addition unit 205 Processing target pixel determination unit 210, 211 , 212, 213 multiplier 214 adder 301, 302, 303, 304 macroblock 311, 312, 313, 314 motion vector (for each macroblock) 401 reference pixel 402, 403, 404 moving pixel 501, 751 motion vector (primary 520) Coordinate position (reference pixel) 521, 522, 523 Coordinate position (moving pixel) 551, 552 Coordinate position 751 Area having pixel value 805 Playback information input unit 901, 902, 903, 904 Motion vector storage unit 905 Motion vector Tough 906 changes information storage unit 1301 the motion vector interpolation unit 1310,1311,1312,1313,1320,
1321, 1322, 1323 Multiplier 1330 Adder 1401 Interpolation target pixel 1402 Interpolation area 1403 Interpolation reference pixel 1404 Motion vector (interpolation target pixel) 1601 Threshold determination processing unit 1701 Motion vector processing unit 1901 Area determination processing unit

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 7/32 H04N 7/137 Z

Claims (20)

[Claims] 1. An image processing apparatus for generating image data for display by inputting image data and change information which is information indicating a temporal change of the image data, the image processing device comprising: An image processing apparatus, comprising: a motion generation unit that performs a weighted addition process on pixels constituting image data and generates display image data. 2. The image processing apparatus according to claim 1, wherein the motion generation unit uses a change information assigned to each unit image data in advance to perform weighted addition of pixels constituting the unit image data. An image processing apparatus for performing processing. 3. The image processing apparatus according to claim 1, wherein the motion generation unit is configured to generate a smaller unit of the unit image data based on a plurality of pieces of change information previously assigned to each unit image data. Generating change information corresponding to a certain processing unit image data, and performing weighted addition processing of pixels constituting the processing unit image data using the change information corresponding to the generated processing unit image data. Image processing device. 4. The image processing apparatus according to claim 2, wherein the motion generating unit determines a moving amount and a weighting factor for each pixel from the change information, A processing target pixel determination unit that determines a pixel to be processed based on the amount, and a weighting addition unit that weights and adds the pixel data of the processing target pixel using the weighting coefficient. Image processing apparatus. 5. The image processing device according to claim 3, wherein the motion generation unit is a unit that is a unit smaller than the unit image data based on a plurality of pieces of change information attached to each unit image data. A motion vector interpolation unit that generates change information corresponding to the unit image data; a movement weight coefficient determination unit that determines a movement amount and a weight coefficient for each pixel from the change information corresponding to the processing unit image data; A processing target pixel determination unit that determines a pixel to be processed based on the amount, and a weighting addition unit that weights and adds the pixel data of the processing target pixel using the weighting coefficient. Image processing apparatus. 6. The image processing apparatus according to claim 1, wherein the motion generation unit acquires change information transition information indicating a temporal change of the change information, An image processing apparatus, comprising: performing weighted addition processing of pixels constituting image data based on change information transition information to generate display image data. 7. The image processing apparatus according to claim 4, wherein the movement amount weight coefficient determination unit includes a change information storage unit that stores input change information; An image processing apparatus characterized in that a moving amount and a weighting factor are determined for each pixel based on a plurality of pieces of change information having different times stored in a storage unit. 8. The image processing apparatus according to claim 1, further comprising: a reproduction information input unit configured to input reproduction information for instructing a display state of the display image data. An image processing device, wherein the motion generation unit generates display image data using the reproduction information. 9. The image processing apparatus according to claim 1, wherein the input change information is compared with a threshold, and second change information is generated based on the comparison result. An image processing apparatus, further comprising: a threshold determination processing unit that performs the processing, wherein the motion generation unit generates display image data using the second change information. 10. The image processing apparatus according to claim 1, wherein a value of a change information to be input is multiplied by a predetermined coefficient to obtain a second value.
An image processing apparatus, further comprising: a motion vector processing unit that generates change information of (a), wherein the motion generation unit generates display image data using the second change information. 11. The image processing apparatus according to claim 1, further comprising an area determination unit that determines an area of image data to be processed. apparatus. 12. An image processing method for generating image data for display by inputting image data and change information which is information indicating a temporal change of the image data, the image processing method comprising: An image processing method, comprising: performing weighted addition processing of pixels constituting image data to generate display image data. 13. The image processing method according to claim 12, wherein the change information previously assigned to each unit image data is used for
An image processing method, comprising: performing weighted addition processing of pixels constituting the unit data. 14. The image processing method according to claim 12, wherein, based on a plurality of pieces of change information previously assigned to each unit image data, the processing unit image data smaller than the unit image data is processed. An image processing method comprising: generating corresponding change information; and performing weighted addition processing of pixels constituting the processing unit image data using the change information corresponding to the generated processing unit image data. 15. The image processing method according to claim 12, wherein change information transition information indicating a temporal change of the change information is acquired, and the change information transition information is acquired based on the acquired change information transition information. And performing a weighted addition process on pixels constituting the image data to generate display image data. 16. The image processing method according to claim 12, wherein reproduction information indicating a display state of the display image data is input, and the display information is displayed using the reproduction information. An image processing method for generating image data. 17. The image processing method according to claim 12, wherein the input change information is compared with a threshold, and the second change information generated based on the comparison result. An image processing method comprising: generating image data for display using 18. The image processing method according to claim 12, wherein display is performed using second change information generated by multiplying a value of input change information by a predetermined coefficient. An image processing method, comprising: generating image data for use. 19. The image processing method according to claim 12, wherein a region of the image data to be processed is determined, and display image data is generated using the determination result. An image processing method, characterized in that: 20. A computer-readable program for recording a program for causing a computer to execute a weighted addition process of pixels constituting image data based on change information which is information indicating a temporal change of the image data. Possible recording medium.