JP2006197256A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the reproduction quality of moving images including a scene change. <P>SOLUTION: In an image processing apparatus 100, an input part 110 acquires a prescribed plurality of original images to be displayed as moving images. A scene change detection part 120 detects a scene change included in the acquired original image string. An image composition part 150 generates a composite image by composing the original images acquired before and after the scene change. A stream generation part 170 inserts the generated composite image into the original images between which the scene change is held to generate a data stream for reproducing moving images, which consists of the acquired original images and the composite image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing technique, and more particularly to an apparatus for encoding moving image data.

  The development of Internet technology has enabled high-speed and large-capacity communication. As a result, the user can easily enjoy moving image content with a relatively large amount of information by Internet distribution or the like. In recent years, the diversity of environments in which users can enjoy moving image content, such as the distribution of moving images through the Internet, in addition to the reception of traditional television broadcasts and playback from recording media such as DVDs (Digital Multipurpose Discs), has been increasing recently. It is increasing dramatically. As a method for efficiently handling moving image data, a technique for detecting a scene change based on user's subjectivity has been proposed (for example, see Patent Document 1).

JP 2004-194197 A

  Now that the environment that allows users to view video content on various media is taken for granted, it is doubtful that what should be pursued next is how high-quality content can be provided to users. There is no room. The pursuit of content quality is the key to further development of digital culture in the future.

  The inventor has made the present invention based on the above problems, and an object thereof is to improve the reproduction quality of moving image content.

  An image processing apparatus according to an aspect of the present invention includes an image input unit that acquires a plurality of continuous original images, a scene change detection unit that detects a scene change between two of the plurality of original images, and a scene A combined image generating unit that generates a combined image based on two original images sandwiching a change, a stream generating unit that generates a reproduction data stream including a plurality of original images and the combined image, the stream generating unit The data stream is generated by inserting a composite image between two original images sandwiching a scene change.

  The “scene change” is not necessarily objectively determined, and it may be considered that there is a scene change when there is a difference in the image to some extent. When playing a video that includes a scene change, the viewer may get the impression that the flow of the video is interrupted. In this mode, a data stream in which a composite image is inserted between the original images sandwiching the scene change is used. Since it is generated, it is possible to alleviate the feeling of disconnection of the video when played back.

  Another embodiment of the present invention also relates to an image processing apparatus. The image processing apparatus according to this aspect generates an interpolation image based on two or more original images before or after a scene change among a plurality of original images, instead of the composite image generation unit of the image processing apparatus according to the above-described aspect. An interpolated image generation unit is provided, and the stream generation unit inserts an interpolated image instead of the composite image between two original images sandwiching a scene change to generate a data stream.

  Still another aspect of the present invention also relates to an image processing apparatus, and instead of the composite image generation unit of the image processing apparatus according to the first aspect described above, two or more original images before or after a scene change among a plurality of original images. An interpolated image generating unit for generating an interpolated image based on the interpolated image based on the two or more original images before the scene change and an interpolated image based on the two or more original images after the scene change The stream generation unit generates a data stream by inserting the composite image generated by the composite image generation unit between two original images sandwiching a scene change.

  Also in these modes, since a data stream for reproduction is generated with an interpolated image or a composite image inserted between the original images sandwiching the scene change, the moving image including the scene change is the same as the first mode described above. Can improve the quality.

  In the data stream, the display interval of the composite image and the interpolation image may be set shorter than the display interval of the original image. In this case, for example, a moving image stream for realizing a smoother scene change can be generated by sufficiently drawing out the performance of a display with a high frame rate.

  Still another aspect of the present invention also relates to an image processing apparatus, an image input unit that acquires a plurality of continuous original images, a scene change detection unit that detects a scene change when two of the plurality of original images sandwich a scene change, A stream generation unit that generates a reproduction data stream including a plurality of original images and an image newly generated according to one mode selected from the plurality of modes, and the stream generation unit generates the newly generated An image is inserted between two original images sandwiching the scene change to generate a data stream. Thereby, the quality of the moving image containing a scene change can be improved similarly to the aspect described first.

  The image processing apparatus of the present invention may further include a correspondence information data generation unit that generates correspondence information data for two adjacent original images that do not sandwich a scene change among a plurality of original images. May generate a data stream further including correspondence information data. As a result, the stream of the original image can be compressed while improving the image quality of the moving image including the scene change as described above. In this specification, the terms “frame” and “image” are used without distinction unless otherwise specified.

  In any of the above aspects, the scene change detection unit may determine that a scene change has occurred when the difference between two consecutive original images is between a predetermined upper limit value and a lower limit value. In addition, the scene change detection unit detects a case where a change between two consecutive original images is large while maintaining the continuity of the semantic content of the moving image between two consecutive original images. Also good.

  Those in which the above configurations and processes are arbitrarily replaced, or the expression is changed to a computer program, a recording medium, or the like are also effective as the present invention.

(Embodiment 1)
The image processing apparatus according to an embodiment of the present invention acquires moving image frames including scene changes as original images and reproduces moving image data suitable for reproduction on a display or the like that supports high frame rate display. Create a stream. In this embodiment, a series of images displayed as moving images at a rate of 30 images per second is handled as an original image, but the original image is not necessarily limited to this. Regarding the scene change, it can be considered that the scene change has occurred when the difference between the images is large. In the detection of the scene change, which will be described later, for example, the scene change is detected by comparing the absolute value of the difference between the pixel values corresponding to each other in position and the sum of squares with a predetermined threshold value. be able to.

  Here, the predetermined threshold value is composed of an upper limit value and a lower limit value, and it is determined that a scene change has occurred when the above difference value is any value between the upper limit value and the lower limit value. Is desirable. This is because there are generally two types of scene changes. That is, in a narrow sense, a scene change refers to one in which semantic continuity in the display content of a moving image is cut off, for example, when a person image is suddenly switched to a landscape image. In this case, the contents of the original images sandwiching the scene change are naturally different, and the difference between the original images described above becomes very large. On the other hand, for example, when a person opens a door and goes out of the room, it can be said that it is a scene change in a broad sense. In this type of scene change in which the continuity of the displayed content is maintained at a certain level, the difference between the original images is somewhat larger, but the degree of the difference compared to the above-mentioned scene change in the narrow sense. Is small.

  In view of the fact that the present invention aims to eliminate the sense of disconnection of a moving image at the time of a scene change, in the present invention, the latter type should be detected as a scene change mainly in terms of the semantic content of the moving image. This is a scene change when the continuity of the image is maintained and the change between two consecutive original images is large. This is because when there is a break in meaning and the scene changes completely, that is, in the former type of scene change, it is often better not to maintain the continuity of moving images. As described above, since the difference between the original images is larger in the former type of scene change than in the latter type, the difference between the images is simply equal to or greater than a predetermined value as a threshold for scene change determination. In addition to the lower limit value for determining whether or not, the introduction of the upper limit value makes it easy to limit the type of scene change detected to the latter. The threshold value may be determined by experiments for both the upper limit value and the lower limit value. The above discussion regarding scene change detection is valid throughout this specification.

  FIG. 1 shows a configuration of an image processing apparatus 100 in the present embodiment. Each component of the image processing apparatus 100 is centered on an arbitrary computer CPU, memory, a program that realizes the components shown in the figure loaded in the memory, a storage unit such as a hard disk for storing the program, and a network connection interface. However, it will be understood by those skilled in the art that there are various modifications in the implementation method and apparatus. Each drawing described below shows a functional unit block, not a hardware unit configuration.

  The image processing apparatus 100 includes an input unit 110 that acquires an original image, a scene change detection unit 120 that detects a scene change by the above-described method, and a mode table 130 that stores settings related to an operation mode of the image processing apparatus 100 by a user or the like. A mode determination unit 140 that distributes the original images acquired according to the settings in the mode table 130, an image synthesis unit 150 that generates a synthesized image by synthesizing a predetermined image, and extrapolating the predetermined original image. An interpolated image generating unit 160 that generates an interpolated image, and a stream generating unit 170 that generates an image data stream for reproduction composed of an original image, a composite image, and an interpolated image. The image synthesizing unit 150 synthesizes a predetermined original image or interpolated image by, for example, α blending to generate a synthesized image.

  The image processing apparatus 100 has three operation modes. The user inputs the mode selected by the user via a mode input unit (not shown). The user's selection regarding the mode is stored in the mode table 130. The content of the data stream that is finally generated by the image processing apparatus 100 varies depending on the difference in each operation mode. Details of specific differences will be described below. In any mode, the image processing apparatus 100 generates a reproduction data stream with improved image quality based on an original image sequence including a scene change. There is no distinction in terms. Therefore, the mode to be set may be determined according to, for example, the appearance preference of the reproduced data stream.

  The first mode is a mode that should also be referred to as a composite image mode, and the stream generation unit 170 finally generates a reproduction data stream including a composite image of the original image before and after the scene change. The second mode is a mode that should be referred to as an interpolated image mode. An interpolated image is generated using two original images immediately before a scene change, and a reproduction data stream including the generated interpolated image is generated. The third mode is a composite mode. That is, the interpolated image based on the two original images immediately before the scene change and the interpolated image based on the two original images immediately after the scene change are synthesized, and a reproduction data stream including the generated synthesized image is generated. .

  In the first mode, first, the original image acquired by the input unit 110 is sent to the detection unit 120. The detection unit 120 detects a pair of original images sandwiching a scene change, and uses information indicating the scene change between the original images (hereinafter referred to as “scene change information”) and the original image as a mode determination unit 140. The data is sent to each of the stream generation units 170. The mode determination unit 140 refers to the mode table 130 and confirms the operation mode. Here, the first mode is currently set. In this case, the mode determination unit 140 sends the pair of original images sandwiching the scene change to the image composition unit 150 based on the scene change information.

  The image composition unit 150 performs α blending on the pair of original images received from the mode determination unit 140 to generate a predetermined number of composite images. The generated composite image is sent to the stream generation unit 170. The stream generation unit 170 generates a data stream for moving image reproduction based on the original image, the scene change information, and the composite image. In this data stream, a composite image is inserted at a display interval shorter than the display interval of the original image between the original images sandwiching the scene change.

FIG. 2 is a diagram illustrating an original image acquired by the input unit 110 of FIG. An original image displayed as a moving image at a rate of 30 frames per second is denoted as KF _i ( _i is a non-negative integer). In the figure, it is shown that there is a scene change between KF ₂₂ and KF ₂₃ . T _s ( _s is a non-negative number) indicates the display timing of each image. That is, T ₀ indicates the display timing of KF ₀ , and there is a difference of 1/30 seconds between T _s and T _{s + 1} . For example display timing _{T 15} of _{KF 15} 0.5 seconds after _{T 0,} display timing _{T 30} of _{KF 30} is 1 second after the _{T 0.}

FIG. 3 is a diagram illustrating a structure of a data stream generated by the stream generation unit 170 in the first mode of the image processing apparatus 100 based on the original image sequence in FIG. As in FIG. 2, the original image is represented by KF _i . Further, the composite image is represented by CF _j ( _j is a non-negative integer). Further, as the information indicating the display timing of each image, T _s is given immediately after the image. The meaning of T _s is the same as in FIG. In this data stream, KF ₀ , KF ₁ ... KF ₂₂ should be displayed at timings T ₀ , T ₁ , T ₂₂ , respectively, and between KF ₂₂ and KF ₂₃ sandwiching a scene change, a composite image A certain CF ₁ , CF ₂ ,..., CF ₉ is to be displayed at the timing of T _22.1 , T _{22.2 ,.} That is, the stream generation unit 170 generates a data stream in which the composite image display interval is shorter than that of the original image.

FIG. 4 is a diagram showing an image sequence when the data stream of FIG. 3 is reproduced. In the figure, between KF ₂₂ and KF ₂₃ which are original images sandwiching a scene change, composite images CF _{1 to} CF ₁₀ are displayed at a display interval of 1/10 of the original image display interval. It is shown. When this is played back on a high frame rate display or the like having a playback capability of more than 30 images per second, many composite images are displayed before and after the scene change, so the difference between the images of KF ₂₂ and KF ₂₃ is The sense of discontinuity of the user's apparent image due to the largeness can be alleviated, and the playback quality of moving images including scene changes is improved.

  In the second mode of the image processing apparatus 100 according to the present embodiment, the operations of the input unit 110 and the detection unit 120 are the same as in the first operation mode. As a result of the mode determination unit 140 referring to the mode table 130, if it is determined that the second operation mode is selected, the mode determination unit 140 sends the two original images immediately before the scene change and the interpolated image generation unit 160.

  The interpolated image generating unit 160 generates a predetermined number of interpolated images by extrapolating the two original images immediately before the scene change in the forward direction of the time axis, and sends them to the stream generating unit 170. The stream generation unit 170 generates a data stream for moving image reproduction based on the original image, the scene change information, and the interpolated image. In this data stream, extrapolated interpolated images are inserted between original images sandwiching a scene change at a display interval shorter than the original image display interval.

FIG. 5 is a diagram showing a structure of a data stream generated in the second operation mode of the image processing apparatus 100 of the present embodiment based on the original image sequence of FIG. Only differences from FIG. 3 will be described. In FIG. 5, the interpolation image is represented by EF _k ( _k is a non-negative integer). As shown in FIG. 5, the stream generating unit 170, composite image _CF 1 of FIG. 3, instead of the · · · CF _9, interpolated image _EF 1, · · · EF ₉ is a _{KF 22} sandwiching the scene change KF ₂₃ , a data stream inserted at an interval shorter than the display interval of KF _i is generated. Therefore, the image reproduced from the data stream generated in the second operation mode is obtained by replacing the synthesized image of the reproduced image sequence related to the first operation mode shown in FIG. 4 with the interpolation image. If this is played back on a high frame rate device, a large number of interpolated images are inserted and displayed before and after the scene change, and the visual image quality of the moving image can be improved as in the first operation mode. In the second operation mode, an extrapolated interpolation image may be generated based on the two original images immediately after the scene change instead of the original image immediately before the scene change.

  Even in the third mode of the image processing apparatus 100 according to the present embodiment, the operations of the input unit 110 and the detection unit 120 are the same as those in the first mode and the second mode. When the mode determination unit 140 refers to the mode table 130 and confirms that the mode is the third operation mode, the mode determination unit 140 generates an interpolated image between the two original images immediately before the scene change and the two original images immediately after the scene change. Send to section 160.

  The interpolated image generating unit 160 extrapolates the two original images immediately before the scene change in the forward direction of the time axis to generate a predetermined number of interpolated images, and also generates the two original images immediately after the scene change. A predetermined number of interpolated images are generated by extrapolating in the reverse direction of the time axis. These interpolated images are sent to the image composition unit 150. The image synthesizing unit 150 synthesizes an interpolated image based on the original image before the scene change and an interpolated image based on the original image after the scene change by α blending to synthesize the synthesized image. Generate. The generated composite image is sent to the stream generation unit 170. The stream generation unit 170 generates a data stream similar to the data stream in the first mode shown in FIG. 3 based on the original image, the scene change information, and the composite image. However, in the third mode, the method for generating a composite image is different from that in the first mode as described above. If a moving image is reproduced based on this data stream, a synthesized image can be inserted and displayed between key frames sandwiching a scene change, so that the same visual effect as in the first and second modes can be obtained.

  The operation of the image processing apparatus 100 based on the above configuration will be described. FIG. 6 is an example of a flowchart illustrating the operation of the image processing apparatus 100 according to the first embodiment. The input unit 110 acquires an original image (S10). The scene change detection unit 120 detects a scene change (S12), and sends the original image and the scene change information to the mode determination unit 140 and the stream generation unit 170 (S14). The mode determination unit 140 refers to the mode table 130 and, in the first operation mode, sends two original images sandwiching the scene change to the image composition unit 150, and the image composition unit 150 generates a composite image ( 1 of S16, S18). In the case of the second operation mode, the two original images immediately before the scene change are sent to the interpolation image generation unit 160, and the interpolation image generation unit 160 generates an interpolation image (S16-2, S20). In the case of the third operation mode, the two original images immediately before the scene change and the two original images immediately after the scene change are sent to the interpolation image generation unit 160, and an interpolation image is generated from each image pair (3 in S16). , S22). Those interpolated images are sent to the image composition unit 150 to generate a composite image (S24). The stream generation unit 170 generates a data stream using the original image and scene change information received from the scene change detection unit 120 and the composite image received from the image composition unit 150 or the interpolation image received from the interpolation image generation unit 160. (S26).

(Embodiment 2)
An image processing apparatus according to another embodiment of the present invention acquires frames extracted at intervals of 0.5 seconds from a moving image of 30 frames per second as an original image, and generates a correspondence information file between the original images. It has a configuration. As a result, for example, while compressing moving image data so as to be suitable for transmission via a network, the feeling of disconnection of an image at the time of a scene change is reduced. Of course, the relationship between the display interval of the original moving image and the extracted original image is not limited to the above.

  FIG. 7 is a diagram illustrating a configuration of the image processing apparatus 200 according to the second embodiment. The image processing apparatus 200 of this embodiment includes a correspondence information data generation unit 210 in addition to the image processing apparatus 100 of the first embodiment shown in FIG. The correspondence information data generation unit performs matching calculation between adjacent original images and generates data related to corresponding points between the original images. For the generation of the correspondence information data, it is preferable to use the method of acquiring the correspondence relationship between the pixels disclosed by the present applicant in Japanese Patent No. 2927350, but is not limited thereto. Since there is no difference between the image processing apparatus 100 of FIG. 1 and the image processing apparatus 200 of FIG.

  The image processing apparatus 200 according to the second embodiment also has the same three operation modes as the image processing apparatus 100 according to the first embodiment. First, in the first operation mode, the input unit 110 acquires a frame extracted every 0.5 seconds from a moving image of 30 frames per second as an original image. The scene change detection unit 120 detects a scene change from these original images, and sends the original image and scene change information to the correspondence information data generation unit 210 and the mode determination unit 140. The mode determination unit 140 refers to the mode table 130 and, upon confirming the first operation mode, sends the original images before and after the scene change to the image composition unit 150. The image synthesizing unit 150 generates a predetermined number of synthesized images from these received original images by α blending, and sends them to the stream generating unit 170.

  On the other hand, the correspondence information data generation unit 210 generates correspondence information data of the original image received from the detection unit 120. However, based on the scene change information, matching calculation is skipped for two original images sandwiching the scene change, and no corresponding information data is generated. As will be described later, the correspondence information data is used to generate an intermediate image between the original images and reproduce the moving image. However, if the intermediate image is generated using the correspondence information data for the original image sandwiching the scene change. This is because an image such as unintended morphing is generated, which hinders the reproduction of a desired moving image. The correspondence information data generated by the correspondence information data generation unit 210, each original image, and scene change information are then sent to the stream generation unit 170.

  The stream generation unit 170 generates a data stream for moving image reproduction using the composite image received from the image composition unit 150 and the correspondence information data, the original image, and the scene change information received from the correspondence information data generation unit 210. The structure of the data stream will be described later.

FIG. 8 is a diagram schematically illustrating an original image input to the image processing apparatus 200 according to the second embodiment. As described above, in this embodiment, frames extracted every 30 seconds from a moving image of 30 frames per second are used as original images. That is, in FIG. 2, images corresponding to KF ₀ , KF ₁₅ , KF ₃₀ ... Are input as original images. FIG. 8 shows this state, and it is assumed that a scene change exists between KF ₁₅ and KF ₃₀ . The meaning of T _s added to the image is the same as in FIG.

  FIG. 9 shows the structure of a data stream generated by the stream generation unit 170 in the first operation mode of the image processing apparatus 200 based on the original image of FIG. In the first operation mode, the stream generation unit 170 generates a data stream including an original image, corresponding point information between the original images that do not sandwich the scene change, and a composite image inserted between the images that sandwich the scene change. .

In FIG. 9, KF _i , CF _j , and T _s are the same as those in FIG. Also shows the corresponding information data of the KF _i and KF _{i + alpha} at _{M i, i + α.} As the figure, corresponding information data _{M 0, 15} is continued between them _KF 0, _{KF 15} displayed at the timing of _{T 0, T 15,} respectively. As described later, corresponding information data _{M 0, 15} is used to generate an intermediate image between the KF ₀ and _{KF 15.} Then 14 sheets of the composite image _{CF 1, CF 2,} and _{··· CF 14, T 16, T} 17 are the respective display timing, · · · _{T 29} is followed alternately. Further, KF ₃₀ and its display timing T ₃₀ , KF ₄₅ and T ₄₅ , and correspondence information data M ₃₀ and ₄₅ between KF ₃₀ and KF ₄₅ are arranged _thereafter . Note that the order of arrangement of the data stream generated by the stream generation unit 170 is not necessarily as shown in FIG. 9 as long as the content is composed of an original image, corresponding point information, and a composite image.

FIG. 10 is a diagram showing an image sequence of a moving image reproduced based on the data stream of FIG. In reproducing a moving image, a moving image reproducing apparatus or the like having a configuration for generating an intermediate image between original images by using correspondence information data and interpolating original images is required. The intermediate image is represented by IF _l ( _l is a non-negative integer). The meanings of KF _i , CF _j , and T _s are the same as in FIG.

FIG. 10 shows that KF ₀ is displayed at timing T ₀ and IF ₁ through IF ₁₄ are displayed at timing T ₁ through T ₁₄ . After that, KF ₁₅ is displayed at the timing of T _{15, and} the intermediate images are not generated because the scene change is sandwiched between KF ₁₅ and KF ₃₀ , the synthesized images CF _{1 to} CF ₁₄ are the timings of T _{16 to} T ₂₉ . Is displayed. That is, images are reproduced in the order of KF ₀ , IF ₁ , IF ₂ ,... IF ₁₄ , KF ₁₅ , CF ₁ , CF ₂ ,... CF ₁₄ , KF _30. Is done.

  According to the image processing apparatus 200 described above, it is possible to transmit a data stream including only an original image extracted from a moving image, correspondence information data, and a combined image, and to reproduce a moving image similar to the original on the receiving side. The compression effect of the image data amount is remarkable. On the other hand, by inserting a composite image between original images before and after a scene change that should avoid unnatural matching, it is possible to reduce the uncomfortable feeling during moving image reproduction caused by the scene change. In particular, when skipping matching and not generating an intermediate image, if no image is inserted in between, the sense of disconnection due to a scene change is further increased, but this can be avoided by inserting a composite image. If the data stream receiving side has a configuration corresponding to the composite image generation unit, instead of transmitting the composite image including the composite image, only the command for generating the composite image may be included. Thereby, the compression rate of transmission data can be improved. On the side that receives and plays the data, the composite image can be displayed before and after the scene change, so the image quality can be maintained.

  In the second operation mode of the image processing apparatus 200 according to the second embodiment, the operations of the input unit 110, the detection unit 120, and the correspondence information data generation unit 210 are the same as those in the first operation mode. When the mode determination unit 140 refers to the mode table 130 and confirms the second operation mode, the mode determination unit 140 sends the two original images immediately before the scene change to the interpolation image generation unit 160. The interpolated image generation unit 160 extrapolates the received original image in the forward direction of the time axis to generate a predetermined number of interpolated images, and sends the generated image to the stream generation unit 170.

  The stream generation unit 170 uses the interpolation image received from the interpolation image generation unit 160 and the correspondence information data, the original image, and the scene change information received from the correspondence information data generation unit 210 to generate a data stream for moving image reproduction. .

FIG. 11 is a diagram illustrating a structure of a data stream generated by the image processing apparatus 200 according to the second embodiment in the second operation mode. Only differences from FIG. 9 will be described. In FIG. 11, the interpolated image is represented by EF _k ( _k is a non-negative integer). That is, the composite image _CF 1 of FIG. 9, instead of the · · · _{CF 14,} 11, interpolated image _EF 1, · · · _{EF 14 is} inserted between the _{M 0, 15} and _{KF 30.} If this is reproduced, the reproduced image sequence according to the first operation mode shown in FIG. 10 is obtained by replacing the synthesized image with the interpolated image, and the same effect as in the first operation mode is produced. In the second operation mode, an extrapolated interpolation image may be generated based on the two original images immediately after the scene change instead of the original image immediately before the scene change.

  Even in the third operation mode of the image processing apparatus 200 according to the second embodiment, the operations of the input unit 110, the detection unit 120, and the correspondence information data generation unit 210 are the same as those in the first mode and the second mode. It is. When the mode determination unit 140 refers to the mode table 130 and confirms that the mode is the third operation mode, the mode determination unit 140 generates an interpolated image between the two original images immediately before the scene change and the two original images immediately after the scene change. Send to section 160.

  The interpolated image generating unit 160 extrapolates the two original images immediately before the scene change in the forward direction of the time axis to generate a predetermined number of interpolated images, and also generates the two original images immediately after the scene change. A predetermined number of interpolated images are generated by extrapolating in the reverse direction of the time axis. These interpolated images are sent to the image composition unit 150. The image synthesizing unit 150 synthesizes an interpolated image based on the original image before the scene change and an interpolated image based on the original image after the scene change by α blending to synthesize the synthesized image. Generate. The generated composite image is sent to the stream generation unit 170. The stream generation unit 170 uses the interpolated image received from the interpolated image generation unit 160 and the corresponding information data, original image, and scene change information received from the corresponding information data generating unit 210 to perform the first operation mode shown in FIG. A data stream similar to the data stream by is generated. However, in the third operation mode, the method of generating a composite image is different from the first mode as described above. If a moving image is reproduced based on this data stream, an image similar to the reproduced image sequence in the first operation mode shown in FIG. 10 can be obtained, and effects similar to those in the first and second modes can be obtained.

  As described above, in the second embodiment, in order to generate a moving image stream to be played back on a display having a display capability of 30 frames per second, the display timing of the generated intermediate image, synthesized image, and interpolated image is set in that way. However, in an environment where playback on a high frame rate compatible display or the like is possible, it is possible to increase the number of images generated and shorten the display interval to obtain higher quality moving images.

1 is a configuration diagram of an image processing apparatus according to an embodiment of the present invention. It is a figure which shows the original image which the image processing apparatus of FIG. 1 acquires. It is a figure which shows the data stream which the image processing apparatus of FIG. 1 produces | generates. It is a figure which shows the image reproduced | regenerated based on the data stream of FIG. It is a figure which shows another data stream which the image processing apparatus of FIG. 1 produces | generates. 3 is a flowchart illustrating processing of the image processing apparatus in FIG. 1. It is a block diagram of the image processing apparatus which concerns on another embodiment of this invention. It is a figure which shows the original image which the image processing apparatus of FIG. 7 acquires. It is a figure which shows the data stream which the image processing apparatus of FIG. 7 produces | generates. FIG. 10 is a diagram illustrating an image reproduced based on the data stream of FIG. 9. It is a figure which shows another data stream which the image processing apparatus of FIG. 7 produces | generates.

Explanation of symbols

  100, 200 image processing apparatus, 110 input unit, 120 scene change detection unit, 130 mode table, 140 mode determination unit, 150 image composition unit, 160 interpolated image generation unit, 170 stream generation unit 170, 210 corresponding information data generation unit.

Claims (9)

An image input unit for acquiring a plurality of continuous original images;
A scene change detection unit for detecting when a scene change is sandwiched between two of the plurality of original images;
A composite image generating unit that generates a composite image based on two original images sandwiching the scene change;
A stream generation unit that generates a reproduction data stream including the plurality of original images and the composite image;
The stream processing unit generates the data stream by inserting the composite image between two original images sandwiching the scene change. An image input unit for acquiring a plurality of continuous original images;
A scene change detection unit for detecting when a scene change is sandwiched between two of the plurality of original images;
An interpolation image generation unit that generates an interpolation image based on two or more original images before or after the scene change among the plurality of original images;
A stream generation unit that generates a reproduction data stream including the plurality of original images and the interpolation image;
The stream processing unit generates the data stream by inserting the interpolated image between two original images sandwiching the scene change. An image input unit for acquiring a plurality of continuous original images;
A scene change detection unit for detecting when a scene change is sandwiched between two of the plurality of original images;
An interpolation image generation unit that generates an interpolation image based on two or more original images before or after the scene change among the plurality of original images;
A composite image generating unit that generates a composite image based on an interpolated image based on two or more original images before the scene change and an interpolated image based on two or more original images after the scene change;
A stream generation unit that generates a reproduction data stream including the plurality of original images and the composite image;
The stream processing unit generates the data stream by inserting the interpolated image between two original images sandwiching the scene change.   4. The image processing apparatus according to claim 1, wherein in the data stream, a display interval of the composite image is set shorter than a display interval of the original image. 5.   The image processing apparatus according to claim 2, wherein in the data stream, a display interval of the interpolation image is set shorter than a display interval of the original image. An image input unit for acquiring a plurality of continuous original images;
A scene change detection unit for detecting when a scene change is sandwiched between two of the plurality of original images;
A stream generation unit that generates a reproduction data stream including the plurality of original images and an image newly generated according to one mode selected from the plurality of modes;
The stream processing unit is configured to generate the data stream by inserting the newly generated image between two original images sandwiching the scene change. A correspondence information data generation unit that generates correspondence information data for two adjacent original images among the plurality of original images that do not sandwich the scene change;
The image processing apparatus according to claim 1, wherein the stream generation unit generates the data stream further including the correspondence information data.   The scene change detection unit determines that the scene change has occurred when a difference between two consecutive original images among the original images is between a predetermined upper limit value and a lower limit value. The image processing apparatus according to any one of 1 to 7. The scene change detection unit detects, as a scene change, a case where a continuity in semantic content of a moving image is maintained between two consecutive original images and a large change between the two consecutive original images is large. An image processing apparatus according to claim 1, wherein:

Images