JP4095336B2 - Original image restoration apparatus and original image restoration program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an original image restoration apparatus and an original image restoration program, and in particular, an original image for restoring an original image before being superimposed from an image on which additional information such as a character super or telop is superimposed. The present invention relates to a restoration device and an original image restoration program.
[0002]
[Prior art]
In television broadcasting, text information such as emergency news, music program lyrics, sports scores, time information, or telop (hereinafter referred to as additional information) is a text image of the original image that is the original program. Superimpose and broadcast to viewers.
[0003]
In recent years, with the increase in the number of channels and the diversification of programs, many methods have been used in which a superimposed image previously used in a program is used again as another program material. By reusing an image once used, a program can be efficiently generated and provided to the viewer.
[0004]
Here, when reusing an image, the original image cannot be restored from the superimposed image, and the corresponding part of the text is blurred by special editing such as mosaic or overwritten with another image. To deal with it. The above contents will be described with reference to the drawings.
[0005]
FIG. 1 is a diagram illustrating an example of an image when a conventional superimposed image is reused.
[0006]
In FIG. 1, it is assumed that, for example, an image in which a telop “Temporary News” is superimposed is generated and broadcast in order to inform the viewer of urgent news information (FIG. 1 (a)). After that, if you decide to use this image in another program, you can overwrite the telop area of the screen that displays "Extra News" with another image or put a mosaic, Special editing is performed on the image so that the viewer cannot understand the displayed content, and the image is broadcast (FIG. 1B).
[0007]
On the other hand, as described above, there is a method in which additional information is encoded and transmitted separately from image information in the vertical blanking period or the data broadcasting area of digital broadcasting without superimposing the additional information on the image from the beginning. It is done.
[0008]
Further, in order to enable the receiver to restore the original image before being superimposed, a method of separately encoding the original image corresponding to the superimposed position is disclosed in JP 2000-41185 (TV). John signal encoding device and decoding device).
[0009]
[Problems to be solved by the invention]
However, in the above-described method, when special editing such as mosaic is performed, it is necessary to perform special editing as described above when reusing an image, and the specially edited image area gives viewers a sense of incongruity. Therefore, it is not a preferable method.
[0010]
Also, in the case of a method of encoding and transmitting character information etc. separately from image information, since it cannot be viewed with a receiver that does not support data broadcasting, for example, emergency news at the time of a disaster, etc. to all viewers It is not effective when it is necessary to inform.
[0011]
Also, in the case of a method of separately encoding an original image of only a portion to be superimposed, a moving image that is not superimposed can be restored by using the separately encoded original image. Since the portion to be imposed is equivalent to storing images twice, the information amount of the image becomes enormous.
[0012]
Therefore, in order to efficiently reuse an already superimposed image (moving image), a method for restoring the original image before being superimposed from only the image is required.
[0013]
The present invention has been made in view of the above points. When the additional information is superimposed, another frame image is added to the image portion where the additional information is superimposed using the motion information of the image. An object of the present invention is to provide an original image restoration apparatus and an original image restoration program capable of restoring an original image before being superimposed by synthesizing images acquired from the above information.
[0014]
[Means for Solving the Problems]
In order to solve the above problems, the present invention employs means for solving the problems having the following characteristics.
[0015]
  The invention described in claim 1 is an original image restoration device that restores a first image on which additional information is superimposed to an original image, and an area detection unit that detects a superimpose area of the first image. An image generation unit that generates a second image from which the image information of the superimpose region is removed from the first image, and a superimpose from an image before and / or after the first image. An image acquisition unit that acquires a third image that is a superimpose area before being processed, and an original image restoration unit that combines the second image and the third image to restore the original image YesAnd when the original image is restored by overwriting the third image on the second image, the original image restoration unit performs space interpolation in the frame image. Has interpolation partIt is characterized by doing.
[0016]
  According to the first aspect of the present invention, it is possible to efficiently restore the image superimposed with the additional information to the original image before being superimposed. In addition, an image once superimposed can be effectively reused.Furthermore, when there is a pixel that cannot be restored, the original image can be restored with higher accuracy by performing spatial interpolation in the frame image.
[0017]
According to a second aspect of the present invention, the image acquisition unit includes a motion vector estimation unit that estimates a motion vector of an image from the first image, and an image before and / or after the first image. And an image extracting unit that extracts an image of the superimpose region based on the motion vector.
[0018]
According to the second aspect of the present invention, by estimating the motion vector, the image of the superimpose region in which the additional information is not superimposed is extracted from the image before and / or after the first image. can do.
[0019]
The invention described in claim 3 is characterized in that the motion vector estimation unit detects a motion vector using a camera parameter.
[0020]
According to the third aspect of the present invention, it is possible to grasp the motion of the image captured using the camera parameters, and to easily estimate the motion vector.
[0021]
The invention described in claim 4 is characterized in that the motion vector estimation unit detects a motion vector by comparing with a frame image unit.
[0022]
According to the fourth aspect of the present invention, even when there is no camera parameter, when the overall motion of the image can be grasped, the motion vector can be easily estimated by performing the comparison for each frame.
[0023]
According to a fifth aspect of the present invention, the motion vector estimation unit divides the image into predetermined pixel blocks, and detects a motion vector by comparing the corresponding pixels that are temporally different for each block. It is characterized by that.
[0024]
According to the fifth aspect of the present invention, even when there is no camera parameter and the movement of the entire image cannot be grasped, the motion vector is easily estimated by comparing each corresponding pixel that is temporally different for each block. be able to.
[0025]
The invention described in claim 6 is characterized in that the area detecting unit uses data of a display area of additional information set in advance when detecting a superimpose area.
[0026]
According to the sixth aspect of the present invention, it is possible to reduce the process of detecting the superimpose area of the first image in the area detection unit, and to efficiently detect the superimpose area.
[0029]
  Claim7The invention described in 1 is a program for causing a computer to execute a process for restoring a first image on which additional information is superimposed to an original image. An area detecting step for detecting the image, an image generating step for generating a second image obtained by removing image information of the superimpose area from the first image, and before and / or after the first image. An image acquisition step for acquiring a third image, which is a superimpose area before being superimposed, from the image, and the second image and the third image are combined to restore the original image. And execute the original image restoration step.In the original image restoration step, when there are pixels that cannot be restored when the second image is overwritten with the third image and the original image is restored, a spatial interpolation is performed in the frame image. Characterized by having an insertion stepThe
[0030]
  Claim7According to the described invention, an image on which additional information is superimposed can be efficiently restored to an original image before being superimposed. In addition, an image once superimposed can be effectively reused.Furthermore, when there is a pixel that cannot be restored, the original image can be restored with higher accuracy by performing spatial interpolation in the frame image.
[0031]
  Claim8The image acquisition step includes: a motion vector estimation step for estimating a motion vector of an image from the first image; and the motion vector from the previous and / or after images superimposed. And an image extracting step of extracting an image of the superimpose area based on the above.
[0032]
  Claim8According to the described invention, by estimating a motion vector, an image of the superimpose region in which the additional information is not superimposed can be extracted from an image before and / or after the first image. it can.
[0033]
DETAILED DESCRIPTION OF THE INVENTION
According to the present invention, a state where an area where an additional information is superimposed in an image (region) is not superimposed is displayed in an image in which the movement of the moving image moves back and forth in time other than the corresponding frame image. The image area before being superimposed is obtained by acquiring an image area that is not imposed at the same position from another image at the superimposed position and combining (overwriting) the image area. The main purpose is to restore the image.
[0034]
Here, in order to easily understand the contents of the present invention, an example of an image will be shown and described.
[0035]
FIG. 2 is a diagram showing an example of an image for explaining the original image restoration method in the present invention.
[0036]
In FIG. 2, for example, when there is an image (first image) in which a “temporary news” telop is superimposed in a moving image, a telop part extraction process for extracting a telop part region from the image, Non-telop generation processing (FIG. 2 (1)) for generating an image (second image) obtained by removing the telop portion from the first image, and a large-screen image from a plurality of image frames before and after the image. A telop area cutout process ((2) in FIG. 2) is created and cut out the image area (third image) at the same position that is not superimposed and corresponds to the telop area, and the second image And the third image can be combined to restore the original image. As a result, the original image can be easily and efficiently restored, so that the image can be used effectively in other programs, and the viewer who has viewed the re-used moving image can also view it without any discomfort. .
[0037]
Note that the original image restoration method for restoring the original image in the present invention is not limited to this. For example, in the region extraction processing described above, the region extracted by the telop portion extraction processing is included in one frame. In some cases, it is not necessary to create a large screen image, and only the frame image immediately before the corresponding frame image is used to perform the telop area extraction process, and the original image is restored using the extracted telop area portion. Can do.
[0038]
Next, embodiments of the present invention will be described with reference to the drawings. For ease of explanation, the explanation is based on an image in which a telop, which is one of the additional information in the present invention, is superimposed, but the same processing can be performed with other additional information.
[0039]
FIG. 3 is a block diagram of an example of the original image restoration apparatus in the present invention.
[0040]
3 includes a cut detection unit 11, a motion estimation unit 12, a telop part extraction unit 13, an image update unit 14, an image storage unit 15, a telop cutout unit 16, and a non-telop. The partial generation unit 17, the image synthesis unit 18, and the in-frame pixel compensation unit 19 are configured.
[0041]
In FIG. 3, the superimposed image signal is supplied to the cut detection unit 11, the motion estimation unit 12, the telop part extraction unit 13, the image update unit 14, and the non-telop part generation unit 17.
[0042]
The cut detection unit 11 detects whether or not the superimposed image frame is a frame immediately after moving image cut switching, and outputs the result. Note that the detection method of cut switching can be detected by, for example, calculating a difference value from the previous frame image and determining that cut switching has been performed when the difference value exceeds a predetermined threshold. it can.
[0043]
The motion estimation unit 12 receives the superimposed image signal and the cut detection signal output from the cut detection unit 11. The motion estimation unit 12 performs image motion estimation in accordance with, for example, a motion estimation process flowchart shown in FIG.
[0044]
In FIG. 4, first, it is determined whether or not there is a camera parameter at the time of image capture (S11). Here, the camera parameters are obtained by parameterizing camera movements such as panning indicating horizontal movement of the camera used during photographing, tilting and zooming indicating vertical movement, and the like.
[0045]
If the camera parameter is present in S11 (YES in S11), the camera parameter is converted into motion data that matches an image motion model, for example, a perspective projection transformation or an affine transformation model. The frame image is output as motion vector data (S12).
[0046]
If no camera parameter exists at S11 (NO at S11), a motion vector is estimated from the image. For estimation of the motion vector from the image, it is determined whether or not to perform global motion estimation (S13). In S13, if the entire image is moving like panning or tilting of the camera (YES in S13), global motion estimation is performed in which a motion vector is estimated in units of frame images, and there is a camera parameter. As in the case, motion vector data matching the motion model of the image is estimated and output (S14).
[0047]
Note that, as a method of global motion estimation, for example, Jozawa, Kamikura, Segata, Kotera, Watanabe, “Two-stage motion compensation using adaptive MC and local affine MC”, I Circuit and Systems for VideoTech. , Vol. 7, no. 1, pp. 75-85, Jan. 1997 and the like.
[0048]
On the other hand, if there are various movements in the image and the movement of the entire image cannot be specified, global movement estimation is not performed (NO in S13), and the image is 16 × 16 pixels or 8 × 8 pixels. The block is divided into large size blocks, and motion estimation is performed in units of these blocks, and motion vectors in units of blocks are estimated and output (S15). In addition, a general block matching method, a gradient method, etc. can be utilized for the estimation of the motion vector per block.
[0049]
With the above-described contents, the motion estimation unit 12 can output the motion vector of the input image signal.
[0050]
Next, when the superimposed image signal is input to the telop portion extraction unit 13, for example, according to the processing flowchart of the telop portion extraction as shown in FIG. Extract a telop-shaped area.
[0051]
In FIG. 5, when a telop shape has already been extracted in the previous frame and there is no change compared with the telop shape extracted in the previous frame (YES in S21), a telop shape region is shown. The extracted mask image is not used, and the already extracted mask image is used as it is.
[0052]
In S21, if the frame shape is not extracted in the previous frame or if the frame shape is extracted but the shape has changed (NO in S21), the telop shape pattern is predetermined. It is determined whether or not this pattern is met (S22).
[0053]
In S22, when the telop shape pattern corresponds to a predetermined pattern (YES in S22), for example, a number between 0 and 9, and a colon “ If it can be specified that the telop shape is a combination of “:”, the shape area data 21 is stored in advance as a predetermined pattern, and the telop shape area corresponding to the telop pattern is determined by referring to the shape area data 21. The extracted result is output as a mask image indicating the telop shape (S23).
[0054]
Note that the area is detected by pattern matching using the above-described motion vector motion vector estimation method. Also, patterns can be set in advance and stored in shape area data, and superimposing on the program editing side to match the pattern makes it possible to easily extract telop-shaped areas as well as various other than time signals. It can also handle telop patterns.
[0055]
If the telop-shaped pattern does not correspond to the predetermined pattern in S22 (NO in S22), the user confirms the image on the display or the like, and newly sets the telop-shaped area, The region is cut out and output as a mask image showing a telop shape (S24).
[0056]
Next, the image update unit 14 includes a superimposed image signal, a cut detection signal detected by the cut detection unit 11, a motion vector signal output by the motion estimation unit 12, and a telop part extraction unit. The telop-shaped mask image signal output at 13 and the reference image signal output from the image storage unit 15 are input.
[0057]
For example, the image update unit 14 updates a reference image used for image restoration in accordance with a reference image update processing flowchart as illustrated in FIG. 6. In FIG. 6, first, based on the cut detection signal input from the cut detection unit 11, it is determined whether or not the superimposed image signal is a frame image immediately after cut switching (S31), and the frame image immediately after cut switching is determined. If it is (YES in S31), the reference image signal input from the image storage unit 15 is the reference image of the previous cut, so the data is refreshed without being used (S32).
[0058]
Next, it is determined whether or not only the frame image immediately before the corresponding frame image is used as the reference image update process (S33). As a determination criterion, for example, for a telop that is suddenly displayed with no motion in the background image, a reference image for overwriting the telop-shaped region can be created using only the immediately preceding frame image. On the other hand, when there is a motion in the image and not all image positions in the telop area are displayed only with the immediately preceding frame, a reference image is created by generating a large screen image from a plurality of frame images. Whether or not the image is moving is determined based on the motion vector signal. In addition, the update processing method may be set in advance without performing the determination in S33.
[0059]
In S33, when only the previous frame image is used (YES in S33), the image is updated immediately before the image of the portion excluding the telop portion indicated by the telop-shaped mask image signal from the superimposed image signal. An image overwrite update process for overwriting the frame image is performed (S34). In this case, since the image size is the same as that of the input superimposed image signal, it is not necessary to perform coordinate conversion for overwriting using the motion vector signal input from the motion estimation unit 12.
[0060]
In S33, when a large-screen image signal created with a motion vector estimated as a reference image is used (NO in S33), for example, Taisenji, Ishibashi, Kobayashi, “Sprite automatic by camera motion extraction” Generation ", Science Theory, D-II, vol. J82-D-II, no. 6, PP. A large screen image update process is performed by the same method as that for creating a large screen background image by using a sprite generation technique (method) for pasting moving images using parameters disclosed in 1018-1030, 1999 and the like (S35). ).
[0061]
In this case, the image to be newly pasted (overwritten) is an image obtained by removing the telop area portion indicated by the telop-shaped mask image signal from the superimposed image signal, and is based on a coordinate system based on the motion vector signal. The image converted to.
[0062]
In addition, when creating a large-screen image signal using a motion vector obtained using camera parameters, it is described in the application “Japanese Patent Application No. 2001-334049 (background image generation apparatus and program thereof)” by the present applicant. Can be processed using the methods that have been proposed. According to this method, it is possible to generate a large screen image by accurately pasting each screen by geometric conversion of the image in real time.
[0063]
The image signal updated by the above processing (S34, S35) is output to the image storage unit 15, and the updated image is stored in the image storage unit 15.
[0064]
By storing the reference images as described above, for example, if the telop-shaped regions of the superimposed images are the same in consecutive images, the reference image generated once is changed to the telop-shaped image of the image. Until the area changes, it can be used continuously and efficient restoration processing can be performed.
[0065]
The telop cutout unit 16 receives the reference image signal stored in the image storage unit 15, the motion vector signal output from the motion estimation unit, and the telop shape mask image signal output from the telop part extraction unit 13. To do. The telop cutout unit 16 performs coordinate conversion of the reference image signal in accordance with the motion vector signal, cuts out only the portion of the image indicated by the telop shape mask image signal from the reference image, and outputs the telop corresponding portion image signal.
[0066]
The non-telop portion generation unit 17 receives the superimposed image signal and the telop-shaped mask image signal, and removes the non-telop-shaped portion represented by the telop-shaped mask image signal from the superimposed image signal. Generate and output a telop partial image signal.
[0067]
The image composition unit 18 inputs the telop corresponding portion image signal output from the telop cutout unit 16 and the non-telop partial image signal output from the non-telop portion generation. The image synthesizing unit 18 synthesizes the telop corresponding portion image signal of the non-telop partial image signal so that the telop corresponding portion image signal is overwritten, and outputs the combined image signal.
[0068]
The in-frame pixel compensation unit 19 receives the composite image signal output from the image composition unit 18. The in-frame pixel compensation unit 19 performs a process of compensating for a portion that is not interpolated because there is no reference image corresponding to the pixel position in the pixels of the composite image signal by spatial interpolation in the frame. Here, as the spatial interpolation, for example, it is possible to interpolate in units of pixels by using a padding method used in the MPEG-4 standard of the moving image encoding method.
[0069]
In addition, although the original image restoration method for restoring the original image has been described above, an original image restoration program for executing each process to restore the original image is generated, and based on the program, It is also possible to execute processing for restoring the original image.
[0070]
Here, an example of processing executed by the above-described original image restoration program will be described with reference to a flowchart.
[0071]
FIG. 7 is a flowchart of an example of processing executed by the original image restoration program.
[0072]
In FIG. 7, first, additional information part extraction processing is performed to extract additional information (for example, telop) shape part (region) from an image (first image) on which additional information is superimposed (S41). Next, based on the additional information shape part extracted in S41, non-additional information part generation processing for generating an image (second image) obtained by removing the additional information shape part from the superimposed image is performed ( S42).
[0073]
Further, a cut detection process for detecting whether or not the frame image on which the additional information is superimposed is a frame image immediately after the cut switching is performed (S43), and then from the camera parameters or the frame images before and after the image. A motion estimation process for estimating a motion vector is performed (S44). After that, based on the results output in S43 and S44, an image of the additional information shape part (third image) before being superimposed is acquired as a reference image from the frame images before and after the superimposed image. A reference image acquisition process is performed (S45). In addition, you may perform S43 and S44 in S45, without dividing a step.
[0074]
Next, an image composition process for synthesizing the image (second image) generated in S42 and the image (third image) acquired in S45 is performed (S46), and the image is restored. In S46, if there is a place where pixels are not interpolated in the output image, pixel compensation processing is performed by spatial interpolation in the frame (S47). As a result, a highly accurate original image can be restored.
[0075]
3 is executed by the above-described original image restoration program stored in the memory of the computer.
[0076]
The original image restoration program described above can be easily restored by storing it in a storage medium such as a CD-ROM and installing it in a computer. Further, as another method, the original image restoration program can be executed by acquiring the original image restoration program in a file format from the communication interface and the network and installing the program in a computer.
[0077]
As described above, by using the original image restoration apparatus 1 shown in FIG. 3 or the original image restoration program, it is possible to restore the original image before being superimposed from the image signal in which the additional information is superimposed. In addition, it is possible to easily perform effective reuse of a program using an image by using the restored original image. Furthermore, by using the restored original image in the present invention, it is possible to provide the viewer who views the program without feeling uncomfortable.
[0078]
【The invention's effect】
As described above, according to the present invention, when additional information such as a telop is superimposed on an image, a frame image other than the corresponding frame image can be used to restore the original image before being superimposed, Images can be reused efficiently.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating an example of an image when a conventional superimposed image is reused.
FIG. 2 is a diagram illustrating an example of an image for explaining an original image restoration method according to the present invention.
FIG. 3 is a block diagram illustrating an example of an original image restoration apparatus according to the present invention.
FIG. 4 is a processing flowchart example of motion estimation.
FIG. 5 is a processing flowchart example of telop portion extraction.
FIG. 6 is a flowchart illustrating an example of a reference image update process.
FIG. 7 is a flowchart illustrating an example of processing executed by an original image restoration program.
[Explanation of symbols]
1 Original image restoration device
11 Cut detector
12 Motion estimation unit
13 telop part extractor
14 Image update unit
15 Image storage unit
16 Telop cutout
17 Non-telop part generator
18 Image composition part
19 In-frame pixel compensation
21 Shape area data

Claims (8)

In the original image restoration device for restoring the first image on which the additional information is superimposed on the original image,
An area detection unit for detecting a superimpose area of the first image;
An image generation unit that generates a second image from which the image information of the superimpose area is removed from the first image, and an image before and / or after the first image are superimposed. An image acquisition unit that acquires a third image that is the previous superimpose area;
It possesses said second image, and an original image restoration unit for restoring an original image by synthesizing the said third image,
The original image restoration unit performs spatial interpolation in a frame image when there is a pixel that cannot be restored when the second image is overwritten with the third image and the original image is restored. original image restoration apparatus characterized by have a part. The image acquisition unit
A motion vector estimation unit that estimates an image motion vector from the first image;
2. The original image according to claim 1, further comprising: an image extracting unit that extracts an image of the superimpose area from an image before and / or after the first image based on the motion vector. Restore device. The motion vector estimation unit
3. The original image restoration apparatus according to claim 2, wherein a motion vector is detected using camera parameters. The motion vector estimation unit
3. The original image restoration apparatus according to claim 2, wherein a motion vector is detected by comparing with a frame image unit. The motion vector estimation unit
3. The original image restoration apparatus according to claim 2, wherein the image is divided into predetermined pixel blocks, and a motion vector is detected by comparing each corresponding pixel that is temporally different in the block unit. The region detection unit
6. The original image restoration apparatus according to claim 1, wherein data of a display area of additional information set in advance when the superimpose area is detected is used. In a program for causing a computer to execute a process for restoring a first image superimposed with additional information to an original image,
An area detecting step for detecting a superimpose area of the first image in the computer;
An image generation step of generating a second image from which the image information of the superimpose area is removed from the first image;
An image acquisition step of acquiring a third image which is a superimpose area before being superimposed from an image before and / or after the first image;
Executing an original image restoration step of restoring the original image by combining the second image and the third image ;
In the original image restoration step, when there is a pixel that cannot be restored when the second image is overwritten with the third image and the original image is restored, spatial interpolation is performed in the frame image. original image restoration program characterized Rukoto to have a step. The image acquisition step includes
A motion vector estimation step for estimating a motion vector of the image from the first image, and an image of the superimpose region is extracted based on the motion vector from images before and / or after being superimposed. The original image restoration program according to claim 7 , further comprising:

Images