JP2008227725A - Image processor, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To correct camera-shake of a moving image including an overlay display without giving a strange feeling. <P>SOLUTION: When there is an overlay display in an input image, camera-shake correction processing is applied to the input image in step S5. In step S6, it is determined whether or not field interpolation can be adopted in deleting the overlay display; when the determination is affirmative, the overlay display of the field image is deleted by field interpolation; and when it is negative, the overlay display of the field image is deleted by in-plane interpolation. This can be applied to, for example, a television receiver, video player, or the like. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing device, an image processing method, and a program. For example, the present invention relates to an image processing device, an image processing method, and a program that are suitable for correcting camera shake of a captured moving image.

  Many of the video cameras currently on the market are equipped with a camera shake correction function, which can correct a camera shake during shooting and record a corrected moving image. As a camera shake correction function provided in the video camera, there are an optical camera shake correction method, an electronic camera shake correction method, and the like (for example, see Patent Document 1).

  In the optical image stabilization method, camera shake is detected by a gyro sensor built in the video camera, and the optical axis at the time of shooting is offset according to the detection result, thereby suppressing blurring of the recorded image. Has been made. The electronic image stabilization method determines an effective portion that is narrower by several tens of scanning lines in the vertical direction than the captured image according to the position of the subject in the image, and records only the determined effective portion, The vibration of the subject in the recorded image is suppressed.

JP-A-5-316404

  By the way, when a moving image recorded by a video camera that does not have a camera shake correction function is reproduced and displayed, it may be difficult to see the image shaking or rotating up and down and left and right due to camera shake. In order to eliminate such inconvenience, the reproduced moving image is subjected to image processing similar to the above-described electronic camera shake correction (hereinafter referred to as camera shake correction processing at the time of reproduction), so that the subject in the image is corrected. A method for suppressing vibration is conceivable.

  The obstacle to the camera shake correction processing during reproduction is the presence of a character string (hereinafter referred to as superimposed display) indicating the shooting date and time superimposed on the captured moving image. For example, as shown in FIG. 1A, FIG. 1B, and FIG. 1C, when there is a superimposed display on a moving image in which camera shake occurs, as shown in the continuous field images f1, f2, and f3, when electronic camera shake correction processing is performed on this, As shown in FIGS. 2A to 2C.

  That is, an effective portion corresponding to the position of the subject (in this case, a house) is extracted from each of the field images f1, f2, and f3, and field images f1 ', f2', and f3 'are generated. As a result, the field image f1 'does not include superimposed display, the field image f2' remains superimposed, and the field image f3 'partially remains superimposed. As a result, although the vertical vibration of the subject is reduced, the position of the superimposed display (in this case, 1995/2/1/10: 30) changes or disappears (hereinafter, the superimposed display is “Swimming”). It is also possible to take some countermeasures.

  In general video cameras, information such as the shooting date and time is recorded separately from the moving image in association with the moving image, and is output by being superimposed on the moving image as necessary during playback. It seems that the above-mentioned problem is depressing.

  However, some conventional video cameras record a character string such as a shooting date and time superimposed on a moving image for the purpose of cost reduction, and the shooting date and time are superimposed from a general video camera. If the input source is a moving image with camera shake that is output in the state, or if the input source is broadcast content with camera shake that is superimposed on the image, such as a broadcast station logo or broadcast time Therefore, it is essential to take measures against the superimposed display when the camera shake correction process during reproduction is performed.

  The present invention has been made in view of such a situation, and makes it possible to correct a subject's vibration caused by camera shake of a moving image on which superimposed display exists without a sense of incongruity.

  An image processing apparatus according to an aspect of the present invention is an image processing apparatus that corrects vibration of a subject in a moving image, and detects a superimposed display that is superimposed on the subject of the moving image and exists at a fixed position on a screen. Detecting means; camera shake correcting means for correcting the vibration of the subject in the moving image; and erasing means for erasing the superimposed display detected by the detecting means from the moving image corrected by the correcting means.

  The erasing unit includes a region where the superimposed display of the still image where the superimposed display exists is included in a series of still images constituting the moving image corrected by the correcting unit. The superposition display can be erased by replacing the corresponding area of the image.

  The erasing unit performs a predetermined filtering process on a region where the superimposed display of the still image where the superimposed display is present among a series of still images constituting the moving image corrected by the correcting unit. The superimposed display can be erased.

  An image processing method according to an aspect of the present invention is an image processing method of an image processing apparatus that corrects vibration of a subject in a moving image, and is superimposed on the subject of the moving image and exists at a fixed position on a screen. A detection step for detecting a display; a camera shake correction step for correcting vibrations of a subject in the moving image; and the superimposed display detected by the process of the detection step from the moving image corrected by the process of the correction step. An erasing step for erasing.

  A program according to an aspect of the present invention is a computer control program that corrects vibration of a subject in a moving image, and is superimposed on the subject of the moving image and is present at a fixed position on a screen. A detection step for detecting the image, a camera shake correction step for correcting the vibration of the subject in the moving image, and the superimposed display detected by the processing of the detection step is erased from the moving image corrected by the processing of the correction step. And causing the computer to execute a process including an erasing step.

  In one aspect of the present invention, a superimposed display that is superimposed on a subject of a moving image and is present at a fixed position on the screen is detected, vibration of the subject in the moving image is corrected, and the corrected moving image The detected superimposed display is erased.

  According to one aspect of the present invention, it is possible to correct vibration of a subject in a moving image caused by camera shake during shooting.

  In addition, according to one aspect of the present invention, it is possible to erase the superimposed display without a sense of incongruity while correcting the vibration of the subject of the moving image due to camera shake during shooting.

  Embodiments of the present invention will be described below. Correspondences between constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  An image processing apparatus according to one aspect of the present invention (for example, the image processing apparatus 10 in FIG. 3) is a detection unit (for example, a superimposed display that is superimposed on a subject of a moving image and exists at a fixed position on the screen). 3), a camera shake correction unit (for example, the camera shake correction unit 13 in FIG. 3) that corrects the vibration of the subject in the moving image, and a moving image that has been corrected by the correction unit. Erasing means for erasing the detected superimposed display (for example, the superimposed display erasing unit 17 in FIG. 3).

  An image processing method and a program according to one aspect of the present invention include a detection step (for example, step S2 in FIG. 5) for detecting a superimposed display that is superimposed on a subject of a moving image and exists at a fixed position on the screen. A camera shake correction step (for example, step S5 in FIG. 5) for correcting the vibration of the subject in the moving image, and an erasing step for erasing the superimposed display detected by the detection step process from the corrected moving image by the correction step process (For example, step 7 or S8 in FIG. 5).

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.

  FIG. 3 shows a configuration example of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus 10 corrects camera shake at the time of shooting an input interlaced moving image signal (hereinafter referred to as an input image) and erases the superimposed display. The camera shake correction unit 13, the superimposed display detecting unit 15, and the superimposed display erasing unit 17.

  The camera shake correction execution determination unit 11 has a built-in buffer 12 that can hold several field images constituting the input image. Based on the field image held in the buffer 12, the camera shake correction execution determination unit 11 performs the camera shake correction process. It determines whether or not, and notifies the camera shake correction unit 13 of the determination result.

  The determination by the camera shake correction execution determination unit 11 divides a series of moving images for each scene based on inter-field correlation, histogram change, and the like, and performs determination for each scene. For example, when the vibration of the subject in the field image is within a range in which the camera shake correction process is possible (for example, about several tens of scanning lines), it is determined that the camera shake correction process is performed. For example, when the vibration of the subject in the field image is extremely small, it is determined that the camera shake correction process is not performed because the camera shake correction process is unnecessary. Further, for example, when the movement of the subject is too fast, the effect of the camera shake correction process is not achieved, so it is determined that the camera shake correction process is not performed.

  The camera shake correction unit 13 has a built-in buffer 14 that can hold several field images constituting the input image, and corresponds to the determination result of the camera shake correction execution determination unit 11 with reference to FIGS. 1 and 2. The above-described conventional camera shake correction processing is performed on the input image and output to the superimposed display erasure unit 17. When the camera shake correction process is not performed, the camera shake correction unit 13 outputs the input image to the superimposed display erasing unit 17 as it is.

  The superimposed display detection unit 15 has a built-in buffer 16 that can hold several field images constituting the input image, and continues to exist at a fixed position on the screen of the input image and does not move (shooting date and time). , A broadcasting time, a character string or mark such as a logo of a broadcasting station) is detected, and the detection result (presence / absence of superimposed display, position, size, etc.) is notified to the superimposed display erasure unit 17.

  For the detection of superimposed display by the superimposed display detection unit 15, a method based on image correlation using a symmetric phase only matched filter (SPOMF), a method based on a frequency spectrum by FFT (Fast Fourier Transforms), or a connection is used. Any of these methods or a combination of these methods can be adopted.

  In the method based on image correlation using SPOMF, since a superimposed display that does not move on the screen and a subject that moves on the screen (background of the superimposed display) each show a peak of a correlation value between adjacent field images, The superimposed display is detected using the peak of the correlation value. The phase-only filter is described in, for example, the literature “Symmetrical Phase-Only Matched Filtering of Fourier-Mellin Transforms for Registration and Recognition”, IEEE Transaction on Pattern analysis and Machine Intelligence, VOL.16 No.12 December 1994, etc. Yes.

  In the method based on the frequency spectrum by FFT, the superimposed display is detected by using the characteristic of the frequency spectrum of the portion where the superimposed display exists.

  The method for specifying the connected image output apparatus depends on a connection interface with an image output apparatus (not shown) that supplies an input image to the image processing apparatus 10. For example, when the connection interface is IEEE1394, the image processing apparatus Since 10 can identify the model and model of the image output apparatus that is the connection partner, the identified model and model number are used. Specifically, a database in which the position, size, brightness, etc., where the superimposed display is displayed corresponding to the model, model number, etc. is prepared in advance, and the database is referred to according to the identified model, model number, etc. As a result, the presence / absence, position, size, brightness, etc. of the superimposed display are acquired.

  Note that methods other than the above three examples may be adopted for detection of the superimposed display by the superimposed display detection unit 15.

  The superimposed display erasure unit 17 includes a buffer 18 that can hold several field images input from the camera shake correction unit 13. Based on the notification result of the superimposed display notified from the superimposed display detection unit 15, camera shake correction is performed. The superimposed display existing in the field image input from the unit 13 is deleted.

  As a method of erasing the superimposed display, a rectangular area including the superimposed display (an area having another shape may be blurred) using a low-pass filter, or a rectangular area including the superimposed display (an area having another shape may be used). Is replaced with another rectangular region, or a method of replacing only the superimposition display region (only the region within the outline of characters or the like) with another region.

  Note that the method of blurring a rectangular area including a superimposed display using a low-pass filter has a drawback that the superimposed display cannot be completely erased and the background of the superimposed display is blurred.

  A method of replacing a rectangular region including superimposed display with another rectangular region is a method of using a nearby region in the same field image as a rectangular region used for replacement (hereinafter referred to as in-plane interpolation), There are two types of methods (hereinafter referred to as field interpolation) in which regions of corresponding positions of different field images held in time are retained.

  In the field interpolation, for example, when the field images f1 ′ to f3 ′ shown in FIG. 2 are held in the buffer 18, the region including the superimposed display of the field images f2 ′ and f3 ′ is displayed in the superimposed display shown in FIG. 4A. By substituting with the diversion area 21 of the nonexistent field image f1 ′, as shown in FIGS. 4B and 4C, the superimposed display of the field images f2 ′ and f3 ′ is deleted, and the field images f2 ″ and f3 ″ are generated. To do.

  There are two types of methods for replacing only the superimposed display area with other areas, as in the method for replacing the rectangular area including the superimposed display with another rectangular area, in-plane interpolation and field interpolation.

  When in-plane interpolation and field interpolation are compared, field images that are close in time are similar, and it is considered that field interpolation results in higher image quality after the superimposed display is erased. Therefore, field interpolation is basically employed, and in-plane interpolation is employed when field interpolation is not possible.

  A field image used for field interpolation (hereinafter referred to as a diverted field image) is determined based on the following criteria. In other words, the diverted field image is as close in time as possible to the field image from which the superimposed display is erased, and there is no superimposed display in the diverted area and is not blurred. A field image for which there is no scene change and the superimposed display is erased and an image having no significant change in brightness are adopted. If there is no field image that satisfies such conditions, in-plane interpolation is adopted.

  Next, camera shake correction processing by the image processing apparatus 10 will be described with reference to the flowchart of FIG.

  The interlaced moving images supplied from the preceding image output apparatus are input to the camera shake correction execution determination unit 11, the camera shake correction unit 13, and the superimposed display detection unit 15, respectively.

  In step S <b> 1, the camera shake correction execution determination unit 11 determines whether or not to perform the camera shake correction process on the input image based on the field image held in the buffer 12, and notifies the camera shake correction unit 13 of the determination result. . If it is determined that the camera shake correction process is to be performed, the process proceeds to step S2.

  In step S2, the superimposed display detection unit 15 determines whether or not there is a superimposed display in the input image. If it is determined that there is no superimposed display in the input image, the process proceeds to step S3. In step S3, the superimposed display detecting unit 15 notifies the superimposed display erasing unit 17 that there is no superimposed display in the input image.

  In step S <b> 3, the camera shake correction unit 13 performs a camera shake correction process on the input image and outputs the input image to the superimposed display erasing unit 17. Thereafter, the process proceeds to step S9. In step S9, the superimposed display erasure unit 17 outputs the field image subjected to the camera shake correction process input from the camera shake correction unit 13 to the subsequent stage.

  On the other hand, if it is determined in step S2 that superimposed display exists in the input image, the process proceeds to step S4. In step S <b> 4, the superimposed display detection unit 15 detects the position and size of the superimposed display and notifies the superimposed display erasing unit 17.

  In step S <b> 5, the camera shake correction unit 13 performs a camera shake correction process on the input image and outputs the input image to the superimposed display erasing unit 17 as in the process of step S <b> 3 described above.

  In step S <b> 6, the superimposed display erasing unit 17 determines whether or not field interpolation can be adopted when erasing the superimposed display of the field image input from the camera shake correction unit 13. When it is determined that the field interpolation can be adopted, the superimposed display erasure unit 17 advances the process to step S7, erases the superimposed display of the field image by the field interpolation, and in step S9, deletes the field image from which the superimposed display is erased. Output to.

  Conversely, if it is determined in step S6 that field interpolation cannot be employed, the process proceeds to step S8. In step S8, the superimposed display erasure unit 17 erases the superimposed display of the field image by in-plane interpolation, and outputs the field image from which the superimposed display is erased in step S9.

  If it is determined in step S1 that the camera shake correction process is not performed on the input image, steps S2 to S8 are skipped and the process proceeds to step S9. In step S9, the camera shake correction unit 13 outputs the input image as it is to the superimposed display erasing unit 18, and the superimposed display erasing unit 17 also outputs the input image as it is to the subsequent stage. This is the end of the description of the camera shake correction process performed by the image processing apparatus 10.

  According to the camera shake correction process performed by the image processing apparatus 10, since the superimposed display is erased from the field image after the camera shake correction, the superimposed display does not swim on the output image. Therefore, it is possible to provide an output image that is easy to view without camera shake.

  In the present embodiment, an interlaced moving image is used as an input image. However, a progressive moving image is used as an input image, and a frame image constituting the moving image may be processed. Good.

  In addition, as in the present embodiment, the present invention may not only process a moving image that already exists, but may also process a moving image output in real time from a video camera, for example.

  The present invention can be applied to, for example, a television receiver, a video player, a video camera, and the like. Further, the present invention can also be applied to a case where a photographic film or broadcast content shot and stored in the past is edited and stored again as digital data.

  By the way, the series of processes described above can be executed by hardware configured as shown in the block diagram of FIG. 3 or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 6 is a block diagram illustrating a hardware configuration example of a computer that executes the above-described series of processing by a program.

  In the computer 50, a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, and a RAM (Random Access Memory) 53 are connected to each other by a bus 54.

  An input / output interface 55 is further connected to the bus 54. The input / output interface 55 includes an input unit 56 including a keyboard and a mouse, an output unit 57 including a display and a speaker, a storage unit 58 including a hard disk and a nonvolatile memory, a communication unit 59 including a network interface, and a magnetic field. A drive 60 for driving a removable medium 61 such as a disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer 50 configured as described above, for example, the CPU 51 loads the program stored in the storage unit 58 to the RAM 53 via the input / output interface 55 and the bus 54 and executes the program. A series of processing is performed.

  Note that the program executed by the computer 50 may be a program that is processed in time series in the order described in this specification, or a necessary timing such as in parallel or when a call is made. It may be a program in which processing is performed.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a figure which shows the example of the field image with a camera shake and a superimposed display. It is a figure which shows the example which performed only the camera shake correction to the field image of FIG. It is a block diagram which shows the structural example of the image processing apparatus to which this invention is applied. It is a figure which shows the example which performed the camera-shake correction process which applied this invention to the field image of FIG. It is a flowchart explaining the camera-shake correction process to which this invention is applied. And FIG. 11 is a block diagram illustrating a configuration example of a general-purpose computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Image processing apparatus, 11 Camera shake correction execution determination part, 12 Buffer, 13 Superimposition display detection part, 14 Buffer, 15 Camera shake correction part, 16 Buffer, 17 Superimposition display deletion part, 18 Buffer

Claims (5)

In an image processing apparatus that corrects vibration of a subject in a moving image,
Detecting means for detecting a superimposed display superimposed on a subject of the moving image and existing at a fixed position on a screen;
Camera shake correcting means for correcting the vibration of the subject in the moving image;
An erasing unit for erasing the superimposed display detected by the detecting unit from the moving image corrected by the correcting unit. The erasing unit includes a region where the superimposed display of the still image where the superimposed display exists is included in a series of still images constituting the moving image corrected by the correcting unit. The image processing apparatus according to claim 1, wherein the superimposed display is erased by performing replacement using a corresponding region of the image. The erasing unit performs a predetermined filtering process on a region where the superimposed display of the still image where the superimposed display is present among a series of still images constituting the moving image corrected by the correcting unit. The image processing apparatus according to claim 1, wherein the superimposed display is erased. In an image processing method of an image processing apparatus for correcting vibration of a subject in a moving image,
A detection step of detecting a superimposed display superimposed on a subject of the moving image and existing at a fixed position on a screen;
A camera shake correction step for correcting vibration of a subject in the moving image;
An erasing step of erasing the superimposed display detected in the detection step from the corrected moving image in the correction step. A computer control program for correcting vibration of a subject in a moving image,
A detection step of detecting a superimposed display superimposed on a subject of the moving image and existing at a fixed position on a screen;
A camera shake correction step for correcting vibration of a subject in the moving image;
A program that causes a computer to execute a process including: an erasing step of erasing the superimposed display detected in the process of the detection step from the moving image corrected by the process of the correction step.

Images