JP2005303738A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a high-efficiency encoding technique having high versatility without requiring preprocessing for setting a background image. <P>SOLUTION: An image processing apparatus has a moving image inputting means (2101) for inputting a moving image consisting of a plurality of frames; an image selecting means (101) for selecting a frame to be a background, irrespective of the presence or absence of an object in an image in selecting the frame to be the background image from among the moving images; a shape data generating means (103) for comparing the frame to be the background image with a frame of an input image, and generating shape data on the basis of the difference value; a background data correcting means (104) for correcting the background image on the basis of the shape data; and an arbitrary shape image encoding means (2105) for encoding the input image as an arbitrary shape image together with the shape image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing technique, and more particularly to an image processing technique for encoding a moving image.

  In recent years, attention has been focused on processing for separating and synthesizing images for each object (object) using digital technology. In particular, in moving picture coding, the MPEG-4 coding system is standardized as an international standard. In the MPEG-4 encoding method, image data of an arbitrary shape can be handled, and encoding / decoding is performed for each object, thereby improving encoding efficiency, data distribution according to a transmission path, image reprocessing, and the like. Various applications that were difficult are expected.

  As a method for extracting an object in moving image processing, a method called a background difference method is generally known. In this method, a change point is detected by comparing a background image captured in advance with an actual input image. The principle will be briefly described below.

  First, assume that the pixel value of the input image at the point at the coordinates (x, y) on the image plane is Pc (x, y), and the pixel value of the background image is Pb (x, y). At this time, the difference between Pc (x, y) and Pb (x, y) is taken and the absolute value is compared with a certain threshold Th.

An example of the determination formula is as follows.
if (| Pc (x, y) −Pb (x, y) | ≦ Th) S (x, y) = 0;
else S (x, y) = 1; (1)

  When the absolute difference value is equal to or smaller than the threshold Th, this point (x, y) is unchanged, and Pc (x, y) is determined as the background, and S (x, y) = 0. On the other hand, when the difference absolute value exceeds the threshold Th, it is regarded as an extraction target because the value has changed, and S (x, y) = 1. By performing the above determination at all points on the screen, extraction for one frame is completed.

  FIG. 15 is a block diagram showing the configuration of a conventional system that combines the background difference method and the MPEG-4 encoding method. In FIG. 15, an image input unit 2101 is an imaging unit of a camera, for example, and is a part for inputting a moving image. The image separation unit 2102 is a switch circuit that switches between a frame to be processed as a background image and a frame to be processed as an arbitrary shape image. The frame set as the background image is encoded for one frame by the rectangular image encoding unit 2104. The shape data generation unit 2103 generates shape data S (x, y) by comparing the previous background image with the currently input image. In general, the shape data S (x, y) is binary data indicating whether or not the object is an object. The arbitrary shape image encoding unit 2105 receives image data and shape data as input, and outputs the encoded result as a bit stream. In addition to the two types of bit streams of the rectangular image and the arbitrary shape image shown in the figure, the multiplexing unit 2106 performs a multiplexing process so as to combine the audio bit stream and the like into one data.

  FIG. 17 is a diagram for explaining the block diagram of FIG. 15 more specifically. Frames 2301 to 2318 are data sequences of frames input from the image input unit 2101. The top frame 2301 is an image in which only the background is shown, and the frames 2311 and after are images in which an extraction target is also shown. In the image separation unit 2102, the frame 2301 is switched to the background image, and the frames after the frame 2311 are switched to the arbitrary shape image. The simplest method of switching is surely operated manually while viewing the input image.

  The actual data processing flow will be described with reference to FIG. Assume that the image 2500 corresponds to the background image 2301 and the image 2501 corresponds to the arbitrary shape image 2311. At this time, the images 2500 and 2501 are input to the difference processing unit 2351, and binarized data is output according to whether the difference between corresponding pixels is equal to or less than a threshold value by the background difference method described above. The shape data 2511 is binarized shape data, and the black portion indicates the background and the white portion indicates the object. Similarly, if the images 2502 and 2503 correspond to the frames 2312 and 2313, respectively, the generated shape data becomes the shape data 2512 and 2513 as a result of the difference threshold processing with the background image 2500.

  In FIG. 17, an MPEG-4 Core Profile encoder 2353 is used for arbitrary shape image coding. Hereinafter, this encoding method will be described.

  When encoding an object (object), it is necessary to encode information on the shape and position of the object. For this purpose, first, a rectangular area containing the object is set, and the coordinates of the upper left position of the rectangle and the size of the rectangular area are encoded. This rectangular area is called a bounding box. A region inside the object expressed by image data and shape data is called a VOP (Video Object Plane).

  FIG. 21 is a block diagram illustrating a detailed configuration of the encoding unit 2353 of FIG. The input data is image brightness / color difference data and shape data, which are processed in units of macroblocks.

  First, in the intra mode, each block is subjected to discrete cosine transform (DCT) in the DCT unit 2701 and quantized in the quantization unit 2702. The quantized DCT coefficient and the quantization width are variable length encoded by the variable length encoding unit 2712. In addition, in order to generate a reference image used in the inter mode, the once quantized data is returned to the image data through the inverse quantization unit 2703 and the inverse DCT unit 2704. This is also called a locally decoded image. This image is stored in the memory unit 2705.

  On the other hand, in the inter mode, the motion detection unit 2707 detects a motion from another temporally adjacent VOP stored in the memory unit 2705 by a motion detection method such as block matching, and the motion vector prediction unit 2708. The prediction macroblock with the smallest error is detected with respect to the target macroblock. Data indicating the motion to the prediction macroblock with the smallest error is a motion vector. An image referred to for generating a prediction macroblock is referred to as a reference VOP.

  Based on the detected motion vector, the motion compensation unit 2706 performs motion compensation on the reference VOP to obtain an optimal prediction macroblock. Next, the difference between the target macroblock and the corresponding predicted macroblock is obtained, the DCT is applied to the difference image by the DCT unit 2701, and the DCT transform coefficient is quantized by the quantization unit 2702. Also at this time, the quantized data is returned to the image data through the inverse quantization unit 2703 and the inverse DCT unit 2704. Since the output of the inverse DCT unit 2704 at this time is a difference image, it is stored in the memory unit 2705 after being combined with the previous image.

  On the other hand, the shape data is encoded by the shape encoding CAE unit 2709. However, only the boundary block is actually subjected to CAE encoding here, and blocks in the VOP (all data in the block is in the object) and blocks outside the VOP (all data in the block is outside the object) Only the header information is sent to the variable length coding unit 2712. In the inter mode, the motion detection unit 2707 performs motion detection on the boundary block on which CAE encoding is performed, and the motion vector prediction unit 2708 predicts motion vectors in the inter mode. Then, CAE encoding is performed on the difference value between the shape data subjected to motion compensation and the shape data of the previous frame.

  In FIG. 17, an MPEG-4 Simple Profile encoder 2352 is used for rectangular image coding. Hereinafter, this encoding method will be described. The Simple Profile encoder 2352 is backward compatible with the Core Profile encoder 2353. In FIG. 21, processing relating to an arbitrary shape, that is, a shape profile CAE unit 2709, a memory unit 2710, and a motion compensation unit 2711 are removed as an encoder of a simple profile. The processing procedure of the image data is the same as the processing of the image data of the Core Profile. Since the background image only needs to be encoded for one frame, the moving image encoding method is not necessarily used, and the still image encoding method may be used.

  The MUX processing unit 2354 corresponds to the multiplexing unit 2106 and performs multiplexing.

Next, processing on the decoding side will be described.
FIG. 16 is an overall schematic block diagram. The bitstreams combined into one on the encoding side are separated into bitstreams that can be input by the decoders in the separation unit 2201. Among these, the encoded background image is decoded into one frame of image data by the rectangular image decoding unit 2202. The arbitrary shape image decoding unit 2203 decodes shape data and image data corresponding to the shape data. The image composition unit 2204 generates a composite image by switching the background image and the arbitrary shape image in units of pixels based on the value of the shape data. The image output unit 2205 is generally an image display device such as a monitor.

  The block diagram of FIG. 16 will be described more specifically with reference to FIGS. 18 and 20. The separation unit 2201, the rectangular image encoding unit 2202, the arbitrary shape encoding unit 2203, and the image composition unit 2204 in FIG. This corresponds to the decoder 2453 and the composition processing unit 2454. Frames 2411 to 2418 are data strings of frames displayed in the image output unit 2205 and correspond to the input images 2311 to 2318 in FIG.

  The output of the MPEG-4 Simple Profile decoder 2452 is the background image 2600 in FIG. Since the background image is first decoded by one frame, the decoder may be a still image decoding method. Further, since the background image is always combined with another image, the background image is not output as it is.

  The MPEG-4 Core Profile decoder 2453 first outputs shape data 2601 and image data 2611. In the composition processing 2454, a pixel of the background image 2600 is selected for the pixel determined to be the background in the shape data 2601, and a pixel of the image data 2611 is selected for the pixel determined to be the object to generate the composite image 2621. To do. This image corresponds to the image 2501 on the encoding side. A decoded image corresponding to the image 2502 is synthesized from the shape data 2602, the image data 2612, and the decoded background image 2600 to become an image 2622. Similarly, the decoded image corresponding to the image 2503 is an image 2623.

  Details of the MPEG-4 Core Profile decoder 2453 will be described with reference to FIG. Basically, the operation is the reverse of FIG. 21, and the luminance / color difference data and shape data of the image are decoded in units of macroblocks.

  First, in the intra mode, the variable length decoding unit 2801 decodes the quantized DCT coefficient and inputs it to the inverse quantization unit 2802. The output of the inverse quantization unit 2802 becomes the decoded DCT coefficient and becomes the input of the inverse DCT unit 2803. The inverse DCT unit 2803 outputs a decoded image by performing inverse DCT processing. The image at this time is stored in the memory unit 2804 so as to be a reference image used in the inter mode.

  On the other hand, in the inter mode, an image decoded through the inverse quantization unit 2802 and the inverse DCT unit 2803 is a difference image between frames. The motion vector decoding unit 2806 decodes the motion vector. The motion compensation unit 2805 uses the decoded motion vector to generate a motion compensated image from the previous frame image stored in the memory unit 2804. By synthesizing this image and the previous difference image, the image in the inter mode is decoded.

  Further, the shape data is decoded from the variable length decoding unit 2801 through the shape decoding CAE unit 2807. In the case of the inter mode, the motion compensation unit 2809 performs motion compensation using the motion vector obtained by decoding the shape data of the previous frame stored in the memory unit 2808 by the motion vector decoding unit 2806, and then performs shape decoding CAE. Decoded by the unit 2807.

  In FIG. 18, an MPEG-4 Simple Profile decoder 2452 is used for rectangular image decoding. Hereinafter, this encoding method will be described. The Simple Profile decoder 2452 is backward compatible with the Core Profile decoder 2453. In FIG. 22, processing relating to an arbitrary shape, that is, a shape decoding CAE unit 2807, a memory unit 2808, and a motion compensation unit 2809 are removed as a simple profile decoder 2452. The processing procedure of the image data is the same as the processing of the image data of the Core Profile. Since the background image only needs to be decoded for one frame, it is not always necessary to use a moving image decoding method, and a still image decoding method may be used.

  However, the system described above has a drawback in that an image of only the background must be prepared in advance. There is also a problem that an object cannot be correctly extracted if a relative position shift occurs between the input image and the background image. If the camera movement range is known in advance, it is possible to take some measures by preparing a wide range of images called sprites as disclosed in Patent Document 1 below. There is no change in what is necessary.

  In other words, the above system is a good configuration for obtaining accurate extraction results, but the range of adaptation is limited for the purpose of improving the encoding efficiency, so that a rectangular moving image encoding is simply performed. There was a problem that the system could not be replaced with a video coding system of arbitrary shape.

JP 2002-118843 A

  The present invention has been made in consideration of such circumstances, and does not require pre-processing, and can be simply replaced with a rectangular moving image encoding system. The purpose is to provide.

The image processing apparatus according to the present invention detects a moving image input unit that inputs a moving image composed of a plurality of frames, and whether or not there is a subject in the image when selecting a frame as a background image from the moving images. Regardless of the image selection means for selecting a frame as a background image, the shape data generation means for comparing the frame as the background image and the frame of the input image, and generating shape data based on the difference value; The image processing apparatus includes a background data correcting unit that corrects the background image based on shape data, and an arbitrary shape image encoding unit that encodes the input image together with the shape data as an arbitrary shape image.
The image processing method according to the present invention also includes a moving image input step of inputting a moving image composed of a plurality of frames, and a selection of a frame as a background image from the moving images. An image selection step for selecting a frame as a background image regardless of the presence, a shape data generation step for comparing the frame as the background image with the frame of the input image, and generating shape data based on the difference value; A background data correction step for correcting the background image based on the shape data, and an arbitrary shape image encoding step for encoding the input image as an arbitrary shape image together with the shape data.

The program of the present invention is a program for causing a computer to execute each step of the image processing method.
The recording medium of the present invention is a computer-readable recording medium recording a program for causing a computer to execute each step of the image processing method.

  Since a frame as a background image can be selected regardless of the presence or absence of a subject in the image, pre-processing for setting the background image is not necessary. Further, the rectangular image encoding means is not required, and the encoding can be performed only by the arbitrary shape image encoding means. Further, by correcting the background data, it is possible to suppress the influence of moving image noise and minute fluctuations, and it is possible to obtain a more natural image even in a composite image after decoding.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.
<First Embodiment>
A first embodiment of the present invention will be described. FIG. 1A is a block diagram showing an overall configuration on the encoding side. The major difference from FIG. 15 described in the background art is that the selection method of the background image and the arbitrary shape image is different (image selection unit 101), and the background data correction unit 104 that reflects the shape data after generation is provided. That is, the rectangular encoding unit 2104 is unnecessary. A major feature of this embodiment is that a background image corresponding to 2301 in FIG. 17 is not prepared in advance. In the present embodiment, a background image is selected from the frame sequence and encoded.

  A moving image composed of a plurality of frame sequences input from the image input unit 2101 is selected by the image selection unit 101 as a background frame and other frames. At that time, a frame as a background is selected regardless of the presence or absence of a subject in the image. The shape data generation unit 103 inputs a frame image, compares the frame of the input image with a background frame, binarizes the data generated based on the frame difference, and generates shape data for arbitrary shape encoding. . The background data correction unit 104 performs a correction process on the frame selected as the background based on the generation result of the shape data. The arbitrary shape image encoding unit 2105 inputs the previously generated shape data and image data at the same frame timing and performs encoding. At this time, switching is performed between the intra mode (intraframe coding) and the inter mode (interframe coding). Further, the data temporary storage (memory) function for matching the timing is not explicitly shown here, but it is assumed that each function block has. Details of the encoding of the image data and the encoding of the shape data are as described in the background art. The multiplexing unit 2106 multiplexes a plurality of encoded bit streams. In addition to video, an audio bit stream and the like are multiplexed. However, since there is no processing specific to the present invention, description thereof is omitted here.

  The background image selection method will be described in detail later. First, regarding the subsequent generation of shape data generation and background data correction processing associated therewith, the screen image of FIG. 3 and FIGS. This will be described with reference to the flowchart of FIG.

  First, in step S701 in FIG. 5, the first image is input. Here, the image 501 in FIG. 3 is the first image. In the background frame determination step S702, since this frame is determined to be the background, it is temporarily stored in step S703. Temporary storage is for performing difference processing. In step S704, shape data of the full screen (all pixels) object is generated as shape data of the frame. The shape data 511 indicates this shape data. In end determination S709, it is determined whether or not it is the last frame. Since NO is determined at this time, the process returns to step S701 to shift to image input for the next frame. Since the second image 502 is determined as NO in the background frame determination S702, the process advances to step S705 to detect a difference from the background frame. At this time, the background image is the frame 501 temporarily stored. In step S706, the difference value is binarized to generate shape data. Shape data generated from the difference value between the frames 501 and 502 is shape data 512. In the correction determination step S707, it is determined whether or not the background frame is to be corrected. Details of the determination method will be described later. Here, it is determined as NO, and the process returns to step S701. Since the third image 503 is determined as NO in the background frame determination S <b> 702, the processes of steps S <b> 705 and S <b> 706 are performed in the same manner as the second image 502. Shape data generated from the difference value between the frames 501 and 503 is shape data 513. If YES is determined in the correction determination step S707, the process proceeds to step S708 to correct the background data. At the time of background data correction, all input images including the background frame and the shape data generation result in each frame are used. Details of the correction method will be described later. In the end determination in step S709, it is determined whether or not all the frames have been processed. If the end (YES), a series of processing is ended.

  Next, details of a method for processing frames other than the background frame will be described with reference to FIG. In FIG. 6A, in step S801, an initial setting process necessary for background data correction is performed. Details of the initial setting process are shown in FIG. The setting of the count value in step S808 is to set a counter value indicating how many data are determined to be background at each position. Assuming that the counter value at a certain position (x, y) is c (x, y), the following equation is obtained for the entire screen when only the background frame is input.

    c (x, y) = 1 (2)

  The set of pixel values in step S809 is for initial setting of the cumulative pixel value of the background at each position. Assuming that the pixel value at a certain position (x, y) is p (x, y), p (x, y) is the pixel value of the background frame when only the background frame is input. This initial setting ends when all the pixels in the screen are processed in the end determination in step S810. If the number of pixels in the horizontal direction is W and the number of pixels in the vertical direction is H, the above processing is performed the number of times W × H. In the case of an RGB color image, W × H × 3, and in the case of a format called 4: 2: 2, the format is W × H × 2. Assuming that the frame 501 in FIG. 3 is a background frame, the accumulated pixel value becomes the pixel value of the background frame at the region 551, that is, at all positions on the screen.

  In step S802 of FIG. 6A, a frame other than the background is input, and in step S803, preprocessing for background data correction is performed. Details will be described with reference to FIG. First, a difference value between the background frame stored in advance in step S811 and the current input is detected. In step S812, the difference value and the threshold value are compared and binarized. If the binarized shape data is background in step S813, the number of pixels is counted in step S814. If the binarized result at a certain position (x, y) is the background, c (x, y) is changed from 0 to 1. In step S815, pixel values are added. If the pixel value of the first background frame at (x, y) is 52 and the pixel value of the current frame is 50, the pixel value after the addition is as follows.

    p (x, y) = 52 + 50 = 102 (3)

  If the binarized result is not the background, no processing is performed. These processes are repeated for the total number of pixels in the screen. If the frame 502 in FIG. 3 is a frame other than the first background, the black area in the shape data 512 is the background area in this frame. Therefore, the region 553 is a pixel value of the background frame, and the region 552 is a value obtained by adding the pixel value of the frame 501 and the pixel value of the frame 502.

  A series of pre-processing is finished at the stage where all the pixels have been processed (YES in step S816).

  Step S804 is a process for determining whether or not to start the post-correction process. Since the period from one background frame to the next background frame is one processing unit, the loop processing is repeated until the frame immediately before the next background frame. If the background frame insertion interval is constant, the determination can be made before the next background frame arrives. If the interval between the background frames is not constant, it is necessary to retain the processing contents until then until the next background frame is detected. Also, when the last frame is detected, post-processing is performed from the previous result. Assuming that the frame 503 in FIG. 3 is a frame immediately before the background, several regions are generated from the shape data generation result of the shape data 513 depending on the overlapping state of the background data. An area 553 is an area determined to be the background only in the frame 501, the counter value c (x, y) in this area is 1, and the accumulated pixel value is the pixel value itself of the background frame 501. An area 555 is an area determined to be the background in the frames 501 and 502, the counter value c (x, y) in this area is 2, and the accumulated pixel value is the sum of the pixel values of 501 and 502. An area 554 is an area determined to be the background in the frames 501, 502, and 503. In this area, the counter value c (x, y) is 3, and the accumulated pixel value is the sum of the pixel values of the frames 501, 502, and 503. It becomes.

  Step S805 is post-processing of background data correction. This will be described with reference to FIG. In step S817, an average pixel value is calculated. From the pixel values and counter values accumulated so far, the average pixel value a (x, y) at a certain position (x, y) can be obtained by the following equation.

    a (x, y) = p (x, y) / c (x, y) (4)

  This process is repeated for the total number of pixels in the screen, and the series of post-processing is finished at the stage where the processing of all the pixels is completed (the determination in step S818 is YES). In FIG. 3, since the counter value is 1 in the area 553, the average pixel value remains the accumulated pixel value, and the counter value is 2 in the area 555, so the average pixel value is the accumulated pixel value / 2. Since the value is 3, the average pixel value is cumulative pixel value / 3. That is, a common area of the background image and shape data is obtained, an average image for each common area is generated, and one piece of composite background image data is generated.

  Step S806 is a final end determination. If the processing of the last frame of the sequence is completed (YES), the process ends.

  In this way, by averaging images using a plurality of frames, the influence of noise and minute fluctuations can be suppressed, and a more natural image can be obtained even in a composite image after decoding. The corrected background data and its shape data 511 are input to the arbitrary shape encoding unit 2105 in FIG.

  As described above, in the MPEG-4 arbitrary shape image coding method, there are an intra mode (intra-frame coding) for performing processing within a frame and an inter mode (inter-frame coding) for performing processing between frames. . The problem of how to select the encoding processing mode is not directly related to the problem of how to determine the frame to be set as the background, and can be freely set. However, if the frame set as the background is set to the intra mode, efficient encoding becomes possible. In the inter mode, the shape data is also matched between frames, so that if the shape data of each frame is similar, the amount of generated codes in the shape data can be reduced. When the frame 511 is set to the intra mode and the frames 512 and 513 are set to the inter mode, the shape data 513 is similar to the shape data 512, so that the amount of generated codes can be reduced.

  Here, a background frame setting method will be described with reference to FIGS. FIG. 10 shows a method of selecting a background periodically regardless of the contents of the scene. By periodically updating the background from the first frame, the generated code amount can be suppressed even if the content of the image changes. First, as an initial setting, a value i for counting the number of frames is set to 0 in step S1201. In step S1202, it is determined whether the current frame is the top frame. If the count value i = 0, since it is the first frame, it is set as a background frame in step S1204. If it is not the first frame, it is determined in step S1203 whether or not the current frame is a multiple of the period T. If the remainder obtained by dividing the count value i by the period T is 0, it is a multiple of the period T, so that a background frame is set in step S1204. If the current frame is not a multiple of the period T, the process proceeds to step S1205 for end determination without setting the background. If it is not the last frame, the end determination S1205 is NO, the count value i of the number of frames is incremented by 1 in step S1206, the process returns to step S1202, and the process for the next frame is started. Thereafter, the same processing is repeated, and when the final frame processing is completed, the end determination S1205 is YES, and the series of processing ends.

  FIG. 11 shows a method of analyzing the contents of a moving image scene and setting a background frame when a scene change occurs, and shows a method different from FIG. Since the image immediately after the scene change detection has a low correlation with the immediately preceding image, large difference data is generated. By using a frame in which a scene change has occurred as a background, difference processing across scenes can be prevented, and the amount of generated codes can be suppressed. First, in step S1301, it is determined whether or not the current frame is the first frame. If the current frame is the first frame, the background frame is set in step S1303. If the current frame is not the first frame, it is determined in step S1302 whether a scene change has occurred. The scene change detection method is not particularly limited to this embodiment, but one example is that the absolute value of the difference between frames is binarized by threshold processing and the determination is made based on the area of the region exceeding the threshold. In this way, when it is determined that the current frame is a frame in which a scene change has occurred, the background frame is set in step S1503. If there is no scene change, the process proceeds to step S1304 for end determination without setting the background. If it is not the last frame, the end determination S1304 is NO, the process returns to step S1301, and the process proceeds to the next frame. Thereafter, the same processing is repeated, and when the processing of the last frame is completed, the end determination S1304 becomes YES, and the series of processing is finished.

  FIG. 12 shows a background frame setting method combining FIG. 10 and FIG. A background is set every time period T, but other frames with scene changes are used as the background. First, in step S1401, a counter value i for counting the number of frames is initialized to 0. Next, in step S1402, it is determined whether i is 0 or not. This is a determination as to whether or not it is the first frame. If it is the first frame, the process advances to step S1405 to set the background frame. If it is not the first frame, it is determined whether or not the current frame is a multiple of the period T in step S1403. For the determination, if the remainder obtained by dividing the frame value i by the period T is 0, a simple method is to use a multiple. If the current frame is a multiple of the period T, it is set as a background frame in step S1405. Otherwise, the process proceeds to the scene change determination in step S1404. If there is a scene change, the background frame is set in step S1405; otherwise, the process proceeds to end determination step S1406. In the end determination, it is determined whether or not it is the final frame. If it is not the final frame, the counter value i is incremented by 1 in step S1407, and the processing from step S1402 is repeated. When the final frame is processed, the end determination loop is exited, and the series of processing ends.

  FIG. 13 is another background frame setting method combining FIG. 10 and FIG. In the case of FIG. 13, unlike the case of FIG. 12, after the scene change is detected, the background is periodically set when a section without a scene change continues. First, in step S1501, it is determined whether it is the first frame. The method for determining the first frame may be the method described above. If it is the first frame, the background frame is set in step S1503, and the frame counter value i is set to 1 in step S1504. The counter value i here is not the number of frames from the top described above, but always counts the number of frames from the background frame as 1. If it is not the first frame, the scene change is determined in step S1502. If it is a scene change frame, the background frame is set in step S1503, and the counter value i is set to 1 in step S1504. If it is not a scene change frame, the frame counter value i is incremented by one in step S1505. If it is immediately after the background frame, i = 2. In step S1506, it is determined whether the current frame is a multiple of the period T. As a determination method, if the count value i from the background frame is the same value as the period T, a multiple of T is convenient. If the current frame is a multiple of the period T, the background frame is set in step S1503, and the counter value i is set to 1 in step S1504. On the other hand, if the count value i from the background frame does not reach the period T, the process proceeds to step S1507 to determine the end. In the end determination, it is determined whether or not it is the final frame. If it is not the final frame, the processing from step S1501 is repeated. As a result, it is possible to set the background frame when detecting a scene change, and to set the background frame at the period T when a section without a scene change continues. When the final frame is processed, the end determination S1507 is YES, and the series of processing ends.

  Next, decoding-side processing corresponding to the encoding-side processing described in the first embodiment will be described with reference to FIGS. 2, 4, and 14.

  FIG. 2 is a block diagram showing the overall configuration on the decoding side. The separation unit 2201 separates a plurality of bit streams into bit streams for each decoder. The type of bit stream includes audio as well as video, but only the video bit stream, which is a process unique to the present embodiment, is illustrated here. The arbitrary shape image decoding unit 2203 receives the separated video bit stream and outputs the image data and shape data as decoded images. For this, it is preferable to use the MPEG-4 Core Profile decoder described in the background art. The image composition unit 201 generates a composite image from the input background image and the current frame image data and shape data, and outputs the composite image to the image output unit 2205. The image output unit 2205 is typically an image display device such as a display, and sequentially displays input frames at a desired timing.

  The processing procedure of the image composition unit 201 will be described with reference to the flowchart of FIG. First, in step S1801, it is determined whether the first input frame is the background. If the shape data is an all-pixel object, it can be determined that the image is the background. If the input frame is the background, in step S1802, the data is temporarily stored for synthesis. In step S1803, the background frame and the current frame are synthesized. Since the current frame is the background frame at this point, the current frame is output as it is in step S1804. In the end determination in step S1805, it is determined whether or not it is the last frame. If it is not the last frame, the process returns to step S1801 to determine the next frame. If it is not a background frame, in step S1803, the background image temporarily stored and the arbitrary shape image of the current frame are combined based on the shape data. The synthesized image is output in step S1804, and the process proceeds to end determination step S1805. The above processing is repeated, and when the final frame is processed, the end determination loop is exited, and the series of processing ends.

  In the first frame in FIG. 4, the image data 611 is image data having an arbitrary shape, the shape data 601 is the shape data, and the image data 641 is the composite image data. Since all the shape data 601 indicates an object, the image data 611 is the combined output image 641 as it is. In the next frame, image data 612 is image data of an arbitrary shape, shape data 602 is the shape data, and image data 641 is background image data. Since the black portion of the shape data 602 is not an object, the pixel of the background image data 641 is applied, and the white portion is an object, so the pixel of the image data 612 is applied. By applying the pixel value of the image data 641 or 612 for each pixel, a composite image 642 can be obtained. Similarly, when the background image data 641 and the arbitrary shape image data 613 are processed based on the shape data 603, a composite image 643 can be obtained.

  As described above, according to the image processing apparatus according to the first embodiment, a background image and an arbitrary shape image can be extracted from frames that are continuously input without adopting a configuration in which a background image is prepared in advance. By selecting, a general-purpose high-efficiency encoding system that is not limited to a scene can be realized. In particular, the background data correction process can suppress the influence of noise and minute fluctuations, and a more natural image can be obtained even in a composite image after decoding.

<Second Embodiment>
A second embodiment of the present invention will be described. The overall configuration is shown in FIG. The difference from the first embodiment is that the data obtained by the background data correction unit 104 is used for correction of shape data. The shape data generation unit 103 in FIG. 1A is the shape data generation / correction unit 105 in FIG. The processing procedure of image data and shape data in these two functional blocks will be described in detail with reference to the flowcharts of FIGS. 8 corresponds to FIG. 5 of the first embodiment, and FIG. 9 corresponds to FIG. 6 of the first embodiment. Parts that perform the same processing are denoted by the same reference numerals. is there.

  First, the overall processing flow will be described with reference to FIG. The processing from step S701 to step S707 is as described in FIG. In step S708, the background data is corrected. In the first embodiment, the correction is performed only once for each processing section. In step S1001, the shape data is corrected based on the background correction data in step S708. When the shape data is corrected, the area determined to be the background within the frame also changes, so that the background data is corrected again in step S708 based on the information. This process is repeated until the correction end determination in step S1002 is YES. When the correction end determination is YES, the process for the section corresponding to one background frame is completed, and the process for the section corresponding to the next background frame is started. When the process reaches the last frame, the end determination in step S709 is YES, and the series of processes ends.

  Next, details of a method of processing frames other than the background frame will be described with reference to FIG. First, in step S801, the initial setting described with reference to FIG. 6 is performed. In step S1101, initial setting for counting the number of frames is performed. Here, the count value k = 0. In step S802, an image is input, and in step S1102, the number of frames is counted. When a frame other than the first background is input, k = 1. In step S803, the preprocessing described with reference to FIG. 6 is performed. Step S804 determines the end of the frame to be processed for one background. If the next frame is the background, or if it is the last frame in the sequence, the determination is YES and the process ends. In the case of NO, the processing from step S802 to S803 is repeated, and the number of repetitions becomes the count value of k. Here, k = kmax. When this loop is exited, background correction post-processing is performed in step S805. Through the processing so far, corrected background data is obtained. Using this background data, the process proceeds to the shape data correction process. In step S1103, the number of frames to be processed is counted. Here, the countdown is performed from kmax. In step S1104, preprocessing is performed using new background data. This process is repeated until it is determined to end in step S1105. The determination of the number of end frames is YES when the count value k = 0. Here, when the post-processing of step S1106 is performed, corrected shape data generated using the first corrected background data and background data corrected again by the corrected shape data are obtained. Step S1107 is a determination process of how many times this re-correction process is repeated. If the end condition is not satisfied, the counter value that has become 0 is set to kmax again in step S1108, and the processing from step S1103 is repeated. The determination of the end of step S1107 may be a method of repeating a preset number of times, or a dynamic process such as calculating the amount of change in the background data or shape data for each repetition and aborting the repetition when it is less than or equal to the threshold value. It is good as a method. When this loop is exited, the processing for the section corresponding to one background frame is completed. When the process reaches the last frame, the end determination in step S806 is YES, and the series of processes ends.

  As described above, when the background data is corrected using a plurality of frames and the shape data is corrected using the corrected background data, more accurate background data and shape data can be obtained. It will be. The corrected background data and the corrected shape data are input to the arbitrary shape encoding unit 2105 in FIG.

  The background frame setting method is the same as that described in the first embodiment. Further, since the process on the decoding side is the same, the description is omitted here.

  As described above, according to the image processing apparatus according to the second embodiment, a background image and an arbitrary shape image can be selected from frames that are continuously input without adopting a configuration in which a background image is prepared in advance. By selecting, a general-purpose high-efficiency encoding system that is not limited to a scene can be realized. In particular, the background data and the shape data correction process provides a highly accurate extraction result, and enables highly efficient encoding.

<Third Embodiment>
FIG. 23 shows a hardware configuration example of a computer according to the third embodiment of the present invention. The present embodiment shows an example in which the devices of the first and second embodiments are realized by a computer.

  A central processing unit (CPU) 2902, a ROM 2903, a RAM 2904, a network interface 2905, an input device 2906, an output device 2907, and an external storage device 2908 are connected to the bus 2901.

  The CPU 2902 performs data processing or calculation and controls various components connected via the bus 2901. The ROM 2903 stores the control procedure (computer program) of the CPU 2902 in advance, and the CPU 2902 is activated when the computer program is executed. A computer program is stored in the external storage device 2908, and the computer program is copied to the RAM 2904 and executed. The RAM 2904 is used as a work memory for data input / output, transmission / reception, and temporary storage for control of each component. The external storage device 2908 is, for example, a hard disk storage device or a CD-ROM, and stores image data and the like, and the stored content does not disappear even when the power is turned off. The CPU 2902 performs the processes of the first and second embodiments by executing the computer program in the RAM 2904.

  The network interface 2905 is an interface for connecting to a network. The input device 2906 is, for example, a keyboard and a mouse, and can perform various designations or inputs. The output device 2907 is a display, a printer, or the like.

  In the present embodiment, a computer-readable recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus, and the computer (or CPU) of the system or apparatus is supplied. Or MPU) can read out and execute the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention. In addition, by executing the program code read by the computer, not only the functions of the first and second embodiments are realized, but also an operating system running on the computer based on an instruction of the program code ( It goes without saying that the case where the functions of the above-described embodiments are realized by performing part or all of the actual processing by the OS) or the like.

  Further, after the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the card or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  When the present embodiment is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

  Note that the present embodiment can be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), but a device (for example, a copier, a facsimile machine, etc.) composed of a single device. ).

  As described above, according to the first to third embodiments, it is possible to select a background image and an arbitrary shape image from continuously input frames without adopting a configuration in which a background image is prepared in advance. However, by separating and synthesizing, a general-purpose high-efficiency encoding system that is not limited to a scene can be realized.

  Since a frame as a background image can be selected regardless of the presence or absence of a subject in the image, pre-processing for setting the background image is not necessary. Further, the rectangular image encoding means is not required, and the encoding can be performed only by the arbitrary shape image encoding means. Further, by correcting the background data, it is possible to suppress the influence of moving image noise and minute fluctuations, and it is possible to obtain a more natural image even in a composite image after decoding.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. It is a block diagram which shows the structure of a decoding system. It is a figure for demonstrating the process of the shape data in the 1st Embodiment of this invention, and image data. It is a figure for demonstrating the process of the shape data and image data in a decoding system. It is a flowchart for demonstrating the flow of the whole process of the shape data in the 1st Embodiment of this invention, and image data. It is a flowchart for demonstrating the flow of the detailed process of the shape data and image data in the 1st Embodiment of this invention. It is a flowchart for demonstrating the flow of the detailed process of the shape data and image data in the 1st Embodiment of this invention. It is a flowchart for demonstrating the flow of the whole process of the shape data and image data in the 2nd Embodiment of this invention. It is a flowchart for demonstrating the flow of the detailed process of the shape data and image data in the 2nd Embodiment of this invention. It is a flowchart for demonstrating the setting procedure of the background frame which concerns on embodiment of this invention. It is a flowchart for demonstrating the setting procedure of the other background frame which concerns on embodiment of this invention. It is a flowchart for demonstrating the setting procedure of the other background frame which concerns on embodiment of this invention. It is a flowchart for demonstrating the setting procedure of the other background frame which concerns on embodiment of this invention. It is a flowchart for demonstrating the synthetic | combination procedure of the image data in a decoding system. It is a block diagram which shows the structure of the image processing apparatus by the side of the encoding which concerns on a prior art example. It is a block diagram which shows the structure of the image processing apparatus by the side of the decoding which concerns on a prior art example. It is a block diagram which shows the structure of the image processing apparatus of the encoding side which concerns on a prior art example, and is a figure for demonstrating especially the process of an image separation part. It is a block diagram which shows the structure of the image processing apparatus by the side of the decoding which concerns on a prior art example, and is a figure for demonstrating especially the process of an image display part. It is a figure for demonstrating the process of the shape data and image data by the side of an encoding concerning a prior art example. It is a figure for demonstrating the process of the shape data and image data by the side of decoding concerning a prior art example. It is a block diagram which shows the structure of an arbitrary shape encoding part. It is a block diagram which shows the structure of an arbitrary shape decoding part. It is a block diagram which shows the hardware structural example of a computer.

Explanation of symbols

101 Image selection unit 103 Shape data generation unit 104 Background data correction unit 105 Shape data generation / correction unit 2101 Image input unit 2105 Arbitrary shape image encoding unit 2106 Multiplexing unit 2901 Bus 2902 CPU
2903 ROM
2904 RAM
2905 Network interface 2906 Input device 2907 Output device 2908 External storage device

Claims (12)

A moving image input means for inputting a moving image composed of a plurality of frames;
When selecting a frame as a background image from the moving image, an image selection means for selecting a frame as a background image regardless of the presence or absence of a subject in the image;
A shape data generating means for comparing the frame as the background image with the frame of the input image and generating shape data based on the difference value;
Background data correction means for correcting the background image based on the shape data;
An image processing apparatus comprising: an arbitrary shape image encoding means for encoding the input image as an arbitrary shape image together with the shape data.   2. The background data correcting unit obtains a common area of the background image and the shape data, generates an average image for each common area, and generates one composite background image data. Image processing apparatus.   The image processing apparatus according to claim 1, wherein the background data correction unit corrects the shape data in accordance with correction of background image data.   The image processing apparatus according to claim 1, wherein the shape data generation unit outputs the shape data as an all-pixel object when the input image is a frame as a background image. .   5. The image processing apparatus according to claim 1, wherein the image selection unit selects a frame to be a background image at a constant cycle from a top frame.   The image processing apparatus according to claim 1, wherein the image selection unit selects the frame as a background image when a scene change of the moving image is detected.   The image processing apparatus according to claim 6, wherein the image selection unit selects a frame as a background image at a constant cycle after the scene change of the moving image is detected.   The image processing apparatus according to claim 1, wherein the arbitrary-shaped image encoding unit performs switching between intra-frame encoding and inter-frame encoding.   9. The image processing apparatus according to claim 8, wherein the image selection unit selects a frame as a background image corresponding to the intra-frame encoding process in the arbitrary shape image encoding unit. A moving image input step for inputting a moving image composed of a plurality of frames;
An image selection step of selecting a frame as a background image regardless of the presence or absence of a subject in the image when selecting a frame as a background image from the moving image;
A shape data generation step of comparing the frame as the background image with the frame of the input image and generating shape data based on the difference value;
A background data correction step for correcting the background image based on the shape data;
An image processing method comprising: an arbitrary shape image encoding step for encoding the input image as an arbitrary shape image together with the shape data.   A program for causing a computer to execute each step of the image processing method according to claim 10.   A computer-readable recording medium on which a program for causing a computer to execute each step of the image processing method according to claim 10 is recorded.

Images