JP4136403B2 - Image processing apparatus, image processing method, program, and storage medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus, an image processing method, a program, and a memory for encoding and transmitting an object image including only an object and a background image including the object in continuous still images constituting a moving image. It relates to the medium.
[0002]
[Prior art]
Conventionally, as an image encoding method, an intra-frame encoding method such as Motion JPEG or Digital Video, or H.264 using inter-frame predictive encoding is used. 261, H.M. Coding schemes such as H.263, MPEG-1, and MPEG-2 are known. These encoding methods are internationally standardized by ISO (International Organization for Standardization) and ITU (International Telecommunication Union). The intra-frame coding method performs coding independently for each frame, and is easy to manage the frames, so that it is most suitable for an apparatus that requires editing of moving images and special reproduction. In addition, the inter-frame predictive coding method has a feature that coding efficiency is high because inter-frame prediction based on a difference in image data between frames is used.
[0003]
Furthermore, the international standardization of MPEG-4 is in progress as a general-purpose next-generation multimedia coding standard that can be used in many fields such as computer, broadcasting, and communication. A major feature of MPEG-4 is the ability to perform object-based coding.
[0004]
Object-based encoding is not a method of encoding an entire rectangular image as employed in MPEG (Moving Picture Expert Group) -1 or MPEG-2, but shape information generated by some method in advance. In this method, encoding is performed for each person or other object cut out in the image, that is, for each object in the image. Hereinafter, in order to distinguish from this shape information image, an image to be processed is generally referred to as a natural image.
[0005]
The shape information image is an image having the same number of vertical and horizontal pixels as the natural image to be encoded and representing the shape of the object. The shape information image includes a binary alpha plane in which each pixel is represented by 1 bit and a grayscale alpha plane in which each pixel is represented by 2 bits or more. In the binary alpha plane, an area having a pixel value “1” generally represents an object area, and an area “0” represents an area outside the object.
[0006]
The other detailed contents of MPEG-4 will be entrusted to international standard documents by ISO / IEC.
[0007]
[Problems to be solved by the invention]
The object-based encoding method will be described with reference to FIG. FIG. 6A is a natural image, and FIG. 6B is a shape information image corresponding to FIG.
[0008]
The shape information image is an image composed of pixel groups indicating the inside and outside of the object, and the pixel value of the pixel group indicating the inside of the object (the white portion in FIG. 5B) is 1 or more and the pixel indicating the outside of the object The pixel value of the group (shaded portion in FIG. 5B) is 0. Therefore, by referring to this shape information image, it is possible to specify a pixel group constituting an object in the natural image.
[0009]
As a result, the natural image can be encoded separately inside the object (object image) and outside (background image) and transmitted to the outside, but the background image has a larger area than the object image. This indicates that the background image has more pixels than the object image. As a result, depending on how the background image is handled, when the natural image is transmitted to the outside, the efficiency is greatly affected.
[0010]
The present invention has been made in view of the above problems, and an object thereof is to improve the coding efficiency of the entire image including the background image and the transmission efficiency of the background image.
[0011]
[Means for Solving the Problems]
In order to solve the problems of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.
[0012]
That is, an image processing apparatus that inputs continuous still images constituting a moving image, encodes each of an object image and a background image in each still image, and transmits the encoded image to the outside.
Storage means for storing the background image;
The block x is extracted from the still image for one frame, the corresponding block x corresponding to the block x is extracted from the background image stored in the storage means, and the difference between the block x and the corresponding block x is calculated. Differential image generation means for performing the process of generating the difference block x shown for x = 1, 2,..., N (N>1);
An averaging means for performing the process of obtaining an average block x of the difference blocks x generated by the difference image generating means for each frame over a preset number of frames for x = 0, 1,.
Frequency conversion means for performing a frequency conversion on the average block x to generate a coefficient block x for x = 0, 1,..., N ;
A determination means for determining whether or not the absolute difference sum of the coefficient block x and the comparison target block is larger than a preset threshold value for x = 0, 1,..., N ;
The determination unit is x = 0, 1, ..., a result of determining the N, large and when the number of times it is determined is greater than a preset threshold, as a new background image, the number of frames that is set in advance Output means for encoding any one of the still images and outputting the encoded image to the outside.
[0013]
In order to solve the problems of the present invention, for example, an image processing method of the present invention comprises the following arrangement.
[0014]
That is, an image processing method performed by an image processing apparatus that inputs continuous still images constituting a moving image, encodes each of an object image and a background image in each still image, and transmits the encoded image to the outside.
A block x is extracted from a still image for one frame, a corresponding block x corresponding to the block x is extracted from a background image stored in a memory, and a difference indicating a difference between the block x and the corresponding block x A difference image generation step of performing processing for generating a block x with respect to x = 1, 2,..., N (N>1);
A process of obtaining an average block x of the difference blocks x generated for each frame in the difference image generation process over a preset number of frames, with respect to x = 0, 1,.
A frequency conversion step of performing a frequency conversion on the average block x to generate a coefficient block x for x = 0, 1,..., N ;
A determination step for determining whether or not the absolute difference sum of the coefficient block x and the comparison target block is larger than a preset threshold value for x = 0, 1,..., N ;
The determination step at x = 0, 1, ..., a result of determining the N, large and when the number of times it is determined is greater than a preset threshold, as a new background image, the number of frames that is set in advance An output step of encoding any one of the still images and outputting the encoded image to the outside.
[0015]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.
[0016]
[First Embodiment]
The image processing apparatus according to the present embodiment inputs a moving image (consisting of continuous still images) captured by an external imaging device, and includes an image area and background of an object included in each still image constituting the captured image. Each of the image areas is encoded, multiplexed as necessary, and output as a stream.
[0017]
With reference to FIG. 1 which is a block diagram showing a functional configuration of the image processing apparatus in the present embodiment, each part of the image processing apparatus in the present embodiment will be described. In the following description, when an input image is encoded, it is divided into blocks of a predetermined size, and the size is 8 pixels × 8 lines. However, the present invention is not limited to this size. Further, hereinafter, both the photographed image and the background image are assumed to be one frame still images constituting the moving image.
[0018]
100 is a background image memory for storing a background image, and 101 is a transmission background image output control for performing output control of a new background image to be transmitted using the currently captured image and the background image in the background image memory 100. It is a vessel. A background image encoder 102 encodes a background image from the transmission background image output controller 101. Reference numeral 103 denotes an object image generator that extracts an object region from an image that is currently captured and generates an object image. An object image encoder 104 encodes the object image from the object image generator 103. Reference numeral 105 denotes a multiplexer that multiplexes encoded data from the background image encoder 102 and / or the object image encoder 104 and outputs a stream.
[0019]
Next, the operation of the image processing apparatus according to this embodiment having the above configuration will be described. First, a background image without an object is captured in advance, and this image is stored in the background image memory 100. At the same time, encoded data of the background image is generated via the transmission background image output controller 101 and the background image encoder 102. Specifically, the captured image is input to the transmission background image output controller 101 at a specified frame rate. The transmission background image output controller 101 controls the output of a new background image to be transmitted using the input image and the background image already stored in the background image memory 100. This control method will be described later. The background image is encoded by the background image encoder 102 and input to the multiplexer 105.
[0020]
On the other hand, the captured image is input to the multiplexer 105 via the object image generator 103 and the object image encoder 104. The multiplexed stream is output from the multiplexer 105.
[0021]
Next, processing for controlling the output of the background image to be transmitted performed by the transmission background image output controller 101 will be described with reference to FIG. FIG. 2 is a block diagram showing a functional configuration of the transmission background image output controller 101.
[0022]
A block extractor 200 extracts a block at a predetermined position in the image from the background image input from the background image memory 100. Reference numeral 201 denotes a block extractor that extracts a block located at a predetermined position in the image from the captured image. Reference numeral 202 denotes a difference image generator that generates a difference block (difference image) from each block input from the block extractors 200 and 201. A difference image integrator 203 accumulates and integrates the difference blocks input from the difference image generator 202.
[0023]
Reference numeral 204 denotes a DCT unit that inputs the integrated difference block from the difference image integrator 203 and performs DCT (discrete cosine transform). The DCT unit 204 obtains the frequency component of the difference block and looks at the value of each component, whereby the amount of change in the image can be seen. Reference numeral 205 denotes a DCT coefficient comparator that inputs a DCT coefficient block from the DCT unit 204, a coefficient block for comparison with the DCT coefficient block, and a threshold block, and outputs a control signal indicating whether to update the background image. . A new background image output unit 206 selects and outputs a captured image as a new background image in accordance with a control signal from the DCT coefficient comparator 205.
[0024]
FIG. 3A shows an example of the position of the block extracted by the block extractors 200 and 201. For example, when the captured image is the image shown in FIG. 3B and the image stored in the background image memory 100 is the image shown in FIG. 3C, 4 shown in FIG. One position block is extracted from each image. In the figure, the number of blocks to be extracted is four, but is not limited to this. Moreover, it is not limited to these positions.
[0025]
FIG. 4 shows a flowchart of processing performed by the transmission background image output controller 101. First, the background output flag outf in the difference image integrator 203 is initialized to 0 and the processing block counter bcnt = 0 (step S300), and further initialized to the accumulation count counter cnt = 0 and the accumulation block Int = 0 (step S301). . Next, a block in the background image is input by the block extractor 200 (step S302). The block extractor 201 inputs a block in the captured image from the captured image (first frame) (step S303).
[0026]
Then, the difference image generator 202 takes the difference between the respective blocks and generates a difference block Diff (step S304). Next, the difference image integrator 203 accumulates the difference block Diff in the accumulation block Int (accumulation for each element), and simultaneously increments cnt by 1 (step S305). The above steps S302 to S304 are repeated until cnt = 2. That is, a block at the same position as the captured image is extracted from the next captured image (second frame that is an image of the next frame of the captured image), and the difference between this block and the background image block is calculated. A block Diff is generated, and the difference block Diff is accumulated in the accumulation block Int (accumulated for each element).
[0027]
Next, the difference image integrator 203 divides Int by the number of repetitions, that is, 2 (step S307). Next, the DCT unit 204 performs DCT on Int (step S308). Next, a comparison coefficient block and a threshold block are input in the DCT coefficient comparator 205 (steps S309 and S310), a difference absolute value sum between the coefficient block (Int coefficient block) from the DCT unit 204 and the comparison coefficient block, and a threshold value The block value is compared (step S311). The comparison formula shall follow the formula shown in No.
[0028]
Σ | Ai−Bi |> threshold block where Ai indicates the i-th element of the Int coefficient block, Bi indicates the i-th element of the comparison coefficient block, and the left side is a position between the Int coefficient block and the comparison coefficient block. The difference between corresponding elements is added to all elements. The elements of the comparison coefficient block and the value of the threshold block are not particularly determined here, but are appropriately determined according to the properties of the image.
[0029]
If left side> right side, the process proceeds to step S312 and the DCT coefficient comparator 205 increments outf by 1 (step S312). Next, bcnt is incremented by 1 (step S313), and it is determined whether bcnt is 4 or more (step S314). That is, it is determined whether or not the processing has been performed on all four blocks (four white rectangles in FIG. 3A). Next, it is determined whether or not outf is greater than 2 (step S315). If it is 3 or more, the process proceeds to step S315, and the DCT coefficient comparator 205 outputs a control signal to the new background image output unit 206. Based on this control signal, the new background image output unit 206 outputs a captured image (which may be either the first frame or the second frame) as a new background image to the background image encoder 102 and a background image memory. 100 (step S316). That is, in three or more blocks, if it is determined that the background image in the background image memory 100 is significantly different from the captured image (in the above formula, the difference is such that the left side is equal to or greater than the threshold block), the image is captured as a new background image. Use images.
[0030]
On the other hand, if the DCT coefficient comparator 205 determines in step S315 that outf is 2 or less, the process proceeds to step S317, and the DCT coefficient comparator 205 outputs a control signal to the new background image output unit 206. The new background image output unit 206 outputs NULL indicating that the background image is not output to the background image encoder 102 based on the control signal (step S317). That is, the background image in the background image memory 100 is largely different from the captured image (in the above expression, the difference is such that the left side becomes greater than or equal to the threshold block) that the number of blocks is less than half. This means that it is not necessary to output a new background image (a device that receives and decodes a stream uses the previously decoded background image).
[0031]
Note that the background image encoder 206 does not perform encoding processing when NULL is input, and therefore, in this case, the stream output from the multiplexer 105 includes only encoded data of the object.
[0032]
In this way, when there is almost no change in the background image, it is not necessary to perform processing such as encoding and transmission of the background image, so that effects such as reduction in processing amount and improvement in transmission efficiency can be obtained.
[0033]
As described above, the image processing apparatus according to the present embodiment encodes and transmits a captured image only when the captured image is significantly different from the previously stored background image. By not transmitting the background image, it is possible to reduce the overall transmission processing and improve the transmission efficiency.
[0034]
[Second Embodiment]
In the first embodiment, the present invention is applied to MPEG-4, but may be applied to other object-based encoding schemes. In the first embodiment, the integration target is up to cnt = 2 in the difference image integrator 203 (that is, only the first frame and the second frame are used), but other values may be used. It doesn't matter. In the first embodiment, the DCT is used as the frequency conversion for the difference block. However, the present invention is not limited to this.
[0035]
[Third Embodiment]
Each process described in the first embodiment can also be executed by a general computer having the configuration shown in FIG. FIG. 5 is a block diagram illustrating a basic configuration of a computer that executes each process described in the first embodiment. Reference numeral 500 denotes a central processing unit (CPU) that controls the entire computer and performs various processes. Reference numeral 501 denotes an operating system (OS), software, data, and a memory that provides a storage area necessary for the operation. It is. It is also used as a work area when the CPU 500 performs various processes.
[0036]
502 is a bus for connecting various devices to exchange data and control signals, 503 is a storage device for storing various software, 504 is a storage device for storing moving image data, 505 is an image or a system message from a computer, etc. Is a monitor that displays.
[0037]
Reference numeral 507 denotes a communication interface that transmits encoded data to the communication circuit 508, and is connected to the outside of the apparatus by a LAN, a public line, a wireless line, a broadcast wave, or the like. Reference numeral 506 denotes an operation unit that inputs various instructions for starting up the computer and setting various conditions such as a bit rate.
[0038]
The memory 501 controls the entire computer as described above, stores an OS for operating various software and operating software, stores an area for reading image data for encoding, and temporarily stores code data. There is a working area for storing a code area, various calculation parameters, and the like.
[0039]
In such a configuration, prior to the encoding process, moving image data to be encoded is selected from the moving image data accumulated in the storage device 504 from the operation unit 506. Then, the software stored in the storage device 503 is expanded in the memory 501 via the bus 502, and the software is activated.
[0040]
Then, for the encoding operation of the moving image data stored in the storage device 504 by the CPU 500, the program code (the above-mentioned software) according to the flowchart shown in FIG. 3 is executed.
[0041]
As described above, the computer according to the present embodiment functions as a device that implements the object-based image coding according to the first embodiment.
[0042]
[Other Embodiments]
An object of the present invention is to supply a storage medium (or recording medium) in which a program code of software for realizing the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus. Needless to say, this can also be achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.
[0043]
Furthermore, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.
[0044]
【The invention's effect】
As described above, according to the present invention, the encoding efficiency of the entire image including the background image and the transmission efficiency of the background image can be improved.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a functional configuration of an image processing apparatus according to a first embodiment of the present invention.
2 is a block diagram showing a functional configuration of a transmission background image output controller 101. FIG.
3A is a diagram showing an example of the position of a block extracted by the block extractor 200, and FIG. 3B is a diagram showing an example of the position of a block extracted by the block extractor 201. FIG.
4 is a flowchart of processing performed by a transmission background image output controller 101. FIG.
FIG. 5 is a block diagram showing a basic configuration of a computer according to a third embodiment of the present invention.
FIG. 6 is a diagram illustrating an object-based encoding method.

Claims (7)

An image processing apparatus that inputs continuous still images constituting a moving image, encodes each of an object image and a background image in each still image, and transmits the encoded image to the outside.
Storage means for storing the background image;
The block x is extracted from the still image for one frame, the corresponding block x corresponding to the block x is extracted from the background image stored in the storage means, and the difference between the block x and the corresponding block x is calculated. Differential image generation means for performing the process of generating the difference block x shown for x = 1, 2,..., N (N>1);
An averaging means for performing the process of obtaining an average block x of the difference blocks x generated by the difference image generating means for each frame over a preset number of frames for x = 0, 1,.
Frequency conversion means for performing a frequency conversion on the average block x to generate a coefficient block x for x = 0, 1,..., N ;
A determination means for determining whether or not the absolute difference sum of the coefficient block x and the comparison target block is larger than a preset threshold value for x = 0, 1,..., N ;
The determination unit is x = 0, 1, ..., a result of determining the N, large and when the number of times it is determined is greater than a preset threshold, as a new background image, the number of frames that is set in advance An image processing apparatus comprising: output means for encoding any one of the still images and outputting the encoded image to the outside.   The image processing apparatus according to claim 1, wherein the output unit stores the output still image as a new background image in the storage unit. If the number of times that the determination unit determines that x = 0, 1,..., N is less than or equal to a preset threshold value , the output unit outputs a signal indicating that there is no background image to be output. The image processing apparatus according to claim 1, wherein the image processing apparatus outputs the image. An image processing method performed by an image processing apparatus that inputs continuous still images constituting a moving image, encodes each of an object image and a background image in each still image, and transmits the encoded image to the outside.
A block x is extracted from a still image for one frame, a corresponding block x corresponding to the block x is extracted from a background image stored in a memory, and a difference indicating a difference between the block x and the corresponding block x A difference image generation step of performing processing for generating a block x with respect to x = 1, 2,..., N (N>1);
A process of obtaining an average block x of the difference blocks x generated for each frame in the difference image generation process over a preset number of frames, with respect to x = 0, 1,.
A frequency conversion step of performing a frequency conversion on the average block x to generate a coefficient block x for x = 0, 1,..., N ;
A determination step for determining whether or not the absolute difference sum of the coefficient block x and the comparison target block is larger than a preset threshold value for x = 0, 1,..., N ;
The determination step at x = 0, 1, ..., a result of determining the N, large and when the number of times it is determined is greater than a preset threshold, as a new background image, the number of frames that is set in advance An image processing method comprising: an output step of encoding any one of the still images and outputting the encoded image to the outside. By be loaded into a computer, a computer program for causing to function as an image processing apparatus according to the computer in any one of claims 1 to 3. A computer program for causing a computer to execute the image processing method according to claim 4 . Storing a program according to claim 5 or 6, computer-readable storage medium.

Images