JP2006203444A - Background image formation device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a background image formation device capable of forming a background image automatically without preparing an extra background image. <P>SOLUTION: The background image formation device for generating background image data by extracting the background part from input image data comprises a section for inputting image data, a section for dividing each frame constituting the input image data into blocks indicative of predetermined image regions, a section for calculating the luminance value for each block, a background image extracting section for extracting adjacent frame series having a small quantity of variation in luminance value and determining the background image data of a block among the adjacent frame series, and a background image synthesizing section for synthesizing determined background image data to generate background image data over the entire region of one frame. With such an arrangement, a background image can be formed automatically without preparing an extra background image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a background image generation apparatus and a background image generation program, and more specifically, a background image generation apparatus and a background image generation program for generating a background image by extracting a background portion from an input image taken by a camera or the like About.

  Representing the movements of athletes in sports broadcasts in an easy-to-understand manner for viewers is quite important, and many technologies have been studied for this purpose. For example, the slow motion playback slows down the movement of the player so that the movement is easy to see, and the multi-motion playback extracts a player image that performs a predetermined motion, and among the extracted player images, a plurality of still images The technique of sequentially superimposing and displaying them on the background image is also used in actual sports broadcast broadcasting.

  At this time, a problem is a method for identifying an image of a player who performs a predetermined action and a background image. The image data for screen display is a set of color information and luminance information, and it is not distinguished whether the information in the data is information about the player's image data or information about the background image data. Therefore, it is difficult to extract a background image.

  On the other hand, a method for extracting a background image from a moving image has been studied and proposed (for example, Patent Document 1). In the method of Patent Document 1, a motion vector of a moving image is calculated, a predicted image is generated from the calculated motion vector, and a background region is determined from the difference between the predicted image and the actual image.

However, such a method has a problem that although a partial background region can be determined, a background image cannot be generated as a whole.
JP 2003-32688 A

  As described above, the background image generation method according to the related art has a problem that although a partial background region can be determined, the background image cannot be generated as a whole.

  The present invention has been made to solve such problems, and an object of the present invention is to provide a background image generation apparatus capable of automatically generating a background image without preparing a separate background image. And

  A background image generation apparatus according to the present invention is a background image generation apparatus that includes a subject image and a background image, extracts a background portion from input image data including a plurality of frames, and generates background image data for one frame, An image input unit that inputs input image data, an image dividing unit that divides the input image data input by the image input unit into blocks each representing a predetermined image area in one frame, and the image division A luminance calculation unit that calculates a luminance value for each block divided by the unit, and an adjacent frame sequence with a small amount of change in the luminance value calculated by the luminance calculation unit is extracted, and the block is extracted from the extracted adjacent frame sequence A background image extraction unit for determining the background image data of the image and a background image data determined for each block by the background image extraction unit A background image generator with the background image synthesizing section that generates a background image data over the entire area of one frame. With such a configuration, it is possible to automatically generate a background image without preparing a separate background image.

  The background image extraction unit includes a difference calculation unit that calculates a difference amount between adjacent frames of the luminance value calculated by the luminance calculation unit, and the adjacent frame based on the difference amount between adjacent frames calculated by the difference calculation unit. A background image determination unit that determines background image data of the block from among the above may be provided.

  The luminance calculation unit may determine a luminance value by calculating a frequency component based on color information included in the image data.

  The background image determination unit determines the background image data of the block from adjacent frames taking the smallest value among the adjacent frame difference amounts of the luminance values of the frames calculated by the difference calculation unit. It is good.

  A background image generation program according to the present invention is a background image generation program that includes a subject image and a background image, extracts a background portion from input image data including a plurality of frames, and generates background image data for one frame, A step of inputting input image data; a step of dividing each frame constituting the input image data inputted by the step of inputting image data into blocks each representing a predetermined image area in one frame; and for each block A step of calculating a luminance value for each block divided by the dividing step, and extracting an adjacent frame sequence having a small change amount of the luminance value calculated by the step of calculating the luminance value, Determining the background image data of the block from the background image data Background image data determined by the determining step to synthesize a background image generating program causes the step of generating a background image data over the entire region of one frame to the computer. This makes it possible to automatically generate a background image without preparing a separate background image.

  The step of determining the background image data includes calculating a difference amount between adjacent frames of the luminance value calculated by the step of calculating the luminance value, and calculating an adjacent amount calculated by the step of calculating the difference amount between the adjacent frames. A step of determining background image data of the block from the adjacent frames based on an inter-frame difference amount may be provided.

  The step of calculating the luminance value may determine the luminance value by calculating a frequency component based on color information included in the image data.

  In the step of determining the block background image data, the background image data of the block is selected from the image data having the smallest value among the difference amounts of the luminance values of the respective image data calculated in the step of calculating the difference amount. It may be characterized by determining.

  According to the present invention, it is possible to provide a background image generation apparatus that can automatically generate a background image without preparing a separate background image.

Embodiment 1 of the Invention
FIG. 1 is a block diagram showing a hardware configuration of a background image generation apparatus according to Embodiment 1 of the present invention. The background image generation apparatus 100 includes a CPU (Central Processing Unit) 101 and a RAM (Random Access Memory) 104. The CPU 101 and the RAM 104 are connected to an HDD (Hard Disk Drive) 113 as an auxiliary storage device via a bus 130. Storage medium driving devices such as the HDD 113 and the CD drive 126 are connected to the bus 130 via various controllers such as the IDE controller 125.

  The CPU 101 is a central processing unit that executes various controls in the background image generation apparatus 100, and various controls are performed by executing programs read into the RAM 104. The RAM 104 is a storage unit that stores programs and data necessary for program execution.

  The keyboard / mouse controller 105 detects the input of the keyboard 106 and the mouse 107 by the user, and outputs the detection result to the CPU 101. The keyboard 106 has a plurality of buttons, and is an input means for a user to input by pressing the buttons. The keyboard 106 is mainly used for inputting character strings such as words and sentences. The mouse 107 is an input unit for designating a position on the screen by a user operation and performing input based on the designated position by pressing a button. The mouse 107 is mainly used as a pointing device for selecting and determining information displayed on a screen such as a GUI (Graphical User Interface).

  The video controller 108 performs control for displaying visual information on a connected display device. The VRAM 109 is storage means for storing data necessary for visual information to be displayed. The VRAM 109 may realize its function by the RAM 104 or may be a dedicated RAM.

  The ROM 114 is a storage unit for storing a program for executing various controls in the background image generation apparatus 100. The ROM 114 is a read-only memory and cannot be rewritten. Therefore, the ROM 114 is mainly used for storing important hardware control programs such as immediately after the background image generating apparatus 100 is turned on.

  The serial port 115 and the parallel port 116 are ports for connecting peripheral devices such as a printer, a scanner, and a digital camera, and inputting / outputting data with the connected peripheral devices. Peripheral devices may be connected to a USB (Universal Serial Bus) or the like instead of the serial port 115 and the parallel port 116.

  The communication adapter 118 is an adapter for connecting to a network such as the Internet and transmitting / receiving data to / from the connected network. The connection to the network may be a wired connection using a cable or a wireless connection.

  The IDE controller 125 is a controller that controls data input / output with the HDD 113 and the CD drive 126. The HDD 113 gives instructions to the CPU 101 and the like in cooperation with the operating system, and can store a program for implementing the present embodiment and data necessary for executing the program. The CD drive 126 reads a program stored in a CD-ROM and data necessary for executing the program. These programs can be compressed and divided and stored in a plurality.

  The hardware configuration shown in FIG. 1 is equivalent to a general computer system. That is, the background image generating apparatus 100 can use a general computer system. The above configuration can be omitted as necessary.

  FIG. 2 is a block diagram illustrating a functional configuration of the background image generation apparatus 100 according to the present invention. The background image generation apparatus 100 includes a data storage unit 1, an image input unit 2, an image division unit 3, a luminance calculation unit 4, a difference calculation unit 5, and an image composition unit 6.

  The data storage unit 1 is a storage unit for storing image data input from the image input unit 2 and various data generated by the image dividing unit 3, the luminance calculation unit 4, the difference calculation unit 5, and the image composition unit 6. Yes, it is realized by the RAM 104 and the HDD 113.

  The image input unit 2 inputs image data. The input image data to be input here may be input by connecting a photographing device such as a camera to the serial port 115 or the parallel port 116, or moving image data such as an already photographed avi file may be input to the HDD 113 or the CD drive. 126 may be read and input. The image input unit 2 stores the inputted input image data in the data storage unit 1.

  The image dividing unit 3 divides each image data constituting the input image data input by the image input unit 2 into blocks. Although how many blocks are divided is not particularly limited, in this embodiment, division is performed into 4 × 4 16 blocks.

  The luminance calculation unit 4 calculates a luminance value for each block data divided by the image dividing unit 3. As a method for calculating the luminance value, there is a method in which the two-dimensional discrete cosine transform is applied to the color information data of each block, the frequency data is obtained by obtaining a coefficient, and a specific part of the obtained frequency data is used as the luminance value. It is done.

  The difference calculation unit 5 performs a difference calculation of the luminance values calculated by the luminance calculation unit 4. In this calculation, the luminance value is simply subtracted. The difference value obtained here is used as image change amount information.

  The image composition unit 6 determines the block image data with the smallest amount of change from the difference value calculated by the difference calculation unit 5, and synthesizes the block image data with the least amount of change determined for each block. Background image data is generated.

  Various controls of the image input unit 2, the image division unit 3, the luminance calculation unit 4, the difference calculation unit 5, and the image composition unit 6 are performed by reading a program stored in the HDD 113 or the CD drive 126 into the RAM 104 and executing it by the CPU 101. Is called.

  Next, a background image generation process flow will be described. FIG. 3 is a flowchart showing a process flow of the background image generation apparatus 100 according to Embodiment 1 of the present invention.

  First, the image input unit 2 inputs image data and stores it in the data storage unit 1 (S1). FIG. 4 is an example of input image data. In this example, the input image data is composed of three frames 2a to 2c, but in reality, the input image data is often composed of more images. Thereafter, block division is performed for each frame constituting the input image data input by the image dividing unit 3 (S2). FIG. 5 is an image diagram of an input image subjected to block division. The background image is determined for each of the blocks divided by the image dividing unit 3, and is finally combined to become a background image. The background image for each block is determined sequentially from the first block (S3). At this time, among the blocks to be divided, the background image may be determined in any order starting with which block.

  Thereafter, the background image is determined for each block. The background image for each block is determined based on the difference value of the luminance value of the block in each frame. First, the luminance calculation unit 4 arranges the image data of each frame of the block along the time axis, and extracts two adjacent frames from the top (S4). Here, “adjacent” does not mean that the positions are lined up, but is adjacent between the same blocks of the respective image data arranged with respect to the time axis. At this time, the calculation amount may be reduced by extracting at intervals of several frames. When two frames are extracted, the luminance calculation unit 4 calculates and acquires frequency data for each input block image data (S5). The frequency data is acquired, for example, by performing two-dimensional discrete cosine transform on the color information data of each block. The frequency data obtained here becomes luminance data. FIG. 6 is a diagram showing a flow when conversion into luminance data is performed.

  When the luminance calculation unit 4 acquires the luminance data, the luminance calculation unit 4 outputs the acquired luminance data to the difference calculation unit 5. The difference calculation unit 5 compares two adjacent frames for a specific portion of the frequency data input from the luminance calculation unit 4 and calculates a difference value (S6). When the calculation of the difference value for the block image data of the first two frames is completed, the difference value is calculated between the second frame and the third frame. This operation is performed between all adjacent image data. FIG. 7 is a diagram showing the calculated difference value. The difference value between 3a and 3b shown in the figure is 4a, and the difference value between 3b and 3c is 4b.

  When the difference calculation unit 5 calculates a difference in frequency data between all adjacent image data (S7), the difference calculation unit 5 outputs the calculated difference value data to the image composition unit 6. When the difference value data input from the difference calculation unit 5 is input, the image composition unit 6 determines image data to be a background image from the input image data using the input difference value data (S8).

  The image data to be the background image is determined as follows, for example. FIG. 8 is a diagram illustrating a method for determining image data to be a background image. In the example of this figure, the input image data is composed of 10 frames. F1 to F9 shown in FIG. 8 are difference values between the adjacent frames. For example, F1 is a difference value between the luminance values of the corresponding blocks of 1 frame and 2 frames, and F2 is a difference value of the luminance values of the corresponding blocks of 2 frames and 3 frames. First, frequency data differences are summed for the same block of four adjacent image data. These total values are S1 to S6, respectively. That is, S1 = F1 + F2 + F3 and S2 = F2 + F3 + F4. The total calculation of the frequency difference of the same block of the four image data is performed for all four adjacent frame sequences. Then, the adjacent frame sequence having the smallest difference value of the frequency data is regarded as a stable adjacent frame sequence, one of the stable adjacent frame sequences, for example, the second frame is selected, and the background image data of the block To do.

  When the background image data of the block is determined, the same operation is performed in the next block (S10). Repeat this process for all blocks.

  When the background image data for all the blocks is determined (S9), the image composition unit 6 synthesizes the background image data of each determined block (S11). The background image data generated as a result of the composition by the image composition unit 6 is the final background image data. FIG. 9 shows final background image data. 5a is the background image data composition of each block, and 5b is the final background image data. The image composition unit 6 stores the generated final background image data in the data storage unit 1.

  By doing so, it becomes possible to automatically generate a background image without preparing it.

Other Embodiments of the Invention
In the above example, the image composition apparatus is realized using a general computer system. However, this may be performed using a dedicated computer. Alternatively, this may be performed using a plurality of computers.

It is a block diagram which shows the hardware constitutions of the image composition apparatus in this invention. It is a block diagram which shows the function structure of the image synthesizing | combining apparatus in this invention. It is a flowchart which shows the flow of a process of the image composition apparatus in this invention. It is a figure which shows the example of the input image data in this invention. It is a figure which shows the example of the block division of the image data in this invention. It is a figure which shows the example of the luminance value calculation in this invention. It is a figure which shows the example of the difference value calculation in this invention. It is a figure which shows the determination method of the image data used as the background image in this invention. It is a figure which shows the example of the last image data in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Data storage part 2 Image input part 3 Image division part 4 Luminance calculation part 5 Difference calculation part 6 Image composition part 100 Background image generation apparatus 101 CPU
104 RAM
105 Keyboard Mouse Controller 106 Keyboard 107 Mouse 108 Video Controller 109 VRAM
113 HDD
114 ROM
115 Serial port 116 Parallel port 118 Communication adapter 125 IDE controller 126 CD drive 130 Bus

Claims (8)

A background image generation apparatus that includes a subject image and a background image, extracts a background portion from input image data including a plurality of frames, and generates background image data for one frame,
An image input unit for inputting the input image data;
An image dividing unit that divides the input image data input by the image input unit into blocks each representing a predetermined image area in one frame;
A luminance calculation unit for calculating a luminance value for each block divided by the image dividing unit;
Extracting a neighboring frame sequence with a small amount of change in luminance value calculated by the luminance calculating unit, and determining a background image data of the block from the extracted adjacent frame sequence;
A background image generation apparatus comprising a background image synthesis unit that synthesizes background image data determined for each block by the background image extraction unit and generates background image data over the entire region for one frame. The background image extraction unit
A difference calculation unit for calculating a difference amount between adjacent frames of the luminance value calculated by the luminance calculation unit;
The background image generation unit according to claim 1, further comprising a background image determination unit that determines background image data of the block from the adjacent frames based on the difference amount between adjacent frames calculated by the difference calculation unit. apparatus.   The background image generation apparatus according to claim 1, wherein the luminance calculation unit determines a luminance value by calculating a frequency component based on color information included in the image data.   The background image determination unit determines the background image data of the block from the adjacent frame sequence that takes the smallest value among the adjacent frame difference amounts of the luminance values of the frames calculated by the difference calculation unit. The background image generation apparatus according to claim 2 or 3, A background image generation program for extracting a background portion from input image data including a plurality of frames including a subject image and a background image and generating background image data for one frame,
Inputting the input image data;
Dividing each block constituting the input image data inputted by the step of inputting the image data into blocks each indicating a predetermined image area in one frame;
Calculating a luminance value for each block divided by the step of dividing for each block;
Extracting an adjacent frame sequence having a small amount of change in the brightness value calculated by the step of calculating the brightness value, and determining background image data of the block from the extracted adjacent frame sequence;
A background image generation program for causing a computer to perform a step of generating background image data over the entire area of one frame by synthesizing the background image data determined in the step of determining the background image data. Determining the background image data comprises:
Calculating a difference amount between adjacent frames of the luminance value calculated by the step of calculating the luminance value;
6. The method according to claim 5, further comprising: determining background image data of the block from the adjacent frames based on the difference amount between adjacent frames calculated by the step of calculating the difference amount between adjacent frames. Background image generation program.   The background image generation program according to claim 5 or 6, wherein the step of calculating the luminance value determines the luminance value by calculating a frequency component based on color information included in the image data.   The step of determining the block background image data includes the step of determining the block background image data of the block from the image data having the smallest value among the difference amounts of the luminance values of the image data calculated by the step of calculating the difference amount. The background image generation program according to claim 6 or 7, wherein:

Images