KR20040049423A - Image composition method in mobile station - Google Patents

Abstract

PURPOSE: An image synthesizing method in a mobile communication terminal is provided to synthesize a region of interest and a background region rapidly using a limited number of CPU and DSP chips. CONSTITUTION: An object region is extracted from an input image. Macro-blocks that will use only a background image are designated to four square regions using the extracted object region information. Parts corresponding to the four square regions are fetched from the background image. Macro-blocks that will use only an object image of the original image are designated to a single square region using the object region information. A part corresponding to the square region is fetched from the original image. For each of macro-blocks corresponding to the other regions, a distance between the first pixel of one line and the center of the object is obtained and a distance increase/decrease to be applied to 8 pixels included in the line is calculated using the distance. Y, Cb and Cr values of the original image and background image are synthesized using the distance.

Description

Image composition method in mobile communication terminal

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a mobile communication terminal, and more particularly, to a mobile communication terminal that can separate an input image into a region of interest and a background region, and can synthesize a region of interest and another background image at high speed using a limited CPU and a DSP chip. Relates to an image synthesis method.

In addition, the present invention relates to an application field for synthesizing a specific object, such as a face, with a background different from the original background, or transmitting a video mail, and in particular, by using a CPU and a DSP chip limited in a field such as a mobile communication. The present invention relates to a method for synthesizing face regions at high speed.

Recently, as high-speed wireless communication services such as IMT2000 and CDMA2000 1x have been implemented, the mobile communication field has been expanded to various fields that provide multimedia services such as still images and videos other than voice communication. One of such various fields replaces a background area except for a specific object, such as a face, with another background, so that a communication or mail switching service can be considered. This example is illustrated in FIG. 1, which shows that only the face region is extracted from the original image and transmitted to the image synthesized with a still background desired by another user.

Such a function should include a function of extracting an object such as a face from the mobile communication terminal and a function of synthesizing the extracted object with another background. These features typically consist of algorithms that use a lot of CPU and memory. In the case of face extraction, many algorithms have been introduced and mainly use skin color information.

Such an extraction algorithm may reduce the image and process a smaller image, regardless of the original input image size, to produce a desired result. In this case, the amount of CPU usage and memory usage can be reduced by the reduction. In some cases, the user may simply describe the face region information directly and semi-automatically or manually extract it. However, in the case of face synthesis, it is not easy to reduce the size of the image to be transmitted and process it as it is, so it is not so easy to reduce the amount of CPU and memory required for compositing.

In general, since a mobile communication terminal has a limited size memory and a limited performance CPU, it is difficult to properly perform a function that requires many CPUs and memories as described above. Recently, in order to overcome this limitation, mobile communication terminals that perform video processing are often equipped with a DSP chip and a direct memory access (DMA) function.

The main processing functions mainly used in mobile communication terminals are to encode data to be transmitted or to decode received data. Therefore, for faster real-time processing, a dedicated hardware is required, which is used as a DSP chip, which can be used to quickly process a signal processing that is simple but has many repetitive operations such as DCT conversion used in encoding / decoding. It can be called a chip. The biggest feature of such a DSP chip for a mobile terminal is that it performs a lot of processing using a small power supply. DMA also allows you to import data from memory at one location into memory at another location without using the CPU.

The present invention proposes a method for synthesizing an object of a video and another background video at a high speed in a mobile communication terminal having such a feature.

The present invention provides an image synthesis method in a mobile communication terminal that can separate an input image into a region of interest and a background region, and synthesize a region of interest and another background image at high speed using a limited CPU and a DSP chip. The purpose is.

1 is a conceptual diagram illustrating a composition of a region of interest and a background region for a general original image, and synthesizing a region of interest from another background image.

2 is a diagram illustrating a threshold for classifying regions in order to perform image synthesis by the method of image synthesis in a mobile communication terminal according to the present invention.

3 is a flowchart illustrating a process of determining image synthesis conditions of a corresponding region by an image synthesis method in a mobile communication terminal according to the present invention.

FIG. 4 is a diagram illustrating a region in which an original image is displayed, an area in which another background image is displayed, and a region in which image synthesis is performed, in the image synthesis being performed by the image synthesis method in the mobile communication terminal according to the present invention. .

5 and 6 are views for explaining a method of calculating a reference distance in order to determine the image synthesis conditions of the area in the image synthesis is performed by the image synthesis method in the mobile communication terminal according to the present invention.

7 is a flowchart illustrating a process of determining image synthesis conditions of each region and performing image synthesis for each region by an image synthesis method in a mobile communication terminal according to the present invention.

8 is a diagram schematically illustrating a configuration of a mobile communication terminal employing an image synthesizing method in a mobile communication terminal according to the present invention.

In order to achieve the above object, an image synthesizing method in a mobile communication terminal according to the present invention provides a background switching service for separating an original image into an object of interest area and a background area, and synthesizing the background image with another object of interest. In the supporting mobile communication terminal,

Designating a first area to be filled with only the other background image by using area information of the object of interest;

Filling an image corresponding to each region in a buffer using a direct memory access (DMA) for the previously designated first region; And

Synthesizing the other background image with the original image using a distance from a center point of the object of interest as a comparison reference with respect to the remaining area; Its features are to include.

According to the present invention, the step of designating a second area to be filled with only the original object of interest image using the area information of the object of interest;

Filling an image corresponding to each area in the buffer using a direct memory access (DMA) for the designated second area; Its features are further included.

According to the present invention, the first region filled with only the other background image is characterized by four rectangular regions.

In addition, according to the present invention, in designating the first area filled with only the other background image as four rectangles, one including all points located above a specific threshold located higher than an upper boundary point of the object region of interest With squares; One rectangle including all points located below a specific threshold lower than a lower boundary point of the object of interest area; A rectangle including all points located to the left of a specific threshold located further to the left than the left boundary point of the object of interest area, but including all points except those already included in the designated upper and lower rectangles; And one rectangle including all points located to the right of a specific threshold located to the right of the right boundary point of the object of interest area, but including all points except those already included in the upper and lower rectangles. The feature is that the first region is designated.

In addition, according to the present invention, the four rectangular regions designated as the first region are characterized in that the region is composed of several macro blocks.

According to the present invention, the second region to be filled with only the original object image of interest using the region information of the object of interest is characterized in that it is a rectangular region composed of a plurality of macroblocks. In designating the second region to be filled with only the original object image of interest, the feature is that the rectangle is completely included in the boundary region of the object of interest.

Further, according to the present invention, in synthesizing the other background image and the original image by using the distance from the center point of the object region of interest as a comparison reference, the shape of the object region of interest is an ellipse, the center of the object region of interest A point from a center of the object of interest region that is greater than a first threshold th1 greater than the reference unit, filled with the other background image, based on a distance from an elliptic boundary to an elliptic boundary; Filled with the original image for points at a distance less than a second threshold th2 smaller than the reference unit from the center of the object of interest area; For points that are less than the first threshold and greater than the second threshold from the center of the object of interest region, two points at the same location of the original image and the other background image are synthesized, but close to the first threshold. The feature is that the higher the specific gravity of the corresponding spot color of the other background image, the closer to the second threshold value, and the higher the specific gravity of the corresponding dot color of the original image.

In addition, according to the present invention, the synthesis of the other background image and the original image is characterized in that it is made in units of macro blocks; If the corresponding point to be synthesized is the point of the first position of each line of the macroblock, the distance from the center of the object region of interest is obtained, and then the distance increments between neighboring points of the same line are obtained; If the point being synthesized is a point other than the first position of the line between macroblocks, the distance is changed to the distance of the previous point using the distance increment between the already obtained neighboring points; Using the obtained distance, the original image and the other background image are combined and synthesized according to whether the corresponding distance is close to the first threshold or the second threshold.

According to the present invention, there is an advantage in that the input image is separated into the ROI and the background region, and the ROI and the other background image can be synthesized at high speed by using a limited CPU and a DSP chip.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In the present invention, a method of effectively synthesizing a face object and another background image using a DSP chip and a DMA in a mobile communication terminal is proposed. For example, a mobile communication terminal having such a structure may employ TI's TMS320C55x as a DSP chip, and the entire frame structure may adopt an OMAP 1510 structure employing two CPUs, a C55x and an ARM925 CPU.

Meanwhile, a general algorithm for synthesizing an image will be described below.

First, as shown in FIG. 2, an image may be divided into three regions. Here, 'region 1' is a background region to be filled with a specific background selected by the user, and 'region 3' is a portion to be filled with the original object image. In addition, 'area 2' is a portion where the original image and the area selected by the user appear to be synthesized, and are located within a specific range around the boundary area of the actually extracted object.

Within this range, the closer to 'area 1', the more weighted the background image is, and the closer to 'area 3', the more weighted to the original object image. The present invention shows an example in which an object is synthesized in an elliptic shape as a face region.

Therefore, as shown in FIG. 2, the distance from the object center point to the boundary between the region 3 and the region 2 is called 'Inner Dist', and the distance from the center point to the boundary between the region 2 and the region 1 is represented. If it is 'Outer Dist', the compound expression of point (x, y) in any 'area 2' is calculated as follows.

Here, A is the x, y color value of the original image containing the object, B is the x, y color value of the background image to be synthesized, α is (Dist-Inner Dist) / (Outer Dist-Inner Dist), β is (Outer Dist-Dist) / (Outer Dist-Inner Dist), and Dist represents a distance from the object center to the positions of the points to be synthesized, x and y, respectively.

Using this concept, the easiest method to think about is that for all the points in the image, find the distance from the center of the object using the location of the point, and if the distance is greater than 'Outer Dist', Otherwise, if it is smaller than 'Inner Dist', the original image including the object is used, and the existing image is synthesized by synthesizing the original image and the background image by using the formula described above for the existing points. Will be made.

In this case, one point is represented by three values of Y, Cb, and Cr in video compression standards such as H.263, MPEG1, and 2. In fact, when the image size is 176 * 144, Y is expressed for all pixels, but Cb Cr is expressed only as large as 88 * 72, which is one quarter of the area. Thus, compositing is done for all pixels in the case of Y, while Cb and Cr are processed for one pixel. This series of processes is described in FIG. 3.

This method is problematic in that it requires a lot of CPU and memory usage in various ways. First, we need to find the distance from the center of the object for every number of points, and compare the distance between 'Outer Dist' and 'Inner Dist' for each point. Synthesis or copying from an existing image should be performed.

In general, when executing a program, if there is a comparison statement in the loop statement, processing takes a very long time because different commands must be processed according to the comparison result. Computing the distance at every point also requires a lot of time, which results in a significant increase in CPU usage when the CPU is processing to move data values from a particular memory to memory, even without compositing. In addition, when using limited memory such as a DSP chip, it is not possible to upload the entire image to the internal memory. Therefore, an external memory should be used. In this case, processing speed is further reduced due to memory access time. In order to solve these various problems, we propose a method that can effectively synthesize an image as follows.

Memory Copy Using Direct Memory Access (DMA)

First, in the present invention, the data of the original image and the background image in the external memory are copied to the internal memory in macroblock units (16 * 16) using DMA. At this time, copy uses DMA, so the CPU processing time is not used.

Meanwhile, as shown in FIG. 4, the object region and the background region described above may be divided into five quadrangles. The macro block completely included in the area indicated by the five rectangles may read data values from the original image or the background image without the above-described comparison syntax or Dist calculation. That is, in FIG. 4, macroblocks belonging to areas 1 (top), 2 (bottom), 3 (left), and 4 (right) appear in the background image, and in the case of macroblocks belonging to area 5 (center). Just read the data value from the video. Areas 1, 2, 3, 4, and 5 know the object area and can be calculated automatically. By doing so, in the basic method described above, macro blocks corresponding to the complete background area and the complete object area can obtain an image using DMA without any CPU operation. In this case, since the size of the area 5 is not large, in some cases, the area 5 may be separately included and processed in the area to be synthesized.

Dist calculation by line

The region to be synthesized, that is, the portion corresponding to region 6 in FIG. 4, should be calculated for each point (position) in the basic method. This section introduces a method of calculating a Dist only at the first point of each line when processing a macroblock, and simply increasing or decreasing a value of a certain size to an existing Dist for the next point. First, read the macro block and calculate Dist at the first point position. However, for other points at the same Y position, a certain amount of value is increased or decreased in the dist of the previous point. The constant value to increase or decrease can be obtained experimentally from the first Dist. Of course, even if one line increases or decreases slightly depending on the x position, the error of one degree in one macro block cannot be distinguished by human vision. Instead, Dist is newly calculated each time as shown in FIG. 5 to prevent the error from expanding for other lines in one macro block. In this way, the Dist calculation can be reduced to 1/8 by performing the calculation.

This calculation method is described in detail as follows.

According to the present invention, a method of synthesizing an image differs depending on whether it is farther or closer than the boundary line of the ellipse. Therefore, here, the distance from the center of the ellipse to the point at the boundary is regarded as the reference Dist, and the image synthesis method at that point is determined depending on whether the distance from each point has a value larger or smaller than the reference Dist. Will be decided.

As is known, the expression of an ellipse is given by x ² / a ² + y ² / b ² = 1, where a is the major axis of the ellipse, and b is the short axis of the ellipse. It will be shown. For example, if the coordinates of the point (x, y) are above the boundary of the ellipse, the distance of the point from the midpoint of the ellipse is one.

That is, if coordinate (x, y) is on an ellipse, the distance Dist will be 1; if coordinate (x, y) is outside the ellipse, the distance Dist will be greater than 1, and conversely, coordinate (x, y) is inside the ellipse. If so, the distance Dist will be less than one.

On the other hand, if the center point of the ellipse has a coordinate value of (Cx, Cy), when the distance from the midpoint (Cx, Cy) of the ellipse is to be calculated at each (x, y) coordinate point, the equation is Dist = ( (Cx-x) * (Cx-x)) / a ² + ((Cy-y) * (Cy-y)) / b ² Here, (Cx-x) is the distance from the X coordinate (Cx) of the elliptic midpoint to the current X coordinate (x), and (Cy-y) is the current Y coordinate at the Y coordinate (Cy) of the elliptic midpoint. The distance to (y) (Euclidean distance) is shown, respectively (refer FIG. 6).

In this case, if the above distance calculation is performed for every coordinate, the operation of multiplication and division takes a lot of time. Therefore, we get an approximation of how much the distance changes as each pixel moves, and calculate the distance from each coordinate using only the addition / subtraction operation with a constant value without calculating the distance from the origin of the ellipse for each coordinate. Can be calculated.

In the above ellipse equation, when x increases by 1, the increase of the total distance can be expressed as

When x increases by one point, put that point in the equation,

(Cx- (x + 1)) ² / a ² + (Cy-y) ² / b ² = Dist1. Where the distance from the previous pixel is

(Cx-x) ² / a ² + (Cy-y) ² / b ² = The difference from Dist0 is the increase in distance when x increases by one.

In other words, the increment of distance when moving one pixel along the x-axis

= [(Cx- (x + 1)) ² / a ² + (Cy-y) ² / b ² ]-[(Cx-x) ² / a ² + (Cy-y) ² / b ² ] .

Solving the above equation, (2 * (Cx-x) -1) / a ² is the distance value that changes when one pixel is increased along the x-axis.

When moving one pixel on the Y axis, the concept of the increment calculation on the X axis is the same, so the value of the changing distance can be expressed as (2 * (Cy-y) -1) / b ² . do.

However, the DSP chip performs integer arithmetic, so if only the difference value is used continuously, the accumulation of errors occurs and the synthesis according to the distance value is unnatural. In order to prevent such a case, when one value is increased on the y-axis, the distance calculation using the actual current pixel position and the center of the ellipse is performed instead of the increment.

At this time, since the division by a ² , b ² in the expression representing the change amount of the above distance is a constant, it can be subtracted out of the Loop statement. Then, only multiplication and addition operations remain in the loop statement, which reduces the overall computation time.

Transformed composite calculation

Meanwhile, in performing image synthesis on the point (x, y) belonging to region 2 of FIG. 2, the above-described synthesis equation requires two multiplication operations, one division operation, and two addition operations. The equation described above can be derived in the same manner as follows.

Here, β / (α + β) can be applied to all the Y, Cb, and Cr values constituting the pixel if it is calculated only once per pixel. Therefore, when it is stored in the constant variable t and used, it can be synthesized as A- (A-B) * t without having to recalculate each of Y, Cb, and Cr. If you calculate and use one dist for four pixels using one Cb and Cr value, that is, four Y values and one Cb and Cr value, this multiplies one by six for each of the six values. The same operation can be performed with the operation and two subtraction operations.

The entire process described so far is listed in FIG. 7. First, when an image is input, the process of extracting an object region, using the extracted object region information, designating macro blocks to use only a background image as four rectangular regions (top, bottom, left, right), and specifying four designated regions DMA does this by instructing the part corresponding to the rectangular area to be used in the background image by using DMA, and designates macro blocks to use only the object image of the original image as one rectangular area by using the extracted object region information. And an area to which the pure background or the original image should be copied using the step of instructing the DMA to perform the DMA from the original image by importing a portion corresponding to the designated rectangular region.

Then, for each macroblock for the macroblocks corresponding to the remaining area, the distance from the object center in the case of the first pixel of a line is calculated, and the calculated distance is applied to the 8 pixels included in the line using the obtained distance. And updating the Dist using incremental and decrement if it is not the first pixel, and synthesizing Y, Cb, Cr values of the original image and the background image of each pixel using the obtained Dist. By synthesizing, all synthesis processes are complete.

8 shows a configuration of a mobile communication terminal capable of performing a background switching service through such a series of processes. 8 illustrates an example of a mobile communication terminal having an image input unit such as a camera. The mobile communication terminal according to the present invention includes three buffers (a raw image buffer) in which a DMA controller and an original image, a background image, and a synthesized image are stored. , A background buffer, a synthesis buffer), an image display unit on which the synthesized image is to be displayed, and a transmission means. The mobile communication terminal according to the present invention further comprises a DPS chip, an engine using the same, and an object extraction engine and a background synthesis engine.

As described above, according to the image synthesizing method in the mobile communication terminal according to the present invention, an input image is divided into a region of interest and a background region, and a background image different from the region of interest at a high speed by using a limited CPU and a DSP chip. There is an advantage that can be synthesized.

In addition, according to the video synthesis method of the mobile communication terminal according to the present invention, a method for effectively synthesizing the background in a mobile communication terminal including a DSP chip is proposed. There is an advantage that can be effectively used in the video transmission service using H.263, MPEG1, 2, 4, etc. in the standard format.

Claims (10)

A mobile communication terminal for supporting a background switching service that separates an original image into an object-of-interest region and a background region, and synthesizes a background image different from the object-of-interest region, Designating a first area to be filled with only the other background image by using area information of the object of interest; Filling an image corresponding to each region into a buffer using a direct memory access (DMA) for the previously designated first region; Synthesizing the other background image with the original image using a distance from a center point of the object of interest as a comparison reference with respect to the remaining area; Image synthesis method in a mobile communication terminal comprising a. The method of claim 1, Designating a second region to be filled with only the original ROI image by using the region information of the ROI; Filling an image corresponding to each area in the buffer using a direct memory access (DMA) for the designated second area; Image synthesis method in a mobile communication terminal further comprising. The method of claim 1, And a first area filled with only the other background image is designated as four rectangular areas. The method of claim 3, wherein In designating the first area filled with only the other background image as four rectangles, One rectangle including all points located above a specific threshold located higher than an upper boundary point of the object of interest area; One rectangle including all points located below a specific threshold lower than a lower boundary point of the object of interest area; A rectangle including all points located to the left of a specific threshold located further to the left than the left boundary point of the object of interest area, but including all points except those already included in the designated upper and lower rectangles; And One rectangle including all points located to the right of a specific threshold located to the right of the right boundary point of the object of interest area, but including all points except those already included in the designated upper and lower rectangles; And a first region is designated as the image synthesis method of the mobile communication terminal. The method of claim 3, wherein The four rectangular regions designated as the first region are regions composed of a plurality of macro blocks. The method of claim 2, And a second region to be filled with only the original ROI image by using the region information of the ROI. The image synthesis method of claim 1, wherein the second region is a rectangular region composed of a plurality of macroblocks. The method of claim 6, And specifying a second area to be filled with only the original ROI image by using the area information of the ROI. The image synthesis method of claim 1, wherein the quadrangle is completely included in the boundary area of the ROI. The method of claim 1, In synthesizing the other background image with the original image using the distance from the center point of the object region of interest, the shape of the object region of interest is an ellipse and the distance from the center of the object region of interest to an elliptic boundary. With reference unit, For the points at a distance greater than the first threshold th1 larger than the reference unit from the center of the object region of interest, the other background image is filled in, For the points at a distance smaller than the second threshold th2 smaller than the reference unit from the center of the object region of interest, the circle image is filled with the original image, For points that are less than the first threshold and greater than the second threshold from the center of the object of interest region, two points at the same location of the original image and the other background image are synthesized, but close to the first threshold. A method of synthesizing a mobile communication terminal, characterized by synthesizing a larger specific gravity of a corresponding dot color of another background image, and having a larger specific gravity of a corresponding dot color of an original image as it approaches the second threshold. The method of claim 8, The image synthesis method of the mobile communication terminal, characterized in that the synthesis of the other background image and the original image is a macro block unit. The method of claim 8, In performing the synthesis of the other background image and the original image in units of macro blocks, If the corresponding point to be synthesized is the point of the first position of each line of the macro block, the distance from the center of the object region of interest is obtained, and then the distance increment between neighboring points of the same line is obtained. If the point being synthesized is a point other than the first position of the line between macroblocks, change the distance using the distance increment from the previous point to the distance of the previous point, And synthesizing and combining the original image and the other background image according to whether the distance is close to the first threshold or the second threshold using the obtained distance.