KR20040052142A - Mobile station and operating method for thereof - Google Patents

Abstract

PURPOSE: A mobile terminal and its operating method are provided to allow a user to compound a specific object such as a face and a selected background, edit it together with text, and then send a mail. CONSTITUTION: An image is obtained by using a camera and stored(S201). One of a background change mail service, a frame mail service and a character mail service is selected by using a face region to be extracted(S202). A suitable background, frame or character is selected according to a kind of the selected mail service(S203). the face region is extracted from the obtained image(S204). The selected background and the extracted face region are compounded(S205). A text is edited as necessary(S206) and the edited image is transmitted(S207).

Description

Mobile communication terminal and operating method thereof

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a mobile communication terminal. In particular, in a mobile communication terminal that supports a multimedia mail service (MMS) such as video mail, a user edits a background such as a face by combining a specific object such as a face with a text selected by the user. The present invention relates to a mobile communication terminal capable of transmitting a mail and a method of operating the same.

Recently, in addition to the basic service functions of the mobile communication terminal called communication means, various additional services have been expanded and provided to the mobile communication terminal. Among them, MMS (Multimedia Mail Service), which provides a multimedia service, has been in the spotlight recently based on IMT2000, CDMA2001, and the like, and still image mail service using a camera has already been commercialized.

In the next few years, video mail using video as well as still images is expected to begin in earnest. Although video mail is simply storing and transmitting video images, many companies have various kinds of differentiated services in mind because it can be used by many users only through different editing functions and services. We are developing an application.

As one of the differentiated MMS application products, the present invention proposes a method for enabling differentiated mail service through various editing functions such as text by automatically / manually extracting a face region.

Disclosure of Invention The present invention provides a mobile communication terminal capable of automatically extracting a face area and efficiently performing user editing with another background in a mobile communication terminal supporting MMS (Multimedia Mail Service), and an object thereof. have.

In addition, the present invention, in the mobile communication terminal supported by MMS (Multimedia Mail Service), while extracting the face region using the object extraction engine of interest to automatically extract the face region, using a manual object extraction interface as a technical limitation Another object of the present invention is to provide a face region extraction method in a mobile communication terminal capable of overcoming and effectively extracting a face region.

In addition, the present invention, in the mobile communication terminal that supports MMS (Multimedia Mail Service), the face area in the mobile communication terminal that can extract a more accurate face area in extracting the face area from a single image, such as a still image It is another object to provide an extraction method.

1 is a diagram illustrating an example of extracting a face region and performing various edits using the same by a method of operating a mobile communication terminal according to the present invention.

2 is a flowchart illustrating a process of extracting a face region and performing various edits using the same according to a method of operating a mobile communication terminal according to the present invention.

3 to 5 are views illustrating an example in which a user designates a range of a face extraction region by a face region extraction method in a mobile communication terminal according to the present invention.

6 is a flowchart illustrating a process of extracting a face region by a method for extracting a face region in a mobile communication terminal according to the present invention.

7 is a block diagram schematically showing the configuration of a mobile communication terminal according to the present invention;

<Explanation of symbols for the main parts of the drawings>

700. Mobile terminal 701. Video input means

Image storage means703 ... Object extraction engine of interest

704 ... Recordable amount display 705 ... Background synthesis engine

706 ... Initial Object Scoping Scoping Interface

707 ... Background Storage Database 708 ... Manual Object Extraction Interface

709 ... Background Selection Interface 710 ... Edit Mode Selection Interface

711 ... means of transport

In order to achieve the above object, a mobile communication terminal according to the present invention,

Image input means for image acquisition;

Image storage means for storing an image obtained by the image input means;

A face region extraction engine for extracting a face region from the image stored in the image storing means;

An editing mode selection interface configured to receive a type of editing mode from a user with respect to a synthesis mode in which user editing is to be performed using the extracted face region; And

A background synthesis engine configured to perform user editing using the extracted face region according to a user edit mode selected through the edit mode selection interface; Its features are to include.

Here, according to the present invention, the synthesis mode in which the user editing is to be performed includes: a background switching mode for switching an area other than the extracted face area to another background, and a frame for placing the extracted face area in a frame such as a frame and combining the same And a character switching mode for positioning and extracting the extracted face region on a specific character.

In addition, according to the present invention, in recording an image input from the image input means, further comprising a recordable amount display unit for providing a user with information on how much more image can be recorded in the image storage means; Has its features.

In addition, according to the present invention, in extracting the face region from the image stored in the image storage means, it is characterized in that it further comprises a manual object extraction interface that can receive a face region extracted manually by the user.

In addition, according to the present invention, in extracting the face region from the image stored in the image storage means in the face region extraction engine, the initial object extraction range specified by the user for the region to be extracted for the face area is specified in advance Its feature is that it further includes an interface.

In addition, according to the present invention, in the background synthesis engine, according to the user editing mode selected through the editing mode selection interface, in the user editing using the extracted face region, for the background image to be edited from the user It is characterized in that it further includes a background selection interface for receiving a selection.

In addition, the operation method of the mobile communication terminal according to the present invention to achieve the above object,

Acquiring and storing an image;

Receiving a type of editing mode from a user with respect to a synthesis mode in which user editing is to be performed using the face region to be extracted from the stored image;

Selecting a background image to be used in the user editing mode;

Extracting a face region from the stored image; And

Performing image synthesis in a user editing mode by using the extracted face region and the selected background image; Its features are to include.

According to the present invention, after the image synthesis is performed in the user edit mode, the image of the user edit mode and the text (text) input from the user is characterized in that it further comprises a point.

In addition, according to the present invention, the synthesis mode in which the user editing is to be performed includes: a background switching mode for switching a region other than the extracted face region to another background, and a frame for placing and synthesizing the extracted face region in a frame such as a frame; It is characterized in that it is one of a mode and a character switching mode in which the extracted face region is placed and synthesized in a specific character.

In addition, in order to achieve the above another object, the method for extracting a face region in a mobile communication terminal according to the present invention,

Receiving, from the user, an initial region to which face region extraction is to be performed on the input video;

Extracting a face region of a first frame within an initial region designated by the user; And

Extracting a face region in a subsequent frame with reference to the extracted face region in the previous frame; Its features are to include.

According to the present invention, when the initial region to be extracted from the face region is designated by a user, a predetermined quadrangle is determined by the position selection through the movement key and the selection key of the mobile communication terminal on the image frame on which the face region extraction is to be performed. The characteristic is that the initial area is specified by inputting two end points of my diagonal line.

In addition, according to the present invention, when receiving an initial region from which the face region extraction is to be performed by a user, a plurality of horizontally and vertically divided images are displayed on an image display unit of a mobile communication terminal, and the divided images are displayed in a rectangle. The characteristic is that the initial region is designated by inputting coordinates of pixels corresponding to two end points of the diagonal line.

In addition, according to the present invention, in receiving an initial region from which a face region extraction is to be performed by a user, with respect to the basic region displayed on the image display unit of the mobile communication terminal, the initial region is adjusted by adjusting the size and position of the basic region. It is characterized by this point.

In addition, according to the present invention, the basic region displayed on the image display unit of the mobile communication terminal is characterized in that the rectangular or elliptical region presented by the result automatically extracted by the facial region extraction engine.

In addition, in order to achieve the above another object, the method for extracting a face region in a mobile communication terminal according to the present invention,

Automatically extracting a face region with respect to the input video; And

For the face area extracted by the user by frame, the user sets the resize mode and position control mode by using the mode select button of the mobile communication terminal, adjusts the size and position by using the move button, and uses the select button. Adjusting the size and position of the extracted face region by selecting the adjusted state; Its features are to include.

In addition, in order to achieve the above another object, the method for extracting the face region in the mobile communication terminal according to the present invention, in extracting the face region from the input still image,

Extracting a face region with respect to a plurality of frames consecutively before and after the still image to which the face region is to be extracted;

Obtaining a skin color region from each of the face regions extracted from the plurality of frames, and obtaining a central color constituting the skin color regions;

Performing grouping of similar colors with respect to the center color on the still image from which the face region is to be extracted; And

Obtaining a lump area composed of flesh colors belonging to the group, and determining whether the lump area is an ellipse shape and whether the shape of the eyes and the mouth is present in the lump, and extracting a face area; Its features are to include.

In addition, in order to achieve the above another object, the method for extracting the face region in the mobile communication terminal according to the present invention, in extracting the face region from the input still image,

Acquiring a plurality of frames including a still image from which the face region is to be extracted and a continuous still image before the still image; And

After extracting the face region of each frame by using the acquired plurality of frames, the face region is extracted from the video, the face region extracted in the last frame is designated as the face region extracted from the still image. There is this.

According to the present invention, in a mobile communication terminal that supports MMS (Multimedia Mail Service), there is an advantage that the user can automatically edit the face area and effectively edit the user with another background.

In addition, according to the present invention, in the mobile communication terminal that supports MMS (Multimedia Mail Service), the face area is extracted using the object extraction engine of interest to automatically extract the face area, but using the manual object extraction interface together technical It has the advantage of overcoming the limitation and extracting the face area effectively.

Hereinafter, with reference to the accompanying drawings will be described embodiments of the present invention;

1 is a diagram illustrating an example in which a face region is extracted and various editing is performed by using the method of operating a mobile communication terminal according to the present invention.

As an example of editing using such a face region extraction, a background switching mail service capable of editing an image by replacing a background except for a face region with another background designated by a user, as shown in FIG. A frame mail service for editing an image by inserting and editing an extracted face region in a frame (frame) designated by a user as shown in (b), and adjusting the size and position of the face region to a part of a specific character as shown in FIG. Character mail service that can be adjusted and replaced and edited can be provided. In order to easily use such a service by a user using a mobile communication terminal, an effective user interface is needed, which will be described later.

2 is a flowchart illustrating a process of extracting a face region and performing various edits using the same according to an operating method of a mobile communication terminal according to the present invention.

First, an image is acquired using an image input means such as a camera, and then stored for the acquired image (step 201). Then, the user selects which mail service to use from the background switching mail service, the frame mail service, or the character mail service using the extracted face region (step 202).

Then, according to the type of mail service selected in step 202, a suitable background, frame or character is selected (step 203), a face area is extracted from the obtained image (step 204), and the selected background and extracted face area Perform the synthesis on (step 205). At this time, if necessary, text editing is added to complete editing of the image, and then the edited image is transmitted (step 207).

In performing the above process, when a user records an image, a limited memory should be used due to the characteristics of the mobile communication terminal. If the user performs recording without considering the remaining memory amount, the user may not be able to acquire a desired image. Therefore, an interface (recordable amount display unit) should be provided to inform the user of the remaining capacity of the memory at the time of video recording so that the user can finish recording properly.

The recordable amount indicator may simply indicate the amount of remaining capacity immediately (for example, 100M), but it is difficult for the user to predict how long the remaining number of recordings can be performed. Calculate the size of the recorded data against the time of the camera to inform you of the remaining recording time. In this case, it may be more effective to consider the amount of recording for a predetermined time closest to the present instead of the amount of data recorded so far. For example, you can calculate the average amount of video recorded over the previous two seconds, predict the future recordable time, and provide that information to the user.

In addition, in the process of extracting a face region, generally, an automatic extraction algorithm using a face region extraction algorithm may be used. However, since the face region extraction algorithm is not very accurate with the current technology, it can be extracted more effectively with the help of the user in some cases. As an effective face region extraction method, the following methods are suggested.

1) Initial range setting method

First, the user sets an approximate area where the initial face area is located to a rectangle. Next, the face region is extracted from the rectangle set by the user using an automatic extraction algorithm. In general, the face region extraction algorithm shows relatively low performance in finding a face region belonging to a complex background, but shows a high performance in a limited range. In addition, it is easy to find the exact face area in the next frame when extracting the correct face area in the first frame, but it is easy to extract the wrong face area in the subsequent frame when the face area is extracted incorrectly in the first frame.

Therefore, the range initially set by the user may be easy but the effect may be large. 3 shows one such example. The user may designate a quadrangle in various ways. Here, a case in which a detection area is designated by displaying two points existing at diagonal positions of the quadrangle will be described as an example. In FIG. 3, point A and point B become two points selected by the user.

In addition, there may be several ways to specify a rectangle by specifying two points. First, a method of moving a pointer to a desired point by using an arrow button of a user interface unit on a video display unit of the general mobile communication terminal described in FIG. 3 and designating a point using a selection button may be used.

As another rectangular designation method, as shown in FIG. 4, a divided screen appears on the video display unit, and two points may be input by inputting the horizontal and vertical coordinates of each divided cell to the mobile communication terminal. have.

As another quadrangle designation method, as illustrated in FIG. 5, a face region is basically displayed in a quadrangle shape or an ellipse shape, and a user may designate a desired area by repositioning and adjusting the size. Here, in the case of adjusting the repositioning or size, after setting the repositioning mode or the resizing mode using the menu button, the position adjustment and the resizing may be performed by moving the arrow. In this case, the basically provided rectangle may be displayed in an arbitrary size in the center, or may be represented using an extraction result using an automatic extraction algorithm.

2) Manual face area extraction from still image

The above introduced method can be used to initially describe the approximate face area to extract the face area from the video. Using the same method, face regions can be extracted completely from still images. That is, as shown in FIG. 1, the face region is represented by an ellipse, and thus, a quadrangle including an ellipse may be designated using the above-described rectangle description method.

3) Modify the result of extracting the face area from the video

If the face area is automatically extracted from the video, the result may be wrong in some cases. In this case, the user may check each frame, and if it is wrong, the result may be corrected by using one of the rectangle designation methods described above. In this case, the last method, adjusting the size and position of the existing rectangle may be more effective.

In detail, the facial region may be more accurately extracted through the following process. That is, according to the method for extracting a face region in a mobile communication terminal according to the present invention, the method includes: automatically extracting a face region with respect to an input video; Verifying, by the user, a face extraction result in each frame with respect to the extracted face area; And correcting the extracted face region result by adjusting the size and position of the face region extracted by the frame by the user. Facial region extraction is performed, including.

4) Facial Region Extraction from Still Image Using Multiple Images

In general, it is more accurate to extract a face region from a continuous frame than to extract a face region from a single image. The reason is that the face regions are similar in successive frames, and the final result can be inferred by combining the results of several sheets. Using this principle, even if a user acquires a single still image to extract a face region and performs background switching, the result of face region extraction can be further improved by acquiring N consecutive images before and after the internally acquired still image. have.

In order to infer the final result of the face region in several images, the face region is extracted from each image, and the skin-based color extraction algorithm is extracted from the region indicated as the face region in many images. And use.

As shown in FIG. 6, the color-based face region extraction algorithm first acquires a flesh color region on an input image and then performs similar grouping on the basis of the most distributed flesh color (center color) (steps 601 to 604). ). Then, a lump area composed of flesh colors belonging to the group is acquired (steps 605 and 606), and a face area is obtained through a face verification process (steps 607 and 608).

In this case, the face verification process includes determining whether the lump is an oval shape and examining whether there is an eye and a mouth inside the lump. In this case, if the color used as the center of the skin color grouping is displayed in an area other than the face, incorrect results can be obtained.If multiple sheets are used, grouping around the main skin color in the area that is commonly determined to be a face area is used. This problem can be solved.

As another method, there may be a method of acquiring a previously continuous frame of the acquired still image and extracting the face region using the same method as the face region extraction algorithm in the video using the same. The extracted face region in the frame becomes the extracted face region of the still image. At this time, the facial region extraction method in the video is a method of extracting the central flesh color from the area corresponding to the facial region extracted from the previous frame, and then extracting the facial region using the flesh color region from the current frame based on the obtained central flesh color. Can be.

On the other hand, Figure 7 is a block diagram schematically showing the configuration of a mobile communication terminal according to the present invention.

Referring to FIG. 7, the mobile communication terminal 700 according to the present invention includes an image input means 701 such as a camera, and an image storage means 702 in which an image input from the image input means 701 is stored. It is configured to include.

In addition, the mobile communication terminal 700 according to the present invention includes a recordable amount display unit 704 that displays a recordable amount in the future, an object of interest extraction engine 703 that can edit the recorded image around the face region, and a background. It includes a compositing engine 705, and includes an initial object extraction range specification interface 706 and a manual object extraction interface 708 that allow a user to extract objects semi-automatically or manually.

In addition, the mobile communication terminal 700 according to the present invention includes a background storage database 707 in which a background image is stored, and a background selection interface 709 in which a user can select a background to be synthesized from the background storage database 707. And an editing mode selection interface 710 for selecting whether to perform a simple background change, a frame change, or a character change. In addition, the mobile communication terminal 700 according to the present invention further comprises a transmission means 711 for transmitting a multimedia mail in which a background change, a frame change or a character change is performed by a user's selection.

As described above, according to the mobile communication terminal and the operation method according to the present invention, in a mobile communication terminal supported by MMS (Multimedia Mail Service), it is possible to automatically extract the face area to perform user editing effectively with other backgrounds. There is an advantage.

In addition, according to the method for extracting a face region in a mobile communication terminal according to the present invention, in a mobile communication terminal supported by MMS (Multimedia Mail Service), a face region is extracted using an object of interest extraction engine that automatically extracts a face region. However, there is an advantage of using the manual object extraction interface together to overcome the technical limitations and extract the face region effectively.

Claims (17)

Image input means for image acquisition; Image storage means for storing an image obtained by the image input means; A face region extraction engine for extracting a face region from the image stored in the image storing means; An editing mode selection interface configured to receive a type of editing mode from a user with respect to a synthesis mode in which user editing is to be performed using the extracted face region; And A background synthesis engine configured to perform user editing using the extracted face region according to a user edit mode selected through the edit mode selection interface; Mobile communication terminal comprising a. The method of claim 1, The synthesis mode in which the user edit is to be performed, A background switching mode for switching an area other than the extracted face area to another background, a frame mode for placing and extracting the extracted face area on a frame such as a frame, and combining the extracted face area with a specific character And a character switching mode. The method of claim 1, In recording the video input from the video input means, the mobile communication terminal further comprises a recordable amount display unit for providing the user with information on how much video can be recorded in the video storage means; . The method of claim 1, And extracting a face region from the image stored in the image storing means, the manual object extracting interface capable of receiving a face region which is manually extracted by the user. The method of claim 1, In extracting the face region from the image stored in the image storage means in the face region extraction engine, the face region extraction engine further comprises an initial object extraction range designation interface which is previously inputted by the user for the region to be extracted. Characterized in a mobile communication terminal. The method of claim 1, In the background synthesis engine, in performing user editing using the extracted face region according to a user editing mode selected through the editing mode selection interface, a background for receiving a selection of a background image to be edited from a user is input. A mobile communication terminal further comprising a selection interface. Acquiring and storing an image; Receiving a type of editing mode from a user with respect to a synthesis mode in which user editing is to be performed using the face region to be extracted from the stored image; Selecting a background image to be used in the user editing mode; Extracting a face region from the stored image; And Performing image synthesis in a user editing mode by using the extracted face region and the selected background image; Method of operation of a mobile communication terminal comprising a. The method of claim 7, wherein And after the image synthesis in the user edit mode is performed, synthesizing the image in the user edit mode with text input from the user. The method of claim 7, wherein The synthesis mode in which the user edit is to be performed, A background switching mode for switching an area other than the extracted face area to another background, a frame mode for placing and extracting the extracted face area on a frame such as a frame, and combining the extracted face area with a specific character Method of operating a mobile communication terminal, characterized in that one of the character switching mode. Receiving, from the user, an initial region to which face region extraction is to be performed on the input video; Extracting a face region of a first frame within an initial region designated by the user; And Extracting a face region in a subsequent frame with reference to the extracted face region in the previous frame; Facial region extraction method in a mobile communication terminal comprising a. The method of claim 10, In receiving the initial region to be extracted from the face region from the user, A face in a mobile communication terminal characterized in that an initial region is designated by inputting two endpoints of a diagonal line in a predetermined rectangle by selecting a position using a moving key and a selection key of the mobile communication terminal on the image frame on which the face region extraction is to be performed. Region extraction method. The method of claim 10, In receiving the initial region to be extracted from the face region from the user, The video display unit of the mobile communication terminal is divided into a plurality of horizontal / vertical image is displayed, the movement is characterized in that the initial area is designated by inputting the coordinates of the pixels corresponding to the two end points of the diagonal line in the divided image Facial region extraction method in communication terminal. The method of claim 10, In receiving the initial region to be extracted from the face region from the user, A face area extraction method in a mobile communication terminal, characterized in that an initial area is designated with respect to a base area displayed on an image display unit of a mobile communication terminal by adjusting the size and position of the base area. The method of claim 13, And a basic region displayed on the image display unit of the mobile communication terminal is a rectangular or elliptical region presented by a result extracted automatically by the facial region extraction engine. In extracting the face area from the input still image, Extracting a face region with respect to a plurality of frames consecutively before and after the still image to which the face region is to be extracted; Obtaining a skin color region from each of the face regions extracted from the plurality of frames, and obtaining a central color constituting the skin color regions; Performing grouping of similar colors with respect to the center color on the still image from which the face region is to be extracted; And Obtaining a lump area composed of flesh colors belonging to the group, and determining whether the lump area is an ellipse shape and whether the shape of the eyes and the mouth is present in the lump, and extracting a face area; Facial region extraction method in a mobile communication terminal comprising a. In extracting the face area from the input still image, Acquiring a plurality of frames including a still image from which the face region is to be extracted and a continuous still image before the still image; And After extracting a face region of each frame by using the obtained plurality of frames, a face region is extracted from the video, the face region extracted in the last frame is designated as the face region extracted from the still image. Facial region extraction method in mobile communication terminal. Automatically extracting a face region with respect to the input video; And For the face area extracted by the user by frame, the user sets the resize mode and position control mode by using the mode select button of the mobile communication terminal, adjusts the size and position by using the move button, and uses the select button. Adjusting the size and position of the extracted face region by selecting the adjusted state; Facial region extraction method in a mobile communication terminal comprising a.