JP2004171543A - Method and device for image processing - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method capable of displaying a portrait clearly and increasing amusement characteristics. <P>SOLUTION: The image processing method comprises the following steps: a step of detecting the position of a body area in an inputted image; a step of deciding the position of an origin of the coordinate system of a decoration, matching the position of the detected body area; a step of outputting an image with a decoration deployed in the inputted image, based on the decided position of the origin; and a step of moving the decoration corresponding to the movement of the portrait in the inputted image. By this arrangement, even when the portrait and the decoration both move, the portrait is not interfered by the decoration and displayed clearly. The movement of the decoration is made to match the movement of the portrait, thereby increasing amusement characteristics. <P>COPYRIGHT: (C)2004,JPO

Description

  The present invention relates to an image processing device incorporated in a communication device using an image (for example, a device for videophone, video conference, video mail or video chat, a door phone, and the like), and more particularly, to a person image (for example, a user). And an image processing apparatus for editing a face image, a whole body image, and the like.

  There are amusement machines that print an image of a user on various print media and output the image. In this type, a frame of a decoration (for example, a pattern or a pattern) is superimposed on a photographed human image, synthesized, printed on a printing medium (for example, a sticker), and output.

  Such a game machine does not have a function of adjusting the positional relationship between a person image and a decoration by itself. Therefore, even if the user arranges the portrait image so that it fits within the decoration frame, the decoration may overlap the portrait image, and the face and head portions may be hidden, and the portrait image may be unclear. Was.

  Patent Document 1 (Japanese Patent Laid-Open No. 2000-22929) considers this point, and the decoration is applied to a body part of a person image, particularly, a body part area set corresponding to the face or head. We propose a technique to add decorations so that they do not overlap.

On the other hand, recently, a camera-equipped mobile phone having a function of superimposing a decorative image on a captured human image with a decorative frame and transmitting the synthesized image has been put into practical use. When the sender uses such a camera-equipped mobile phone, the receiver can enjoy the enjoyment of viewing the image edited by the sender.
JP-A-2000-22929 (pages 3-7)

  Patent Document 1 assumes that the decoration does not move. In the technique described in this document, decorations are added to the body part of a person image, particularly to a body part region set corresponding to the face or head so that the decoration does not overlap.

  In this document, printing on a print medium is used as an output, and it is generally impossible to edit an image once printed after printing, so this concept can be said to be appropriate.

  However, camera-equipped mobile phones as described above have become widespread, and here, the composite image is usually displayed on a display without being printed on a print medium. And in the future, users will want to move the decoration to get more fun.

  However, the prior art only considers adding an immovable decoration to an immovable portrait, and does not consider adding a moving decoration. Therefore, if the conventional technique is simply applied to a moving person image, the person image and the decoration interfere with each other, and the person image becomes unclear or cannot be used to enhance playability.

  Therefore, an object of the present invention is to provide an image processing method and a technique for linking the image processing, which can clearly display a person image and can enhance playability.

  In the image processing method according to the first aspect, a step of detecting the position of the body part region in the input image, a step of determining the position of the decoration coordinate system in accordance with the detected position of the body part region, Outputting an image in which decorations are arranged on the input image based on the input image.

  According to this configuration, when the human subject moves in the input image, the decoration can be moved according to the movement.

  In the image processing method according to the second aspect, in the outputting step, image data in which the decoration is drawn on the input image is output.

  With this configuration, it is not necessary to combine the input image and the decoration at the transfer destination of the output image.

  In the image processing method according to the third aspect, the outputting includes outputting image data in which the input image and the decoration are associated with each other.

  With this configuration, the destination of the output image is easy and convenient to restore the input image.

  In the image processing method according to the fourth aspect, the input image and the decoration are associated by a metafile.

  With this configuration, the input image and the decoration can be simply associated.

  In the image processing method according to the fifth aspect, the metafile is in the SMIL format.

  With this configuration, it is possible to easily associate the input image with the decoration.

  In the image processing method according to the sixth aspect, a step of detecting the size of the body part region in the input image, a step of determining a scale of the coordinate system of the decoration according to the size of the detected body part region, Outputting an image in which decorations are arranged on the input image based on the input image.

  According to this configuration, when the human subject approaches the image pickup body, the size of the decoration and the movement path both increase. Conversely, when the distance increases, the size of the decoration and the movement path both decrease. Accordingly, it is possible to make the decoration appear to follow the movement of the person subject in the front-rear direction.

  In the image processing method according to the seventh aspect, a step of detecting a rotation amount of the body part region in the plane of the input image, and a step of determining a rotation amount of the decoration coordinate system in accordance with the detected rotation amount; Outputting an image in which decorations are arranged on the input image based on the determined rotation amount.

  According to this configuration, when the face is tilted sideways, the decoration can be tilted, and the moving path of the decoration can also be tilted. Therefore, even if the rotation direction changes in the plane of the input image of the person, , You can make it look like the decoration is following.

  In the image processing method according to the eighth aspect, a step of detecting the amount of rotation of the body part region with respect to the plane of the input image, and a step of determining the amount of rotation of the decoration coordinate system in accordance with the detected amount of rotation are determined. Outputting an image in which decoration is arranged on the input image based on the rotation amount.

  According to this configuration, when the person is “turning right”, the decoration and the movement path can be deformed according to the rotation of the person. Therefore, it is possible for the person to make the decoration seem to follow the change in the orientation of the input image with respect to the plane.

  In the image processing method according to the ninth aspect, the step of detecting the body part region in the input image and the degree of opacity of the decoration are different between the region on the foreground side and the region on the background side of the detected body part region. And outputting an image in which decorations are arranged on the input image based on the determined opacity.

  According to this configuration, the decoration can be erased when the movement path of the decoration comes to the back of the body part area, so that the decoration surrounds the body part area and moves three-dimensionally around it. You can make it look like you are.

  In the image processing method according to the tenth aspect, when the decoration collides with the body part region, the moving direction of the decoration is changed to a direction away from the body part region.

  According to this configuration, the decoration can be given a motion that reflects when the decoration collides with the body part region.

  In the image processing method according to the eleventh aspect, the body part region is a face region of a human subject.

  In this configuration, since the face clearly shows the intention and emotion of the person, if the face region is selected as the body part region, the effect of enhancing the playability is great.

  According to the present invention, even when both the person image and the decoration are moving, the decoration can be prevented from interfering with the person image, the person image can be clearly displayed, and the movement of the decoration is determined by the movement of the person image. By cooperating with, the playability can be enhanced.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, important terms will be described prior to the description of a specific configuration.

  The “person image” is an image including an image of a part or all of a person. Therefore, the image may be an image of the entire person, an image of only the person's face or back, or an image of only the upper body. Further, an image in which a plurality of persons are shown may be used. Of course, any pattern such as a landscape or pattern other than a person may be present in the background.

  "Body part" means a part of a person's body, and can be recognized as a part of a person's body even if the skin wearing clothes, hats and shoes is not visible on the body part Is included in the body part. Thus, the face is also a body part, and the head is also a body part. In addition, eyes, nose, mouth, ears, eyebrows, hair, neck, dressed upper body, hands, arms, legs, legs with shoes, head with hat, eyes with glasses, etc. is there.

  The "body part region" is a region defined as a region occupied by a body part in a person image, and the body part region may include a part that is not a body part therein, or may include a part inside the body part. Area.

  That is, assuming that the body part is a face, the body part region may include a region where the body part region corresponding to the face is around the face, or the minimum size surrounding the eyes, mouth, and nose. It may be a rectangle that does not include the forehead or ears.

  “Decoration” refers to an image pattern to be added, and this image pattern may be an image pattern stored in advance or an image pattern generated by computer graphics technology. The image pattern may be a character or symbol pattern or a graphic pattern.

  FIG. 1 is a functional block diagram of an image processing apparatus according to an embodiment of the present invention. As shown in FIG. 1, the image processing apparatus according to the present embodiment has the following components.

  The image input unit 1 obtains an input image. In the present embodiment, a person image is included in the input image, and this person image is allowed to move, and the input image usually means an image of one frame of the moving image.

  However, this input image may be an image immediately after being photographed by the camera, or after photographing, data encoded according to an encoding method such as MPEG (an image recorded on a recording medium may be loaded, It may be one obtained by decoding the data received from the transmission device.).

  The control unit 2 controls each element shown in FIG. 1 according to the flowchart in FIG.

  The image output unit 3 outputs, as an output image, an image in which the decoration is combined with the input image. This output image can be displayed on a display device, coded according to the above-described coding method and stored in a recording medium, or transmitted by another communication device.

  The input image storage unit 4 stores the input image input by the image input unit 1.

  The template storage unit 5 stores a template of the body part region. Hereinafter, in the present embodiment, the face area is taken as the body part area.

  As shown in FIGS. 6 (a), 6 (b) and 6 (c), templates are prepared by modeling contours of face parts (head, eyes, nose, mouth) in various sizes. use.

  Further, as shown in FIGS. 7B and 7C, a template having a rotation amount in the plane of the input image is used in addition to the template having the basic posture shown in FIG. 7A. Thereby, the inclination due to the rotation of the face can be detected.

  Further, as shown in FIGS. 8B and 8C, a template having a rotation amount with respect to the plane of the input image is used in addition to the template having the basic posture shown in FIG. 8A. Accordingly, when the face orientation is different from the basic orientation (when the normal vector passing through the center of the face is not orthogonal to the plane of the input image), the face orientation can be detected.

  In FIG. 1, the detection unit 6 detects a body part region from the input image stored in the input image storage unit 4 using the template in the template storage unit 5.

  Here, as shown in FIG. 9, the detection unit 6 applies a differential filter to the input image of the input image storage unit 4 to extract an edge component. Further, the detecting unit 6 selects the template 53 from the template storage unit 5, and performs pattern match matching using the selected template 53 and the edge component.

  In this pattern matching, the template 53 (Nx * Ny pixels) is moved on a search range (Mx-Nx + 1) (My-Ny + 1) in a larger input image (Mx * My pixels), and the residual This is a process for obtaining the upper left position (a, b) of the template image such that R is minimized.

  However, in (Equation 1), I (a, b) (mx, my) is a partial image of the input image, and T (mx, my) is an image of the template 53.

  Further, instead of (Equation 1), the cross-correlation coefficient C represented by the following equation may be obtained, and the upper left position (a, b) of the template image at which the cross-correlation coefficient C becomes maximum may be obtained.

  The detection unit 6 uses various templates shown in FIGS. 6, 7 and 8, and uses (Equation 1) to minimize the residual R, and (Equation 2) to use The template that maximizes the relation number C is obtained. Then, the detecting unit 6 sets the position, size, inclination, and orientation of the obtained template as the respective values of the face image in the input image.

  In the case of a face image, by performing the same processing for each frame (or for every several frames), it is possible to obtain the transition of the face position, size, inclination, and orientation.

  When a plurality of faces are present in the image, the detection unit 6 determines a position where the residual R is low or a position where the correlation coefficient C is high in an area excluding the already detected first person face partial area. Is detected as the second person's face region.

  Similarly, for the third and subsequent faces, the detection unit 6 sets a position where the residual R is low and a position where the correlation coefficient R is high in the region excluding the already detected face region as the face region. Hereafter, detection is repeatedly performed until the residual R becomes larger than a predetermined threshold value or the correlation coefficient C becomes smaller. As a result, it is possible to detect the face part regions of a plurality of persons.

  Of course, it is not necessary to always perform all of the above detections, and may be appropriately selected except for the detection of the position and size of the face area.

  In FIG. 1, the decoration information storage unit 7 stores decoration information. Next, the decoration information will be described with reference to FIG.

  In the example shown in FIG. 10A (decorative information (part 2)), the decorative information is configured by associating decorative image data with a transform property.

  The “decorative image data” field stores the decorative image data itself, or an ID or pointer indicating the decorative image data itself. As shown in FIG. 10A, the decorative image data may be composed of only image data, or may have additional attributes (such as a behavior at the time of collision). .

  The field of "transform property" is information on how to move the decoration with respect to the decoration layer and how to overlap the decoration layer.

  Specifically, the decoration is defined by a transform property by a local coordinate system (with the anchor point as the origin) different from the input image, and this coordinate system further includes a position, a scale (scale ratio). , Rotation, and opacity.

  More specifically, the anchor point indicates the position and direction of the coordinate origin of the local coordinate system relating to the decoration. The values set in the "position", "scale", and "rotation" fields are the actual image position, scale (enlargement / reduction ratio), and rotation with respect to the coordinate system (local coordinate system) of the anchor point. Indicates the degree of

  In the example (decorative information (1)) shown in FIG. 10B, the decorative information is configured by associating decorative image data with mask data.

  At this time, the “decorative image data” is data of an image displayed as a ground pattern, and the “mask data” is a data portion (a hatched portion) of the ground pattern and a portion (a non-hatched portion) where an image of a face region appears. 2) shows the size and the positional relationship of the region.

  In FIG. 1, the decoration arrangement unit 8 determines the arrangement information of the decoration so as to synchronize with the change in the body part region detected by the detection unit 6.

  If the maximum matching is obtained with the template shown in FIG. 7, the detection unit 6 detects the rotation amount of the body part area in the plane of the input image, and the decoration arrangement unit 8 detects the rotation. The arrangement information of the decoration is determined so as to synchronize with the change in the rotation amount.

  If the maximum matching is obtained with the template shown in FIG. 8, the detection unit 6 detects the rotation amount of the body part region with respect to the plane of the input image, and the decoration placement unit 8 determines the rotation amount detected by the detection unit 6. The arrangement information of the decoration is determined so as to be synchronized with the change of the decoration.

  The detailed processing of the decoration arrangement unit 8 will be described later with reference to several examples.

  In FIG. 1, the synthesizing unit 10 outputs an image in which decorations are arranged in an input image based on the arrangement information determined by the decoration arrangement unit 8. This image is temporarily stored in the output image storage unit 11 and then output from the image output unit 3.

In addition, the synthesizing unit 10 performs display on the display device.
(Data Type 1) Image data in which decoration is directly drawn on an input image is generated, and the synthesizing unit 10 generates image data for transfer to a recording medium or another communication device
(Data Form 2) Image data in which an input image is associated with parameters for drawing a decoration may be generated and transferred. According to the data form 2, the input image can be easily restored, which is advantageous. The data format 2 will be supplemented later.

  Further, in FIG. 1, an operation unit 12 is for a user to input operation information to the control unit 2.

  Next, a specific configuration example of the image processing apparatus shown in FIG. 1 and its peripheral parts will be described with reference to FIG. FIG. 2 is a block diagram of the image processing apparatus.

  In the example of FIG. 2, the image processing device of FIG. 1 is incorporated in a camera-equipped mobile phone. As shown in FIG. 2, the camera-equipped mobile phone has the following elements.

  The CPU 21 controls each element of FIG. 2 via the bus 20 and executes a control program recorded in the ROM 23 and according to the flowchart of FIG.

  A temporary storage area required for processing by the CPU 21 is secured in the RAM 22. The flash memory 24 is a device corresponding to a recording medium.

  Further, the communication processing unit 26 transmits and receives data to and from an external communication device via the antenna 25.

  The image processing unit 27 includes an encoder / decoder or the like related to an encoding method such as MPEG, processes an image captured by the camera 28, and displays an LCD (an example of a display device) 29 based on image data instructed by the CPU 21. Control the display state of.

  The audio processing unit 30 controls input from the microphone 31 and audio output via the speaker 32.

  Further, an interface 33 is connected to the bus 20, and a user can input operation information by a key set 34 via the interface 33, and further, another device can be connected via a port 35. It is configured to be connectable.

  Here, the image input unit 1 in FIG. 1 is realized by the CPU 21 or the image processing unit 27 processing data recorded in the flash memory 24 or data captured by the camera 28.

  The control unit 2, the detection unit 6, the decoration arrangement unit 8, and the synthesis unit 10 are realized by the CPU 21 performing calculations while exchanging data with the RAM 22, the flash memory 24, and the like.

  Further, the input image storage unit 4, the template storage unit 5, the decoration information storage unit 7, the arrangement information storage unit 9, and the output image storage unit 11 correspond to an area secured in any of the RAM 22, the ROM 23, and the flash memory 24. . 1. The key set 34 in FIG. 2 corresponds to the operation unit 12 in FIG.

  The CPU 21 recognizes an operation performed by the user on the key set 34, obtains an image from the camera 28, compresses and saves the camera image in the flash memory 24, reads and expands the saved image, synthesizes the image, Playback, display on the LCD 29, and the like are performed, but some of these processes may be performed by the image processing unit 27.

  FIG. 3 shows the appearance of the image processing apparatus of the present embodiment. As described above, the image processing apparatus according to the present embodiment is incorporated in a camera-equipped mobile phone. The camera-equipped mobile phone of FIG. 3 is of a two-fold type in which an upper housing 42 and a lower housing 41 are foldably coupled by a coupling portion 43. Of course, the external shape of the image processing device is arbitrary, and the present invention can be applied to devices other than a mobile phone such as a PDA.

  Next, with reference to FIG. 4, the data form 2 will be supplementally explained. Here, it is assumed that there is an input image (file name “face.rm”) shown in FIG. 4B and a decoration (file name “star.swf”) shown in FIG. 4C.

  At this time, it is preferable to use a metafile of the SMIL (Synchronized Multimedia Integration Language) format shown in FIG. In FIG. 4A, a line of <video src = "face.rm"> and a line of <animation src = "star.swf"> sandwiched between <par> and </ par> Executed at the same time. That is, at the time of display, the input image of FIG. 4B and the decoration of FIG. 4C are combined.

  In the case of performing the association as shown in FIG. 4, it is not necessary to create an image itself in which the input image of FIG. 4B and the decoration of FIG. 4C are combined.

  Instead of the above SMIL format metafile, an XML (Extensible Markup Language) format file, MP4, or the like may be used. That is, when the data form 2 is used, the input image may be associated with the decoration in some form.

  Next, an outline of a processing flow by the image processing apparatus of the present embodiment will be described with reference to FIGS. Here, it is assumed that the decoration information (No. 1) shown in FIG.

  First, in step 1, the control unit 2 checks whether or not the processing has been completed for all frames constituting the moving image. If completed, the process ends. If not completed, the process proceeds to step 2.

  In step 2, the image input unit 1 obtains an input image as shown in FIG. 12A, and the control unit 2 stores the input image in the input image storage unit 4.

  Next, in step 3, the control unit 2 commands the detection unit 6 to detect a face area. Thus, the detection unit 6 detects the face area using the template (FIG. 12B) in the template storage unit 5.

  Next, in step 4, the control unit 2 instructs the decoration arrangement unit 8 to determine arrangement information. Thereby, the decoration arrangement unit 8 determines the arrangement information using the decoration information in the decoration information storage unit 7 and stores the arrangement information in the arrangement information storage unit 9. As a result, a decorative image on the decorative layer as shown in FIG.

  Next, in step 5, the control unit 2 instructs the synthesizing unit 10 to synthesize the input image and the decoration. Accordingly, the combining unit 10 combines the decoration with the input image according to the decoration information stored in the decoration information storage unit 7 and stores the result in the output image storage unit 11, as shown in FIG.

  Next, in step 6, the control unit 2 outputs the composite image in the output image storage unit 11 to the outside via the image output unit 3.

  The control unit 2 repeats the processing from step 1 onward until the processing for all frames is completed.

  Next, a configuration example of the user interface will be described with reference to FIG. Here, it is assumed that the application selected by the user is image editing software, and that the application is in an image synthesis mode for synthesizing a person image captured by the user with a decoration.

  First, as shown in FIG. 11A, the LCD 29 displays a person image to be edited. Then, an edit button 40 and a return button 41 are displayed along with the person image.

  Here, when the user selects the edit button 40 using the key set 34, a list of insertable decorations is displayed as shown in FIG. 11B.

  Then, when the user selects the item “tears” and presses the enter button 43, the face area of the person image is detected, and decoration arrangement information matching the face area is obtained. The decoration “tears” is displayed in accordance with the decoration arrangement information. Is synthesized with the input image, and the result is displayed on the LCD 29 as shown in FIG. Note that the tear decorations 50 and 51 are added to the input image. Here, the tear is inserted as a moving animation in addition to the still image.

  Now, in the prior art, the tear does not move while being arranged. Even if an animation is inserted, the movement of this animation moves completely independently of the movement of the person image.

  For example, tears are not automatically inserted under the eyes, and tears are not automatically adjusted to stop around the cheeks of the forehead, and tears flow directly below the face. Would. In addition, when the face is tilted or moved, the tear movement cannot be corrected according to the face.

  On the other hand, in the present embodiment, before the tear is inserted, the position and size of the body part region (for example, eyes in the face region in the present embodiment) are detected, so that the body part region is automatically synchronized with the movement of the body part region. The tears are inserted under the eyes. Moreover, the decoration can follow the movement of the face.

  Hereinafter, the decoration arrangement information will be further described with reference to application examples of the present embodiment. FIG. 13 is a diagram illustrating the positional relationship between the input image and the coordinate system of the decoration.

  Here, as a supplement prior to the description of each application example, a procedure for creating a general animation will be briefly described. Basically, animated animation is created by moving an ornament layer associated with a transform property.

  In other words, to create an animation, set the attributes of the transform (position, rotation, etc.) for each key frame (existing on the time axis), and change the values of the attributes over time. Let it.

  Here, a key frame defines a value such as a position of a layer at a reference time. If there are two keyframes having different values (of course, their positions on the time axis are different), for the frame located between these keyframes on the time axis, an intermediate value is calculated to calculate each attribute. The value can be determined. In other words, an animation can be created by repeating the operations of (1) creating a key frame and (2) setting a movement trajectory by interpolation.

  Next, a procedure for creating an animation in the manner described above will be described using the animation shown in FIG. 14 as an example. Here, as shown in FIGS. 14A to 14D, it is assumed that an animation in which an airplane flies from the left front to the upper right is changed by changing the position, rotation, and scale value of the layer.

  First, an anchor point of this coordinate system is set in order to define a coordinate system for arranging the decoration layer. In this example, as shown in FIG. 13, the anchor point is set to (X0, Y0) which is the center of the original image, and the direction of the coordinate axes is assumed to be the same as the original image coordinate system.

  This anchor point is fixed until the end of the animation. As a result, the position and rotation are all values based on the coordinate system of the anchor point.

  Next, on the time axis, the position of the key frame and the movement trajectory connecting the key frame are set. For example, as shown in FIG. 15, it is assumed that the airplane has moved to the point A in the first frame, to the point B in 2 seconds, to the point C in 4 seconds, and that the plane is connected by a curve. In addition, the rotation and the enlargement / reduction ratio are similarly set for a frame as a point. Then, for a frame located between these key frames on the time axis, a movement locus is set by interpolating adjacent key frames. As a result, it is possible to create an animation in which the rocket flies to the upper right back as shown in FIG.

(Application Example 1)
In this example, after the position and range of the face partial region are set by template matching, the origin (anchor point) of the coordinate system expressing the decoration is set at a position corresponding to the position.

  FIG. 16 (a) shows the result of setting the decoration image data, and FIG. 16 (c) shows the result of setting the anchor point which is the origin of the decoration coordinate system that locally draws an elliptical trajectory on the head of the face area. Is shown.

  Next, when the face area of the person is detected, the anchor point is moved to the head of the face area in response to the detection, as shown in FIG. When the size of the face area changes, the scale of the coordinate system is changed in proportion to the size of the face area.

  When the direction changes, the direction of the axis of the coordinate system is changed so as to be in the same direction as the initially set direction.

  As a result, even when a person moves relatively to the photographing apparatus, the decoration that always draws an elliptical orbit on the head follows the movement of the person.

  In addition, in a plurality of face images taken in different scenes, the head can always be moved according to each face position, range, inclination, orientation in a trajectory proportional to the size of the head, Playability will be further enhanced.

(Application 2)
In this example, the decoration information (part 2) is used to change the opacity of the composite frame decoration following the movement of the face area.

  In this example, the decoration information (part 2) shown in FIG. 10B is used. Here, the processing from step 1 to step 3 in FIG. 5 is the same as the case where the decorative information (No. 1) shown in FIG. 10A is used.

  However, in Steps 4 to 6 in FIG. 5, the still image shown in FIG. 12F is arranged so as to be seen in the hatched portion of the mask in FIG. The image of the portion detected as is pasted with the size adjusted to the non-hatched portion, and this is output.

  That is, a composite mask with a still image serving as a background is used as decorative arrangement information, and the center of the composite mask is arranged so as to be the center of the face in accordance with the result of face detection. Controls the setting of opacity when combining with the input image.

  As a result, when combining with another image, a still image can be prevented from covering the face area.

(Application 3)
In this example, as shown in FIG. 17, depth information (Z-axis) for the viewpoint is added to both the input image and the decoration.

  When the decoration is cut at a virtual plane 60 corresponding to the projection plane of the input image, the decoration is divided into a foreground area 61 on the foreground side of the virtual plane 60 and a background area 62 on the background side. .

  Furthermore, the opacity of the foreground area 61 and the background area 62 are changed to perform synthesis. As a result, as shown in FIG. 18, it is possible to create an animation in which stars are rotating around the face.

  For this purpose, an anchor point may be set intentionally so that the trajectory passes through the face. If the background area 62 passing through the back of the face is made transparent, a three-dimensional moving decoration can be added to the input image.

(Application Example 4)
In this example, as shown in FIG. 19, the attribute of the decoration in the next frame is estimated from a series of positions and directions for the past several frames, the current position and direction, the past several frames, and the like. For example, assuming that the decoration moves at a constant speed, the attributes of the decoration can be estimated by a simple formula.

  Further, when the trajectory overlaps with the body part region, the trajectory may be corrected. When displaying a plurality of decorations, it is preferable that the decorations move independently of each other.

  In this example, as shown in FIG. 20, when the decoration collides with the body part region or the outline of the input image, the moving direction of the decoration is changed to a direction away from the body part region. In this way, the decoration can be given a movement that reflects when the decoration collides with the body part region, and the playability can be improved.

  The image processing apparatus according to the present invention can be suitably used, for example, mounted on a mobile phone with a camera.

Functional block diagram of an image processing device according to an embodiment of the present invention 1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. FIG. 1 is an external view of an image processing apparatus according to an embodiment of the present invention. (A) Illustrative diagram of a metafile according to an embodiment of the present invention (b) Illustrative diagram of an input image file according to an embodiment of the present invention (c) Illustrative diagram of a decorative file according to an embodiment of the present invention Flowchart of image processing apparatus according to one embodiment of the present invention (A) Illustrative diagram of a template according to an embodiment of the present invention (b) Illustrative diagram of a template according to an embodiment of the present invention (c) Illustrative diagram of a template according to an embodiment of the present invention (A) Illustrative diagram of a template according to an embodiment of the present invention (b) Illustrative diagram of a template according to an embodiment of the present invention (c) Illustrative diagram of a template according to an embodiment of the present invention (A) Illustrative diagram of a template according to an embodiment of the present invention (b) Illustrative diagram of a template according to an embodiment of the present invention (c) Illustrative diagram of a template according to an embodiment of the present invention Explanatory drawing of pattern matching in one embodiment of the present invention (A) Configuration diagram of decoration information in one embodiment of the present invention (b) Configuration diagram of decoration information in one embodiment of the present invention (A) Example display of a user interface according to one embodiment of the present invention (b) Example display of a user interface according to one embodiment of the present invention (c) Example display of a user interface according to one embodiment of the present invention Figure (A) Illustration of the processing procedure of the image processing apparatus according to one embodiment of the present invention (b) Illustration of the processing procedure of the image processing apparatus according to one embodiment of the present invention (c) Image according to one embodiment of the present invention (D) Process process explanatory diagram of an image processing device according to an embodiment of the present invention (e) Process process explanatory diagram of an image processing device according to an embodiment of the present invention (f) The present invention (G) Processing process explanatory diagram of the image processing device in one embodiment of the present invention Explanatory drawing of a coordinate system in one embodiment of the present invention. (A) Illustration of animation in one embodiment of the present invention (b) Illustration of animation in one embodiment of the present invention (c) Illustration of animation in one embodiment of the present invention (d) The present invention Explanatory diagram of an animation in one embodiment Explanatory drawing of animation in one embodiment of the present invention (A) Illustrative diagram of decoration in one embodiment of the present invention (b) Illustrative diagram of anchor point in one embodiment of the present invention (c) Illustrative diagram of composite image in one embodiment of the present invention Illustrative drawing of a decoration according to an embodiment of the present invention FIG. 2 is a view showing an example of a composite image according to an embodiment of the present invention. Illustrative drawing of a decoration according to an embodiment of the present invention FIG. 2 is a view showing an example of a composite image according to an embodiment of the present invention.

Explanation of reference numerals

Reference Signs List 1 image input unit 2 control unit 3 image output unit 4 input image storage unit 5 template storage unit 6 detection unit 7 decoration information storage unit 8 decoration placement unit 9 placement information storage unit 10 synthesis unit 11 output image storage unit 12 operation unit

Claims (18)

Detecting the position of the body part region in the input image;
Determining the position of the coordinate system of the decoration according to the position of the detected body part region;
Outputting an image in which decorations are arranged on the input image based on the determined position. The image processing method according to claim 1, wherein in the outputting, image data in which the decoration is drawn on the input image is output. 2. The image processing method according to claim 1, wherein the outputting includes outputting image data in which the input image and the decoration are associated with each other. 4. The image processing method according to claim 3, wherein the input image and the decoration are associated by a metafile. The image processing method according to claim 4, wherein the metafile is in a SMIL format. Detecting the size of the body part region in the input image;
Determining the scale of the coordinate system of the decoration according to the size of the detected body part region;
Outputting an image in which decorations are arranged on the input image based on the determined scale. Detecting the amount of rotation of the body part region in the plane of the input image;
Determining the amount of rotation of the decoration coordinate system according to the detected amount of rotation;
Outputting an image in which decorations are arranged on the input image based on the determined amount of rotation. Detecting the amount of rotation of the body part region with respect to the plane of the input image;
Determining the amount of rotation of the decoration coordinate system according to the detected amount of rotation;
Outputting an image in which decorations are arranged on the input image based on the determined amount of rotation. Detecting a body part region in the input image;
Determining the opacity of the decoration to be different between the region on the foreground side of the detected body part region and the region on the background side;
Outputting an image in which decorations are arranged on the input image based on the determined opacity. The image processing method according to claim 1, wherein when the decoration collides with the body part area, the moving direction of the decoration is changed to a direction away from the body part area. The image processing method according to claim 1, wherein the body part region is a face region of a human subject. An input image storage unit for storing an input image,
A template storage unit that stores a template of a body part region;
Using a template of the template storage unit, from the input image stored in the input image storage unit, a detection unit that detects a body part region,
A decoration information storage unit for storing decoration information;
An image processing apparatus, comprising: a decoration placement unit that determines placement information of a decoration so as to synchronize with a change in a body part region detected by the detection unit. The image processing apparatus according to claim 12, further comprising: a combining unit configured to output an image in which the decoration is arranged on the input image based on the arrangement information determined by the decoration arrangement unit. The detection unit detects the amount of rotation of the body part region in the plane of the input image, and the decoration arrangement unit adjusts the arrangement information of the decoration so as to synchronize with the change in the amount of rotation detected by the detection unit. 14. The image processing apparatus according to claim 12, wherein the image processing apparatus determines the image processing. The detection unit detects the amount of rotation of the body part region with respect to the plane of the input image, and the decoration arrangement unit determines the arrangement information of the decoration so as to synchronize with the change in the amount of rotation detected by the detection unit. The image processing device according to claim 12. The image processing apparatus according to claim 12, wherein the decoration information is configured by associating decoration image data with a transform property. 17. The image processing apparatus according to claim 12, wherein the decoration information is configured such that decoration image data and mask data are associated with each other. An input image storage unit for storing an input image,
A template storage unit that stores a template of a body part region;
Using a template of the template storage unit, from the input image stored in the input image storage unit, a detection unit that detects a body part region,
A decoration information storage unit for storing decoration information;
A mobile phone comprising: a decoration placement unit that determines the placement information of the decoration so as to synchronize with the change in the body part region detected by the detection unit.

Images