JP2006229467A - Photo movie creating apparatus, photo movie creating program, and object recognition method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology capable of easily creating a photo movie to which a proper special effect is applied in response to a notice object of a still picture. <P>SOLUTION: In the case of creating a photo movie, a notice object is determined from a still picture being a source. When the notice object is determined, a face image of a person is detected from the still picture, whether the person is located in the middle of the still picture or located at the surrounding of the still picture is determined. The face image is detected by extracting both eyes of the person. When the person is located in the middle of the still picture, the notice object is the person. When the person is located at the surrounding part of the still picture, the size of the face image is discriminated on the basis of an interval between both the eyes of the person. When the face image is greater, the notice objects are two comprising the person and the background, when the face image is smaller, the notice object is the background. Determining contents of the special effect on the basis of the position and the size of the notice object can create the highly attractive photo movie. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a photo movie creation apparatus and a photo movie creation program for creating a photo movie to be viewed as a moving image from a still image of a material, and a subject recognition method for accurately determining a subject of interest according to the content of the still image.

  2. Description of the Related Art Image display devices and photo movie creation software are known that allow a still image recorded by a digital camera or the like to be viewed as a moving photo movie (see Patent Documents 1 and 2, Non-Patent Documents 1 to 3). . A photo movie is a material that is converted from a still image into a moving image through a special processing and editing process that adds dynamic visual effects. It produces an effect that focuses on a specific subject compared to a slide show that displays images sequentially. In addition, it is possible to produce an effect that gives a story to one still image.

  The video of the photo movie can be played on a dedicated playback device or a computer incorporating the above-described software. If the video data of the photo movie is converted into a general-purpose digital video format and recorded on an optical disk such as a DVD, the photo movie can be viewed using a home DVD player or the like.

  Special effects for photo movies that add motion to still images include electronic zooming effects that zoom in / out by focusing on a part of the still image, and electronic scrolling with a part of the still image close-up. Panning effect, 1 frame still image reduced on the screen, frame movement to move in a straight line or curved line, rotation effect to rotate the image around a specific point, distortion effect to give distortion (skew), There are combined effects of combining these. In addition, there are multi-screen effects that reduce and display multiple still images at the same time, animation images, decorative images, visual effects that combine subtitles, and slide shows that do not give special effects, etc. There are many different forms of production.

  In a photo movie, one scene is generated from a specific still image, and usually a plurality of scenes are collected. In order to create a high-quality photo movie that looks good, it is necessary to optimally construct individual scenes from factors such as the quality of still images as materials, the types of special effects, and the playback time of scenes. In addition, images at the start and end of the scene are highly important in determining the appearance of the scene, editing parameters are set for special effects, and the user can change the presentation mode such as image transition in the scene. (Non-Patent Document 1 to Non-Patent Document 3, etc.).

For example, in the special effects of electronic zooming and electronic panning, a trimming frame that defines a range for cutting out a part of a still image is set as an editing parameter, and an image cut out by this trimming frame is displayed on the entire screen. The user designates the start / end point of electronic zooming or electronic panning, or a relay point by designating one or more trimming frames on the still image.
Japanese Patent Laid-Open No. 10-200843 JP 2002-24846 A Digital Stage Co., Ltd. LIFE * with-Photo-Cinema [Search on November 1, 2004] Internet <URL: http://www.digitalstage.net/jp/product/life/index.html> Ai Soft Co., Ltd. Digital Camera de Movie Theater [Search on November 1, 2004] Internet <URL: http://ai2you.com/imaging/products/dcmtheater/dcmt.asp> Canopus Co., Ltd. Imaginate2.0 [Searched on November 1, 2004] Internet <URL: http://www.canopus.co.jp/catalog/imaginate/imaginate20#index.htm>

  However, conventionally, in order to set a trimming frame, the user himself / herself has to determine the subject of interest by determining the content of the still image, and when the number of still images of material is large, There is a problem that it takes time to create a photo movie. In the prior art described in Non-Patent Documents 1 to 3, the trimming frame can be automatically set. However, since the trimming frame is determined at random without considering the subject of interest, A photo movie that emphasizes the background is created from the still image, and a photo movie that emphasizes the person is created from the still image with the background as the theme, and a photo movie that reflects the user's intention cannot be created There is a problem.

  The present invention has been made in consideration of the above-described problems. A photomovie creation apparatus and a photomovie creation program that can accurately determine a target object of a still image and automatically set a trimming frame, and a target object. An object of the present invention is to provide a subject recognition method that determines accurately.

  In order to achieve the above object, the present invention changes a position or size of a trimming frame that defines a range for extracting a part of image data from a still image while changing a plurality of images as a moving image frame. In a photo movie creation device for creating a photo movie to which special effects including electronic zooming and electronic panning are applied by performing processing editing processing that sequentially cuts out, the still image is detected by detecting a human face image from the still image If the face image is in the center, the person is the subject of interest. If the face image is in the periphery, the subject of interest is determined with the person's background as the subject of interest. And the moving path, moving speed, and size of the trimming frame on the moving path based on the determined subject of interest. Characterized by providing the parameter determining means for determining at least one of the free parameters.

  The target subject determination means calculates the size of the face image when the face image is in the periphery of the still image, obtains the ratio of the size of the face image to the size of the still image, When the ratio of the face image is smaller than the reference value, the background of the person is the subject of attention, and when the ratio of the face image is larger than the reference value, both the person and the background of the person are the subject of attention. .

  The target subject determination means calculates the size of the face image when the face image is in the periphery of the still image, obtains the ratio of the size of the face image to the size of the still image, By comparing the shooting distance of the person obtained from the ratio of the size of the face image and shooting distance information incorporated in advance in the image data of the still image, if the two are different, the background of the person is set as the subject of interest. When both are the same, both the person and the background of the person are the subject of attention.

  The face image is detected by extracting both eyes of a person.

  The size of the face image is calculated from the distance between the eyes of the person.

  In addition, the photo movie creation program of the present invention sequentially converts a plurality of images as moving image frames while changing at least one of the position and size of a trimming frame that defines a range of a part of the image data to be cut out. By creating a photo movie with special effects including electronic zooming and electronic panning by performing processing editing processing, the human face image is detected from the still image and the still image is detected. If the face image is in the center, the person is the subject of interest. If the face image is in the periphery, the subject of interest is determined with the person's background as the subject of interest. And a moving path of the trimming frame, a moving speed, and a size of the trimming frame on the moving path based on the determined subject of interest. And characterized by causing a computer to function as the parameter determining means for determining at least one of the parameters.

  Further, the subject recognition method of the present invention detects a human face image from image data of a still image, determines whether the position of the face image is in a central portion or a peripheral portion of the still image, and If the image is in the center, the person is the subject of interest, and if it is in the periphery, the background of the person is the subject of interest.

  According to the photo movie creation device and the photo movie creation program of the present invention, if the face image of the person detected from the still image is located in the center of the still image, the person is set as the subject of interest, and the face image of the person is stationary. If it is located at the periphery of the image, the background of the person is set as the subject of interest, and it is accurately determined whether the subject of interest is the person or the background. Automatic settings can be made, making it easy to create high-quality photo movies that accurately incorporate the theme of still images.

  In addition, when the face image is in the periphery of a still image, it is determined whether only the background is the target subject or both the background and the person are the target subject according to the size of the face image. The subject of interest in the still image of the scene can be accurately determined.

  Also, when the face image is in the periphery of the still image, by comparing the shooting distance of the person obtained from the size of the face image with shooting distance information previously written in the image data at the time of shooting, When they are different, the background of the person is the subject of attention, and when they are the same, both the person and the background of the person are the subject of interest. Therefore, the subject of interest of the still image can be determined more accurately according to various shooting scenes.

  In addition, by detecting the face image by extracting both eyes of a person, the face image is detected using ambiguous information such as the face outline and head shape that changes depending on how the person in the still image is captured. The human face can be detected with higher accuracy.

  In addition, since the size of the person's binocular distance is used as an index as the size of the person's face, the face size can be obtained with higher accuracy than when the size of the person's face is measured by the area. it can.

  In FIG. 1, a photo movie creation apparatus 10 is installed at a store of a DPE store that provides a photo print service or in a store, and is operated as a user by an ordering customer who requests creation of a photo movie. The photo movie creation device 10 includes a storage device 11 that stores a photo movie creation program and a CPU 12 that executes the photo movie creation program. The storage device 11 is a hard disk type magnetic storage device, for example, and stores image data of a still image as a material for a photo movie, intermediate data generated by arithmetic processing of the CPU 12, and the like.

  The monitor 13 displays on the screen an editing menu for creating a photo movie, a created photo movie, and the like. The input operation unit 14 is used to input editing parameters necessary for creating a photo movie. For example, when an input key displayed on the monitor 13 is touched, the input operation unit 14 is input from the coordinates of the touched position. It consists of a touch panel that detects the contents. The communication unit 15 performs data communication of image data with an external device such as the digital camera 16. The media reading unit 17 is loaded with a memory card 18 brought in by a user and reads image data stored in the memory card 18. The media recording unit 19 records the created photo movie on an optical disc 20 such as a CD or a DVD.

  The CPU 12 executes a photo movie creation program stored in the storage device 11, thereby controlling the operation of each unit based on input processing performed by the user, and a photo movie creation unit that creates a photo movie. 26 functions. The photo movie creation unit 26 calculates the size of the detected face image and the ratio of the size of the face image to the entire still image. The face image detection unit 30 detects a human face image from the still image of the material. A face ratio calculation unit 31 to calculate, a target subject determination unit 32 that determines a target subject based on the detected face image, a trimming frame determination unit 33 that determines the position of a trimming frame based on the target subject, and a photo movie And a photo movie output unit 34 for outputting the moving image data.

  The face image detection unit 30 detects a person's face image by extracting both eyes of the person from the still image. As shown in FIG. 2, when extracting both eyes of a person, the division indicated by the hatched portion including many skin-colored pixels estimated to be skin from each of the divided regions S1 obtained by dividing the still image into a plurality of regions. Area S1 is selected. Furthermore, a process of selecting an area having white pixels of the eye and a color pixel estimated to be a pupil from an area containing many skin color pixels is performed. From the selected area, the position coordinates of both eyes of the person on the still image are obtained, and the position of the face image is obtained with the middle point of the position coordinates of both eyes as a representative point.

  The face ratio calculation unit 31 obtains the binocular distance L1 shown in FIG. 3 from the extracted position coordinates of both eyes of the person, and calculates the ratio of the face image to the entire still image from the binocular distance L1. The target subject determination unit 32 determines whether the target subject of the still image is a person or the background of the person from the position coordinates of the face image and the ratio of the face image to the entire still image. As shown in FIGS. 3 and 4, when determining the subject of interest, the position coordinates of the person's face belong to the square central area A1 defined in the central part of the still image or the peripheral area A2 outside the square. Is determined.

  When the face of a person is located inside the central area A1, it is determined that the subject of interest is a person. Further, when the face of the person is located inside the peripheral area A2, the ratio of the face of the person calculated by the face ratio calculation unit 31 is referred to, and when the face ratio is small, the background of the person is noticed. The subject is determined. When the face ratio is large, it is determined that both the person and the background of the person are the subject of interest. The central area A1 is not limited to a square, but may be a rectangle, an ellipse, or other shapes.

  In FIG. 5, the trimming frame determination unit 33 performs trimming as a parameter that defines a rectangular trimming frame F1 that defines a range in which a part of a still image is cut out from the position and size of the target subject determined by the target subject determination unit 32. The center position and diagonal length of the frame F1 are determined. Further, the moving path of the trimming frame F1 is determined from the center position, diagonal length, and vertex position of the trimming frame F1 with respect to the time axis of the photo movie, and the moving speed is determined from the reproduction time of the photo movie and the length of the moving path. . The trimming frame F1 is fixed to have the same aspect ratio as the screen aspect ratio of a general home television, and its size is uniquely determined by the center position and the diagonal length.

  The movement path of the trimming frame F1 can be selected from three cases: placing the subject of interest at the start point of the movement path, placing it at the end point of the movement path, and placing it at the relay point (intermediate point) of the movement path. Is done. The point at which the subject of interest is placed depends on the type of special effect and whether the subject of interest is either a person or a background, or both.

  For example, as shown in FIG. 6A, one target subject is set as the starting point of the movement path of the trimming frame F1, the center position of the trimming frame F1 is substantially fixed, and the vertex position is set so that the diagonal length is increased. In the case of a change, as shown in FIG. 6B, a photo movie is provided with a zoom-out effect that changes from a close-up image of a person to a wide-angle visual image including the background. Conversely, when the subject of interest is set as the end point of a photo movie, the photo movie is a scene with a zoom-in effect that changes from a wide-angle visual image including the background to a close-up image of a person.

  Further, as shown in FIG. 7A, one subject of interest is set as the starting point of the moving path of the trimming frame F1, the center position and diagonal length of the trimming frame F1 are fixed, and the vertex is moved in a circle. When the trimming frame F1 is rotated, as shown in FIG. 7B, a photo movie with a rotation effect of rotating an image of a close-up of a person within the screen is obtained. At this time, if the vertex position is further changed so as to increase the diagonal length, a photo movie of a scene in which the zoom-out effect and the rotation effect that rotate while moving away from the person are combined.

  As shown in FIGS. 8 and 9A, when there are two subjects of interest, a person and a person's background, for example, the person is set as the start point of the moving path of the trimming frame F1, and the background is set as a relay point. When the end point is set to the person and the background, and the diagonal length of the trimming frame F1 is changed according to the size of the subject of interest, as shown in FIG. A photo that combines electronic panning and electronic zooming after moving to a close-up image of the mountain and moving from a close-up image of the background mountain to a wide-angle visual image that includes two subjects of interest, the person and the background. It becomes a movie. Here, when the diagonal length of the trimming frame F1 is not changed, that is, when the size of the trimming frame F1 is made constant, the size is determined with priority on the size of the person.

  When the special effect is designated in advance as an electronic zooming or rotation effect by the user, if the subject of interest is one of a person and a background, the subject of interest is set as the start point or end point of the movement path of the trimming frame F1. If there are two subjects of interest, the person and the background, one of the subjects of interest is set as the start point or end point of the movement path, and the other subject of interest is considered not to exist.

  When the special effect is designated in advance for electronic panning, if the target subject is either a person or a background, for example, the target subject is set as a relay point, and a photo movie whose trimming frame F1 passes through the target subject is displayed. Created. If the special effect is specified in advance for electronic panning and there are two subjects of interest, the person and the background, for example, the person and the background are regarded as one subject of interest, and the subject and the background are individually noticed. However, the three target subjects are allocated to the panning start point / end point / relay point, respectively. If the special effect is not designated by the user, the special effect is selected at random, and the point at which the subject of interest is placed is determined in the same manner as described above.

  Also, the playback time of the photo movie is the total length of the playback time of each scene. Each scene may be set to a predetermined initial value (for example, 5 seconds) or set to a length specified by the user (for example, 15 seconds). The trimming frame F1 is determined to have a long moving path when a long playback time is specified, and is determined to be short when a short playback time is specified. The moving speed of the trimming frame F1 varies greatly depending on the length of the playback time. Adjusted to not.

  The photo movie output unit 34 converts the image cut out from the still image based on the parameters of the trimming frame F1 into a moving image frame constituting the photo movie, and generates, for example, moving image data of 30 frames per second. Further, the generated moving image data is converted into a digital moving image format of, for example, MPEG2-DVD-Video format, and the converted moving image data of the photo movie is stored in the storage device 11. The moving image data of the photo movie stored in the storage device 11 is recorded on the optical disc 20 by the media recording unit 19.

  Next, the operation of the photo movie creation apparatus 10 will be described with reference to FIG. When a digital camera 16 or a memory card 18 storing still images as image data is brought in by an ordering customer, the digital camera 16 is connected to the communication unit 15 and the memory card 18 is set in the media reading unit 17. The control unit 25 reads out still image data from the digital camera 16 or the memory card 18 and stores it in the storage device 11.

  The face image detection unit 30 reads a still image stored in the storage device 11 and performs processing for detecting a human face image from the still image. The face image detection unit 30 divides the still image into a plurality of divided regions S1, and selects a divided region S1 including many skin-colored pixels from the divided images. In addition, a divided area S1 in which both eyes of a person exist is selected from the divided area S1 including many skin-colored pixels. Thereby, the position coordinates of both eyes are obtained, and the position of the face image is obtained as the midpoint position of both eyes of the person.

  The face ratio calculation unit 31 calculates the binocular distance L1 from the coordinates of both eyes of the person detected by the face image detection unit 30, and calculates the ratio of the size of the face image to the entire still image. The subject-of-interest determination unit 32 determines whether the position of the person's face image is located in the central area A1 or the peripheral area A2 of the still image. When the face image is located in the central area A1, the subject of interest is determined to be a person. When the face image is located in the peripheral area A2, the binocular distance L1 is compared with a preset reference value. If the face image is below the reference value, it is determined that the subject of interest is the background. If the binocular distance L1 exceeds the reference value, it is determined that the subject of interest is both a person and a background.

  The trimming frame determination unit 33 determines the center position and the diagonal length of the trimming frame F1 with respect to the time axis of the photo movie based on the determined position and number of the subject of interest and the size of the subject, and the movement path thereof. And determine the moving speed. If the playback time of the photo movie to be created is designated, the length of the movement path is adjusted based on the length of the playback time. When determining the movement route, it is considered whether or not the type of special effect is specified by the user, and which target subject is placed at the start point / end point / relay point of the movement route of the trimming frame F1. Is decided.

  When the same processing is performed for a plurality of still images and the trimming frame determination unit 33 determines the still image clipping range by the trimming frame F1 for all the still images, the movement path, diagonal length, and vertex coordinates of the trimming frame F1 are determined. And the like are sent to the photomovie output unit 34. The photo movie output unit 34 sequentially extracts still images from information on the position and size of the trimming frame F1 with respect to the time axis, and converts the extracted images into moving image frames having the same number of pixels. In addition, an authoring process is performed in which moving image data is generated from the converted moving image frame, and the generated moving image data is MPEG-compressed and then converted to the DVD-Video format. The moving image data subjected to these processes is stored in the storage device 11.

  The moving image data of the photo movie stored in the storage device 11 is written on the optical disc 20 in the media recording unit 19. The optical disk 20 in which the moving image data of the photo movie is written is ejected from the photo movie creating apparatus 10 in exchange for a fee, and delivered to the ordering customer. As described above, the ordering and creation of the photo movie and the delivery of the product are completed.

  It should be noted that the present invention is not limited to the above-described embodiment, and appropriate modifications can be made. For example, as a procedure for determining the subject of interest, a human face image is detected from a still image, and the face image is located in the central portion or the peripheral portion of the still image or in the peripheral portion The target subject is determined based on the size of the face image (the distance between the binocular distances of a person). For example, when the face image is located near the boundary between the central portion and the peripheral portion of the image, You may use the tag information of Exif format which is the prevailing image data standard. As shown in FIG. 11, if image data recorded by a digital still camera or the like incorporates shooting information such as shutter speed, aperture, shooting distance, date and time information, type of lens used, etc., these information are stored. Compare the shooting distance recorded as tag information in the image data and the distance between the person and the camera determined from the person's binocular distance. If they match, the person is the subject of interest. The background can be the subject of interest.

  The photo movie creation device 10 of the above embodiment is configured as a dedicated device that collectively creates an optical disc on which moving image data of a photo movie is recorded from reading of still image data in order to create a photo movie. As another form, an order acceptance machine for reading still image data and specifying special effects, etc., and receiving still image data and various types of information specified by the user from the order acceptance machine In addition, a photo movie output machine that creates a photo movie based on this may be used. In this case, the photo movie creation program of the present invention may be incorporated into a personal computer so that an order can be placed from home via a communication means such as the Internet. Further, by incorporating the photo movie creation program of the present invention into the digital still camera, the captured image may be reproduced and output as a photo movie on the spot.

  The types of special effects are not limited to electronic zooming, electronic panning, and rotation effects. For example, an image obtained by cutting out a part of a still image by trimming F1 is displayed smaller than the display screen and moved within the screen. It may be a frame movement effect. You can also change the position of the image to be cut out with a trimming frame such as a circle or rhombus, and display a black image on the outside of the cutout area to display a special effect that creates a visual effect of irradiating a still image with a spotlight. The special effect such as electronic zooming and electronic panning is not limited to the special effect in which the image cut out by the trimming frame is displayed on the entire screen.

It is a block diagram which shows schematic structure of the photo movie production apparatus. It is a conceptual diagram for demonstrating the principle which detects a person's face image. It is a conceptual diagram for demonstrating the principle which determines an attention object. It is a flowchart which shows the flow of the procedure which determines an attention object. It is a conceptual diagram which shows the trimming frame determined when an attention object is a person. It is explanatory drawing which shows the movement path | route of a trimming frame, and the photo movie to which electronic zooming was provided. It is explanatory drawing which shows the moving path | route of a trimming frame, and the photo movie to which the rotation effect was provided. It is a conceptual diagram which shows the trimming frame in case the attention object is two people and a background. It is explanatory drawing which shows the moving path | route of a trimming frame, and the photo movie to which the special effect which combined electronic zooming and electronic panning was provided. It is a flowchart which shows the process sequence of the photo movie production apparatus. It is a flowchart which shows the other procedure which determines an attention object.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Photo movie production apparatus 11 Memory | storage device 11a Photo movie production program 12 CPU
DESCRIPTION OF SYMBOLS 16 Digital camera 17 Media reading part 18 Memory card 25 Control part 26 Photo movie creation part 30 Face image detection part 31 Face ratio calculation part 32 Attention subject determination part 33 Trimming frame determination part 34 Photo movie output part S1 Division | segmentation area L1 Binocular distance A1 Central area A2 Peripheral area F1 Trimming frame

Claims (7)

By changing the position or size of the trimming frame that defines the range for cutting out a part of the image data of the still image, the processing and editing process is performed to cut out a plurality of images sequentially as a moving image frame. In a photo movie creation device that creates a photo movie with special effects including zooming and electronic panning,
A face image of a person is detected from the still image to determine whether the image is in the center or the periphery of the still image. In some cases, the subject-of-interest determining means that uses the background of the person as the subject of interest;
Parameter determining means for determining at least one of parameters including a moving path, a moving speed of the trimming frame, and a size of the trimming frame on the moving path based on the determined subject of interest; Photo movie creation device.   The target subject determination means calculates the size of the face image when the face image is in the periphery of the still image, obtains the ratio of the size of the face image to the size of the still image, When the ratio of the face image is smaller than the reference value, the background of the person is the subject of attention, and when the ratio of the face image is larger than the reference value, both the person and the background of the person are the subject of attention. The photo movie creating apparatus according to claim 1.   The target subject determination means calculates the size of the face image when the face image is in the periphery of the still image, obtains the ratio of the size of the face image to the size of the still image, By comparing the shooting distance of the person obtained from the ratio of the size of the face image and shooting distance information incorporated in advance in the image data of the still image, if the two are different, the background of the person is set as the subject of interest. 2. The photo movie creating apparatus according to claim 1, wherein when both are the same, both the person and the background of the person are the subject of attention.   4. The photo movie creation device according to claim 1, wherein the face image is detected by extracting both eyes of a person.   5. The photo movie creation device according to claim 4, wherein the size of the face image is calculated from the distance between the eyes of the person. Electronic zooming is performed by changing at least one of the position or size of the trimming frame that defines the range for cutting out a part of the image data of the still image and cutting out a plurality of images sequentially as a moving image frame. , In a photo movie creation program for creating a photo movie with special effects including electronic panning,
A face image of a person is detected from the still image to determine whether the image is in the center or the periphery of the still image. In some cases, the subject-of-interest determining means that uses the background of the person as the subject of interest;
A computer is caused to function as parameter determining means for determining at least one of parameters including a moving path, a moving speed of the trimming frame, and a size of the trimming frame on the moving path based on the determined subject of interest. Photo movie creation program. A face image of a person is detected from the image data of the still image, it is determined whether the position of the face image is in the center or the periphery of the still image, and if the face image is in the center, Is a subject of interest, and if it is in the periphery, the background of the person is the subject of interest.

Images