JP4519531B2 - Image display device, image display method, and program - Google Patents

Description

  The present invention relates to still image processing and display technology, and more particularly, to region control in consideration of an object to be extracted at the time of image display, and to generate and reproduce a short video using at least one or more still images. About.

  In the Japanese camera market in 2001, the number of digital still cameras (hereinafter referred to as “digital cameras”) shipped exceeds the number of silver halide cameras shipped, and it is predicted that the gap will further widen in the future. In the recent mobile phone market, a large number of mobile phones with digital cameras have been released, and their models are gaining great popularity. Thus, the number of digital camera users is increasing.

  On the other hand, portable terminals (for example, mobile phones) having an image display function have become so popular as to be held one by one, and the image display apparatus of the portable terminal (hereinafter simply referred to as “image display apparatus”). The demand for displaying images taken with a digital camera is increasing day by day.

  In general, an image forming apparatus such as a digital camera captures a subject with 1280 × 960 pixels, an image of 1280 × 960 pixels (hereinafter, this image is also referred to as “original image”), and 160 × 120 pixels. The thumbnail image is recorded in a recording medium such as an SD memory card (registered trademark) in the EXIF format. On the other hand, the display of the image display device displays a maximum of about 240 × 320 pixels (QVGA).

  As described above, when there is a difference in resolution between the display of the image display device and the image (display image) to be displayed on the image display device, the display image is enlarged or reduced in accordance with either resolution. Need to shrink.

  Therefore, as a technique related to a conventional image forming apparatus, in a mobile phone or the like, it is determined whether or not an image to be displayed fits in the display area. If the original image does not fit in the display area size, it is stored separately. A method for displaying a thumbnail image is proposed (see, for example, Patent Document 1).

  By the way, with the spread of digital cameras and camera-equipped mobile phones, the number of users who have a large amount of digital still images (hereinafter referred to as “still images”) has increased remarkably. These still images are generally stored in a disk drive of a personal computer, and as a device for reproducing this large amount of still images, short images (short films) such as movies are generated from the still images. An apparatus for generating a short film for reproduction has also been proposed. For example, Patent Document 2 discloses a technique for reproducing a still image while applying a visual effect based on dynamic image processing content and a temporal processing procedure (scenario).

FIG. 38 is a block diagram showing a configuration of a conventional short film generating apparatus.
The conventional short film generation apparatus 900 includes an operation input unit 901, an image input unit 902, an image storage unit 903, a scenario generation unit 904, an image reproduction unit 905, and a display unit 906.

The operation input unit 901 accepts an operation input from the user such as an instruction for capturing still image data and selection of a still image.
The image input unit 902 captures still image data in JPEG compression format captured by the digital camera 910 or the like into the apparatus.

The image storage unit 903 stores the captured still image data.
The scenario generation unit 904 determines the playback order and visual effects of the still images based on the still image selected by the user, generates a scenario that is playback information, and stores it in the memory. Here, the scenario is not the moving image data itself represented by MPEG, but the time-series visual effects (effects), the parameters that define the operation of the effects, the pointer to the still image subject to the effects, and the playback Time and the like are described in a script language represented by XML.

  The image playback unit 905 reads out still image data from the image storage unit 903 according to the generated scenario, and repeats an operation of playing back at 15 frames / second while applying a specified visual effect.

The display unit 906 displays a user interface screen and a reproduction image under the control of the scenario generation unit 904 and the image reproduction unit 905.
The short film production | generation apparatus 900 comprised as mentioned above operate | moves as follows.

First, an image captured by the user with the digital camera 910 is taken into the apparatus via the image input unit 902, and transmitted and stored as it is in the image storage unit 903.
Here, when the operation input unit 901 receives an operation of pressing the short film generation button from the user, the scenario generation unit 904 generates a scenario.

FIG. 39 is a flowchart showing the processing procedure of the conventional scenario generation unit 904.
First, the scenario generation unit 904 generates a screen (image selection screen) that prompts the user to select a plurality of images from the images stored in the image storage unit 903, and outputs the screen to the display unit 906 (S900).

  Here, when the operation input unit 901 receives an operation for selecting a still image to be reproduced from images displayed on the display unit 906 from the user (Yes in S902), the scenario generation unit 904 selects the selected image. The shooting time information is checked, and the scenario outer frame is generated by arranging the reproduction information of each image in the order of the shooting time to form a linked list (S904).

  Here, the operation input unit 901 shows a scenario outer frame generated when the selection of five still images (image 31, image 32, image 33, image 34, and image 35) is received from the user in FIG. Show. It is assumed that the shooting times of the five still images are arranged in time series in the order of image 31, image 32, image 33, image 34, and image 35.

FIG. 40 shows a conventional scenario outer frame.
The conventional scenario frame is constituted by a scenario header 920, a scenario head list 921, a scenario second list 922, a scenario third list 923, a scenario fourth list 924, and a scenario fifth list 925 being linked. .

  At this time, the scenario name and the number of images are set in the scenario header 920. The list 921 to 925 of the scenario outer frame includes the image file name and the storage location in the image storage unit 903 in the reproduction information of each image. Set.

The total playback time of the scenario header 920 and the visual effect information and playback time information of each list 921 to 925 of the scenario outer frame are blank.
The scenario generation unit 904 generates such a scenario outer frame, and then determines a visual effect to be assigned to each image in the order of image reproduction.

  Referring back to FIG. 39, the scenario generation unit 904 first refers to the scenario head list 921 of the scenario outer frame (S906), and determines the visual effect to be assigned to the image 31.

  At this time, the scenario generation unit 904 determines a visual effect to be assigned to the image 31 by generating a random number using a random function and dividing it by 4 (S908) and obtaining the remainder after the division.

  That is, if the remainder is 1 (Yes in S910), the scenario generation unit 904 adds a fade-in effect that gradually sharpens the image to the visual effect information in the scenario head list 921 (S912), and if the remainder is 2 (Yes in S914), a fade-out effect that gradually erases the image is added to the visual effect information in the scenario head list 921 (S916). If the remainder is 0 (Yes in S918), a zoom-in effect for enlarging the image toward the center of the image is added to the visual effect information in the scenario head list 921 (S920). If the remainder is 3 (No in S918). Then, the Pan effect for enlarging and displaying one part of the image and moving the display part is added to the visual effect information in the scenario head list 921 (S922).

  Furthermore, when adding the visual effect to be assigned to the visual effect information of the scenario head list 921, the scenario generation unit 904 sets the playback time of the image and adds it to the playback time information of the scenario head list 921. Here, the reproduction time of the image to which the zoom-in effect is applied is set to 3 seconds (45 frames), and the reproduction time of the other images is set to 2 seconds (30 frames).

  Thereafter, the scenario generation unit 904 determines whether the list to which the visual effect information and the reproduction time information are added is the last list (S924). If the list is the last list (Yes in S924), the scenario generation unit 904 determines the scenario generation. The processing operation is terminated, and if it is not the last list (No in S924), the next list of the scenario outer frame is referred to (S926), and the processing operation for determining the visual effect is repeated.

In this manner, when the scenario generation unit 904 repeats the processing operation for assigning the visual effect to the last image in the reproduction order, the scenario shown in FIG. 41 is completed.
FIG. 41 is a diagram showing a first example of a completed scenario in the related art.

  In this completed scenario, information indicating the total playback time of 12 seconds is described in the scenario header 930, and visual effect information for assigning a fade-in effect is described in the scenario head list 931 in which the playback information of the image 31 is described. Thus, reproduction time information of a reproduction time of 2 seconds is described. Similarly, the visual effect information and the reproduction time information to be assigned are described in the list in which the reproduction information of each image is described.

  That is, according to this completed scenario, the image 31, the image 32, the image 33, the image 34, and the image 35 are reproduced in this order, the image 31 has a fade-in effect, the image 32 has a zoom-out effect, and the image 33 has a fade-out. It is shown that the effect, image 34 is subjected to the Pan effect, and image 35 is subjected to the visual effect of the zoom-in effect. In addition, it is shown that the reproduction times are 2 seconds, 3 seconds, 2 seconds, 2 seconds, and 3 seconds (total 12 seconds), respectively.

  When the operation input unit 901 receives an input of a short film title from the user, the scenario generation unit 904 displays a title image (a black screen image prepared for the title) at the top of the completed scenario list. Playback information may be inserted.

  At this time, the scenario generation unit 904 adds visual effect information and reproduction time information (2 seconds) indicating that the character input by the user is embedded in the center portion of the image to the top of the scenario list. The completed scenario is shown in FIG.

FIG. 42 is a diagram illustrating a second example of a completed scenario in the related art.
In this completed scenario, the title image, the image 31, the image 32, the image 33, the image 34, and the image 35 are reproduced in this order. The title image has a character embedding effect, the image 31 has a fade-in effect, and the image 32 has a zoom. The out effect, the image 33 has a fade-out effect, the image 34 has a Pan effect, and the image 35 has a zoom-in effect. The playback times are 2 seconds, 2 seconds, 3 seconds, 2 seconds, 2 seconds, and 3 seconds (14 seconds in total), respectively.

  That is, in the scenario header 940, information indicating the total playback time of 14 seconds is described, and in the scenario head list 941 in which the playback information of the title image is described, the character embedding effect for embedding the character input by the user in the center portion of the image Is described, and the reproduction time information of the reproduction time of 2 seconds is described. Similarly, the visual effect information and the reproduction time information to be assigned are described in the list in which the reproduction information of each image is described.

  Subsequently, the image reproduction unit 905 receives the address information of the scenario generated by the scenario generation unit 904 by the procedure as described above, and refers to the reproduction information of the image stored in the top list of the scenario.

  Then, the image playback unit 905 reads out and decompresses the compressed image from the image storage unit 903 based on the referenced playback information, and instructs the visual effect information while performing time management based on the playback time information (number of frames). The displayed visual effect is applied and output to the display unit 906. When the reproduction of the first image is completed, the image reproduction unit 905 executes the reproduction process with reference to the reproduction information of the next image in the scenario, and repeats this until the reproduction process of the last image (image 35) is completed. .

Here, the visual effect of the image output and displayed on the display unit 906 will be described with reference to FIG.
FIG. 43 is a diagram showing a screen display example of a conventional short film generating apparatus 900.

Here, the Pan effect that moves in the diagonal direction of the image will be described.
According to the Pan effect in the image diagonal direction, the short film generating apparatus 900 displays the display screen example 950 and then enlarges and displays the first display area 951 and moves the display portion along the display transition line 952. Then, the second display area 953 is displayed and the reproduction is finished.

In this way, the conventional short film generation apparatus 900 generates a short video (short film) as if it were a video while performing visual effects such as zoom-in, fade-in, fade-out, and Pan on a plurality of still images. And play it. Such a conventional short film generating apparatus 900 is revolutionary in that it gives a motion like a moving image to a still image and opens up a new entertainment world using a digital still image.
JP 2002-23914 A Japanese Patent Laid-Open No. 11-15993

  However, in the above-described conventional image display apparatus, when the resolution (number of pixels) of the display image is larger (larger) than the resolution (number of pixels) of the display, the display It is necessary to store a thumbnail image having a smaller number of pixels than the number of pixels. Such a situation is very remarkable in a display terminal such as a mobile phone that has a very small display portion. A thumbnail image having a resolution of 1/64 of the original image is displayed as QVGA (320 × 240 pixels = about 80,000 pixels (76800)). The entire display is performed on a display device having a size display area.

That is, in such a case, since the resolution of the entire image is lowered, there is a problem that visual image quality is deteriorated.
In addition, since the place (interest of interest; ROI; Region Of Interest; for example, a human face) in the image is relatively small, there is a problem that the visibility is lowered as the image quality deteriorates.

  In order to solve such problems, it is conceivable to store the original image and display the original image on a display. However, the user will see a part of the large image on a small display device (screen), and the user This causes a problem that an operation load is applied to. More specifically, since an XGA-sized image is about 10 times larger than a QVGA-sized display area, if a user tries to find an ROI, the user has to search for it while scrolling. End up. In addition, when a person's face appears small, the user needs to zoom in on that portion and display it, and the user's operation load is high.

  Similarly, in the case where about nine thumbnail images are displayed as a list on the display device, the thumbnail images must be further reduced to about 80 × 60 pixels. Even in such a case, it can be pointed out that the visibility of each thumbnail image is reduced due to a reduction in resolution, and there is a problem that it is difficult to understand what is reflected.

  In addition, according to the conventional short film generating apparatus, a visual effect is randomly or fixedly determined for an image selected in advance by a user, and a scenario necessary for reproduction is generated. Can not do it. In other words, when deciding the visual effect to be assigned to each image, the visual effect is arbitrarily assigned without considering the content of the image, so that there is a problem that it is not possible to generate an impact scenario that makes use of the characteristics of the image. is there.

  In addition, according to the conventional short film generating apparatus, there is a problem that it takes time and effort because the user needs to input a title and select an image when generating a scenario.

  Therefore, the present invention has been made in view of such problems, and has reduced image quality degradation, improved visibility, reduced user operation load, and image display apparatus, image display method, and the like. It is a first object of the present invention to provide an image display device that can easily determine what is being displayed when displaying a list of thumbnail images.

  In addition, the present invention provides a short film generation device that can reproduce images with appropriate visual effects, and can generate a scenario without the need for user input such as title input and image selection. It is a second object to provide a short film generating apparatus capable of performing the above.

In order to achieve the above object, the present invention takes the following technical means.
That is, the image display device according to the present invention is an image display device including display means for displaying an image, and holds an original image formed with a larger number of pixels than the number of pixels that can be displayed on the display means. It is determined whether or not an object is included in the original image held by the image holding means and the original image holding means. When at least one or more objects are included, object information relating to each object An object that is an area including an object in the original image, based on an object determination unit that generates the image, an original image designated by the user, and at least one object information corresponding to the original image Extract at least a part of the area from the original image and match the number of displayable pixels on the display means. A display image generating unit configured to generate a display image, wherein the object determination unit includes a plurality of object types including a plurality of types among a person, an animal, a plant, a movable property, and a real estate based on the shape of the object. The object type determination unit that determines the type of one of the objects, embeds the determined type of the object in the object information, and the priority corresponding to the determined type of the object is further set in the object information. An object priority setting unit to be embedded, and the display image generation means indicates that the priority is higher among at least one of the object information based on the priority embedded in the object information. An object area corresponding to the object information is extracted from the original image. And wherein the door.
The image display device according to the present invention is an image display device including display means for displaying an image, and holds an original image formed with a larger number of pixels than the number of pixels that can be displayed on the display means. An image holding unit and an object determination unit that determines whether or not an object is included in the original image held in the original image holding unit, and generates object information related to the object when the object is included And extracting from the original image at least a part of an object area that is an area including the object in the original image based on the original image designated by the user and the object information corresponding to the original image. Display image generating means for generating a display image in accordance with the number of displayable pixels in the display means. And features.

  As a result, a display image including at least a part of the object area is generated from the original image, so that a part including at least a part of the object area is displayed in a large size. Therefore, it is possible to provide an image display device that improves visibility and reduces the operation load on the user.

  Here, the object determination means includes an object extraction unit that extracts the position and size of an object included in the original image and embeds the extracted position and size of the object in the object information, and generates the display image generation The means determines a portion to be extracted from the original image based on the position and size of the object embedded in the object information.

According to this configuration, it is easy to determine the part of the original image that includes at least a part of the object area.
The object extraction unit may extract the object using at least one of pattern matching, color distribution information, edge information, region division information, and layout information.

  The object determination unit further includes an object type determination unit that determines the type of the object extracted by the object extraction unit, and embeds the determined object type in the object information. The display image generation unit includes: A portion to be extracted from the original image may be determined based on the position, size, and type of the object embedded in the object information.

This also facilitates determination of the portion of the original image that includes at least a part of the object area.
The type of the object may include at least one of a person, a human face, a dog, a cat, a book, an article including a character string, a signboard, and a price tag.

As a result, a portion including at least a part of various types of object areas is displayed large.
The object determination unit further includes an object priority setting unit that embeds a priority corresponding to the object type determined by the object type determination unit in the object information, and the display image generation unit includes the display A portion to be extracted from the original image may be determined based on the priority of the object embedded in the object information.

This also facilitates determination of the portion of the original image that includes at least a part of the object area.
The object determination unit includes an object extraction unit that extracts the position and size of an object included in the original image and embeds the extracted position and size of the candidate object in the object information. The means may determine the portion to be extracted from the original image so that the center of the position of the object embedded in the object information is included.

Thereby, the visibility of the object can be improved.
The image display apparatus further includes at least one of an imaging unit that captures an original image, an image input unit that inputs an original image via a storage medium or a transmission medium, and the imaging unit and the image input unit. Storage means for storing the original image acquired by at least one in the original image holding means can be provided.

According to this configuration, the original image can be acquired by various measures.
Further, the display image generating means further extracts at least a part of an object area, which is an area including the object in the original image, from the original image based on the object information, and is predetermined. A thumbnail image generating means for generating a thumbnail image having the number of pixels is provided. Here, the original image holding means holds a plurality of original images, and the display image generating means lists thumbnail images of the original images generated by the thumbnail image generating means in the display means. Preferably, the predetermined number of pixels is the number of pixels assigned to display one thumbnail image in the display means.

As a result, it is possible to easily determine what is displayed even when displaying a list of thumbnail images.
Further, the display image generation means may generate a display image in accordance with the number of pixels that can be displayed on the display means by enlarging or reducing a portion extracted from the original image. The image generation means may determine the portion to be extracted from the original image so that the number of pixels extracted from the original image is equal to the number of pixels that can be displayed on the display means.

  As a result, a display image is generated from the original image so that a portion including at least a part of the object region is displayed large, and thus an image display device that improves visibility and reduces the user's operation load is realized. It becomes possible.

  The short film generation apparatus according to the present invention is a short film generation apparatus that generates an image using at least one still image, and that receives one or more still images input from the outside. And an object extracting means for determining whether or not an object is included for each input still image, and for extracting information related to the object when the input still image includes an object; Generating a scenario that is video playback information by determining the playback order of the input still image and the visual effect to be applied to the object included in the still image according to the determination result of the object extraction means And a scenario generation means. Here, the object is a character area, and the object extracting means determines whether or not a character area is included for each input still image, and the input still image includes a character area. A character area extraction unit that extracts information about the character area, the scenario generation means, according to the reproduction order of the input still image and the determination result of the character area extraction unit, A scenario that is video reproduction information may be generated by determining a visual effect to be applied to a character area included in the still image.

  According to this configuration, an object (character area) included in a still image is extracted and a visual effect is applied to the object (character area). Therefore, the object (character area) is emphasized, and an impact short film is created. Can be generated.

  Here, the scenario generation means may generate a scenario for applying an enlargement effect to the character area included in the still image according to the determination result of the character area extraction unit. In addition, the scenario generation means can generate a scenario that applies a Pan effect to a character area included in the still image according to a determination result of the character area extraction unit. Further, the scenario generation unit may generate a scenario for correcting a character area included in the still image according to a determination result of the character area extraction unit.

  The scenario generation means determines a visual effect to be applied to the character area based on at least one of the shape, size, or color of the character area out of the information related to the character area extracted by the character area extraction unit. Thus, it is preferable to generate a scenario.

  As a result, the visual effect to be applied to the character region is selected according to the characteristics of the character region of the still image, so that a visual effect suitable for the character region is applied, and a more impactful short film can be generated.

  The short film generation apparatus further includes an image selection unit that selects a still image to be reproduced based on a still image shooting time determined by the character region extraction unit to include a character region, and the scenario generation unit May generate a scenario of a reproduced image selected by the image selection means.

As a result, an image group temporally associated with the key image is automatically selected, thereby eliminating the need for the user to select a still image to be used for the short film.
Furthermore, the scenario generation means can generate a scenario using a character area included in the still image as a title according to a determination result of the character area extraction unit.

As a result, the character area of the image is automatically taken into the title image, so that it is possible to save the user from inputting the title character from the keyboard or the like.
Further, the visual effect determined by the scenario generating means is at least an enlargement / reduction effect for displaying the still image by enlarging or reducing the object, and moving the viewpoint to / from the object. The viewpoint moving effect for displaying the still image, the zoom-in / out effect for displaying the still image by performing zoom-in or zoom-out with the object as a focus, and the region including the object in the still image are cut out It may be any one of the clipping effects.

  Thereby, any of various visual effects can be applied to the object, and a short film generating apparatus capable of improving visibility and generating an impact short film is realized.

  Note that the present invention can be realized not only as such an image display apparatus but also as an image display method using steps characteristic of the image display apparatus, or by performing these steps as a computer. It can also be realized as a program to be executed. Furthermore, it can also be realized as a portable terminal having such an image display device.

  Similarly, the present invention can be realized not only as such a short film generating apparatus, but also as a short film generating method including steps characteristic of the short film generating apparatus. These steps can also be realized as a program for causing a computer to execute. Needless to say, such a program can be distributed via a recording medium such as a CD-ROM or a transmission medium such as the Internet.

  As described above, according to the image display device of the present invention, the display image is generated from the portion of the original image that includes at least a part of the object region, so that the portion that includes at least a portion of the object region is large. Is displayed. Therefore, it is possible to provide an image display device that improves visibility and reduces the operation load on the user.

  In addition, according to the image display device of the present invention, the thumbnail image is generated from the portion of the original image including at least a part of the object area. Is done. Therefore, it is possible to easily determine what is being displayed even when displaying a list of thumbnail images.

  Therefore, according to the present invention, it is possible to perform various image processing focusing on the object of the original image, display an image with high visibility even on a display with a small number of pixels, and digital cameras and mobile phones have become widespread. However, the practical value of the present invention is very high today.

  Moreover, according to the short film production | generation apparatus which concerns on this invention, since the suitable visual effect can be selected and given with respect to an image, the short film with a more impact can be produced | generated.

Furthermore, since an image selected by the user is used as a key image and related images are automatically selected and reproduced, it is possible to save the user from having to select all images.
Furthermore, since the character area is automatically embedded in the title image, the user does not need to manually input the title.

  Therefore, it can be said that the practical value of the present invention is extremely high today, because users can enjoy high-impact short films with a simple operation, and the number of users who have a large number of still images is remarkably increasing.

Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.
(Embodiment 1)
First, an image display apparatus according to Embodiment 1 of the present invention will be described with reference to FIGS.

FIG. 1 is a diagram showing an external configuration of a mobile phone having a digital camera and an image display device.
The digital camera 100 is provided with, for example, a 3 million pixel CCD, and the subject is photographed at 1280 × 960 pixels, and the original image of 1280 × 960 pixels and the thumbnail image of 160 × 120 pixels are converted into an EXIF format. Then, it is stored in the recording medium 207.

  On the other hand, the mobile phone 115 includes an antenna 201 for transmitting and receiving radio waves to and from a base station, a flash, and the like, and a camera unit 203 capable of capturing moving images and still images, videos captured by the camera unit 203, A display unit 202 such as a liquid crystal display that displays data such as video received by the antenna 201, a main body unit configured with operation keys 204, an audio output unit 208 such as a speaker for audio output, and a voice input A voice input unit 205 such as a microphone and a slot unit 206 for allowing the recording medium 207 to be attached to the mobile phone 115 are provided.

  The camera unit 203 includes, for example, a CCD with 1.3 million pixels, and images a subject with a maximum of 1024 × 1280 pixels, which is larger than the number of pixels (240 × 320) that can be displayed on the display unit 202. The recording medium 207 is an SD memory card (registered trademark) in which a flash memory element, which is a kind of EEPROM (Electrically Erasable and Programmable Read Only Memory) that is an electrically rewritable and erasable nonvolatile memory, is stored in a plastic case. ) And the like for storing captured moving image or still image data, received mail data, encoded or decoded moving image data or still image data, and the like. .

FIG. 2 is a diagram schematically showing a block configuration of the image display apparatus according to the first embodiment.
The image display device 1 is mounted on the mobile phone 115 described above, and includes an external interface unit 10 that functions as an external interface, a memory unit 20 that functions as a storage device, and an original image stored in the memory unit 20 with an object. If it is determined whether or not an object is included, the object determination unit 30 that generates object information related to the object, and the original image specified by the user and the object information corresponding to the original image Display for controlling to generate at least one display image in accordance with the number of pixels of the display device 50 from the portion of the original image including at least a part of the object area, and to display the generated display image on the display device 50 The image generating unit 40 and a display 50 that is a part for displaying an image are provided. The object determination unit 30 and the display image generation unit 40 are configured by a program for performing predetermined arithmetic processing, a DSP that executes the program, and the like.

The external interface unit 10 includes an image capturing unit 110, an image input unit 120, and a user IF unit 130.
The image capturing unit 110 captures an image using an arbitrary image capturing unit typified by a CCD or the like and captures the image in the image display device 1.

  The image input unit 120 receives an image from an arbitrary recording medium represented by an SD memory card (registered trademark) or a multimedia card, or via an arbitrary network represented by an Ethernet (registered trademark) cable or a wireless LAN. Is acquired and taken into the image display device 1.

  The user IF unit 130 provides a user interface of the image display device 1 through any input means represented by a keyboard, a numeric keypad, a touch panel, voice recognition, and the like. The user IF unit 130 can also improve user convenience through icon display, help display, and the like.

  The constituent elements of the external interface unit 10 are not limited to the image capturing unit 110, the image input unit 120, and the user IF unit 130, but can be connected to / from any external environment / external device and input / output as necessary. It can be performed.

  For example, a microphone unit 205 for voice recognition, a speaker unit 208 that outputs data in the image display device 1 such as help / music information, a light source unit such as a flash when shooting with the image capturing unit 110, and power ON / OFF An arbitrary external input / output device included in a general mobile phone, PDA, personal computer, game machine, camera (digital camera, movie), such as an internal state display unit that indicates the internal state of the image display device 1 with light. The external interface unit 10 can be a constituent element.

  The memory unit 20 includes an image holding unit 210, an object information holding unit 220, a thumbnail image holding unit 230, a user-specified image holding unit 240, and a display image holding unit 250. The memory unit 20 can use an arbitrary recording medium such as an SD memory card (registered trademark) or a hard disk as a memory in addition to a volatile memory and a nonvolatile memory as necessary.

  The image holding unit 210 stores image data input to the image display device. The image holding unit 210 can store data in a plurality of formats, resolutions, and trimming sizes for one original image data as necessary. For example, saving in multiple formats such as JPEG and bitmap, saving at different resolutions such as VGA, XGA, QVGA, etc., and saving as a trimmed (clipped) image part of the image . The image holding unit 210 can store data in any combination without being limited to the storage examples given here.

The object information holding unit 220 holds object information generated by the object determination unit 30.
The thumbnail image holding unit 230 holds a thumbnail image generated by the display image generation unit 40.

The user designated image holding unit 240 holds a display image designated by the user.
In particular, the display image holding unit 250 temporarily holds image data of an image to be displayed on the display device 50, and corresponds to, for example, a VRAM in a general personal computer.

The object determination unit 30 includes an object candidate extraction unit 310, an object candidate determination unit 320, and an object priority determination unit 330.
The object candidate extraction unit 310 estimates and extracts object candidates from the image.

  Here, the candidate extraction method in the object candidate extraction unit 310 and the determination method in the object candidate determination unit 320 are existing arbitrary images disclosed in Japanese Patent Application Laid-Open No. 2002-304627, Japanese Patent Application Laid-Open No. 10-21394, and the like. Processing techniques can be used.

  For example, the object candidate extraction unit 310 uses the pattern matching based on the template, or uses the region division shape, the binarized image, the edge extracted image, or the like for the correlation of the frequency, color, layout, etc. Extract candidates.

The object candidate determination unit 320 determines whether the object candidate is true or false.
In addition, the object candidate determination unit 320 determines the type of the object and whether the object is an object using an arbitrary process for the region that is the object candidate. For example, color information / edge information can be used.

  Here, a specific example is given where the object is a human face. First, a gray color template for determining a human face candidate is prepared. The correlation between the gray color template and the target image is calculated, and an area having a high correlation value is set as a “human face candidate area”. Furthermore, “determination of whether or not the face is a human face” can be performed by obtaining the ratio of the skin color to the candidate area.

The object priority determination unit 330 embeds the priority corresponding to the object type determined by the object candidate determination unit 320 in the object information.
The display image generation unit 40 includes a thumbnail image generation processing unit 410, an enlargement / reduction processing unit 420, a viewpoint movement processing unit 430, a zoom-in / out processing unit 440, a cut-out processing unit 450, and the like. An image to be displayed on the display 50 through the holding unit 250 is generated.

  The thumbnail image generation processing unit 410 sets, for every original image stored in the image holding unit 210, a display area including at least a part of the object in the original image based on the position included in the object information. Then, the thumbnail image generation processing unit 410 generates thumbnail images of the set display area from the original image with a predetermined number of pixels, and displays the generated thumbnail images as a list on the display device 50. This makes it possible to present to the user in an easy-to-understand manner what is reflected in the original image and what the subject is.

  The enlargement / reduction processing unit 420 enlarges and reduces the original image. As a result, the object included in the original image can be shown large, the details of the object can be shown well, or the whole can be shown.

As a condition for automatically enlarging an object,
1) If the user has set in advance to perform “automatic enlargement display”,
2) When “Auto Enlarge Display” is selected as the default setting,
3) When the object is recognized and the “Enlarge display” setting is selected, etc.
Arbitrary conditions can be set depending on whether or not an object exists.

  The viewpoint movement processing unit 430 performs processing corresponding to pan (pan) and tilt (tilt). Here, “pan” is a kind of camera operation at the time of shooting and corresponds to “shooting a wide range while moving the camera left and right while fixing the position of the camera”. In the case of the viewpoint movement processing unit 430, this means that processing is performed so that the image displayed on the display device 50 “displays a wide range while moving left and right”.

  Similarly, “tilt” corresponds to “moving the camera fixed to a tripod up and down”, but in the case of the viewpoint movement processing unit 430, “displays a wide range while moving up and down”. Means to process. Note that the viewpoint movement method of the viewpoint movement processing unit 430 is not limited to the left / right / up / down directions, and the viewpoint can be moved in any direction at any timing.

  The zoom-in / out processing unit 440 performs processing corresponding to zoom-in (zoom-in) and zoom-out (zoom-out). Here, zoom-in (zoom-in) is a kind of camera operation at the time of shooting and corresponds to “obtaining images that are continuously approached by a zoom lens while the position of the camera is fixed”.

  Zoom-out (zoom-out) is equivalent to “obtaining continuously moving images”. In the case of the zoom-in / out processing unit 440, as in the previous viewpoint movement processing unit 430, the image displayed on the display device 50 is "approaching or leaving continuously with the viewpoint fixed." To process. Note that the viewpoint of the zoom-in / out processing unit 440 is not necessarily fixed, and the viewpoint may be moved during processing. As a result, in the case of zooming in, it is possible to concentrate the line of sight to increase interest, to show clearly, or to excite. In the case of zooming out, the surrounding situation can be shown.

Next, the cutout processing unit 450 corresponds to trimming the image.
Here, “trimming” means “to arrange a screen by omitting unnecessary unnecessary parts when enlarging a photo original”. In this cropping processing unit 450, a part of an image is arbitrarily set. This corresponds to cutting and displaying the shape. Here, an arbitrary image / text / texture (pattern to be pasted / design) can be displayed on the background from which the object is cut out. These data may be stored in advance in the image holding unit 210, may be newly generated by the display image generation unit 40, or may be acquired through the external interface unit 10. Good.

The slide show unit 460 displays the display image stored in the user designated image holding unit 240 on the display device 50 as a video.
The display 50 is composed of a liquid crystal panel or the like, and displays an image with horizontal 240 pixels × vertical 320 pixels.

Next, an operation when inputting a still image (original image) on the image display apparatus 1 configured as described above will be described.
FIG. 3 is a flowchart showing an algorithm when an original image is input in the external interface unit 10 and the object determination unit 30.

The image input unit 120 and the object determination unit 30 repeatedly execute the following processing for each original image for all original images recorded on the recording medium 207 (loop A).
First, the image input unit 120 reads the original image recorded on the recording medium 207 to the work memory in the image display device 1 (S100), and stores the read original image in the image holding unit 210 (S102). The image data storage format may be a compression format represented by the JPEG format or an uncompressed format.

  Next, the object candidate extraction unit 310 of the object determination unit 30 reads the original image stored in the image holding unit 210 into the work memory (S104), extracts all object candidates included in the original image (S106), The position and size of the object candidate are acquired. When the acquisition of the position and size of the object candidate is completed, the object candidate determination unit 320 determines the type of the object candidate and acquires the type of the object candidate (S108). When the acquisition of the object candidate type is completed, the object priority determination unit 330 acquires an object priority set in advance for each type (S110).

  After the acquisition of the object priority, the object determination unit 30 collects the object information by collecting the file name of the original image, the position and size of the object candidate, the type of the object candidate, and the object priority in a table format. The generated object information is stored in the object information holding unit 220 (S112).

  When all the original images are stored in the image holding unit 210 and the corresponding object information is stored in the object information holding unit 220, the image input unit 120 and the object determination unit 30 perform the original image input process / object determination. The process ends.

  Through the above processing, all the original images stored in the recording medium 207 (see FIG. 4) are sequentially stored in the image holding unit 210, and object information corresponding to each original image (see FIG. 5). Is generated and stored in the object information holding unit 220.

Here, FIG. 6 shows an object determination example.
The object candidate extraction unit 310 applies, for example, object templates of “face”, “tie”, “apple”, and “marker” to the original image 1301 shown in FIG. As shown in FIG. 6, the object candidate 1311 is the most “face-like area” and the most “tie-like area” is the object from the original image 1301. The candidate 1312 is extracted as the most “apple-like area”, and the object candidate 1313 is extracted as the “marker-like area”. These candidates 1311 to 1314 are an object candidate group 1302.

  Note that the position and size of the object can be represented by the coordinates of the start point and end point of one diagonal line of the rectangle surrounding the extracted object. Further, the position and size of the object may be represented by the coordinates of the center of the circle surrounding the extracted object and the size of the radius. In this case, the size of the object is represented with a margin including the periphery of the object in the first embodiment, but may be represented without a margin.

  Next, the object candidate determination unit 320 performs “object determination” on each of the region candidates 1311 to 1314 as described above, thereby determining “whether it is a face”, “whether it is a tie”, “whether it is an apple”. Is determined, and the type of the object is determined. The determination method includes, for example, a method based on color distribution as described above (a lot of skin color is used for a face) and a method based on a shape (using a typical shape of a tie). Since the object candidate 1313 does not include “apple-like color”, it is determined as “not an apple”, and the object type is “unknown”.

  Furthermore, the object priority determination unit 330 holds the object priority setting table 331 shown in FIG. 7 in advance, and the object priority setting table 331 is used to classify each region candidate 1311 to 1314 as a type. The priority levels “5”, “3”, “1”, and “3” of the objects are acquired from the corresponding relationship.

  Thereby, object information (see FIG. 5) corresponding to the original image 1301 (file name: a001.jpg) shown in FIG. 6 is generated and stored in the object information holding unit 220.

  In the object priority setting table 331 shown in FIG. 7, “person's face”, “tie”, “necklace”,..., “Apple”, “mandarin orange”,. , “Erase”,..., “Unknown” are registered in advance, but as long as the object candidate determination unit 320 can determine the type, person (whole body), person (waist shot), person (breast shot), dog, cat It can be applied to all kinds of objects such as animals such as animals, plants such as lemons, movables such as desks, chairs, books, price tags, real estate such as tourist information maps, signs including station names, company names, etc. Not too long.

  In addition, the priority of the objects “person's face”, “tie”, “necklace”,... Is preset to “5”, “3”, “3”,. You may make it set according to.

As a result, the positions of the objects and the like are obtained for all the original images to be displayed, and image processing centered on the objects becomes possible.
Next, an operation when a thumbnail image is generated by the image display device 1 configured as described above will be described.

FIG. 8 is a flowchart showing an algorithm at the time of thumbnail image generation in the thumbnail image generation processing unit 410 of the display image generation unit 40.
The image display apparatus 1 is configured to form one thumbnail image with 4800 (horizontal 80 × vertical 60) pixels in order to display a list of nine thumbnail images on the display device 50.

The thumbnail image generation processing unit 410 repeatedly executes the following processing for each original image for all original images held in the image holding unit 210 (loop B).
First, the thumbnail image generation processing unit 410 refers to corresponding object information and determines whether an object exists in the original image (S200). If it exists (Yes in S200), the target of the thumbnail image is determined as the object area (S202). If there are a plurality of objects, the area is determined to have a high priority. When the determination of the area is completed, the thumbnail image generation processing unit 410 calculates the reduction ratio of the object area from the ratio of the number of pixels between the object area and the display 50 (S204), and performs thinning or the like based on the calculated ratio. By doing so, a thumbnail image of the object is generated (S206), and the thumbnail image is stored in the thumbnail image holding unit 230 (S208).

  On the other hand, if it is determined that the object does not exist (No in S200), the thumbnail image generation processing unit 410 determines the target of the thumbnail image as the entire original image (S210), and the entire original image and the display 50 are determined. A reduction ratio of the original image is calculated from the ratio of the number of pixels (S212), and a thumbnail image is generated for the entire original image by thinning out based on the calculated ratio (S214), and the thumbnail image is saved. (S216).

  When the generation of thumbnail images for all original images held in the image holding unit 210 is completed for each original image, the thumbnail image generation processing unit 410 ends the generation of thumbnail images.

Here, FIGS. 9A and 9B show examples in which thumbnail images are displayed as a list.
FIG. 9A shows a case where thumbnail images centered on an object are displayed as a list. By extracting and displaying the object as in this display example, it is possible to present to the user what is included in the image in an easy-to-understand manner.

  FIG. 9B shows a thumbnail image created from the entire original image as in the conventional example. In this thumbnail list display, the size of each displayed image is very small, so it is often very difficult to see what is contained in the image. Recognize.

Next, display image generation processing will be described.
FIG. 10 is a flowchart showing display image generation processing.
First, the thumbnail image generation processing unit 410 of the display image generation unit 40 displays a list of thumbnail images (S300), and waits for a thumbnail image to be selected (S302). This selection is performed by inputting a number assigned to the thumbnail image or moving a cursor key. When the thumbnail image is selected, the display image generation unit 40 reads the corresponding original image (S304), then reads the object information (S306), and determines whether or not the object exists in the original image. (S308).

  When the object exists (Yes in S308), the display image generation unit 40 determines the effect type (S310) and generates a display image corresponding to the effect type (S312). The generated display image is transferred to the display device 50 via the display image holding unit 250 and displayed by the display device 50 (S314).

  On the other hand, when the object does not exist (No in S308), the display image generation unit 40 generates a display image from the entire original image (S316). The generated display image (entire image) is transferred to the display device 50 via the display image holding unit 250 and displayed by the display device 50 (S314).

  When the transfer to the display device 50 is completed, the display image generation unit 40 determines whether or not the return button (F1 button) has been pressed (S318). When the return button is pressed (Yes in S318), it is regarded that the browsing of the image has been completed, and the process returns to Step S300 and waits for a thumbnail image to be selected. As a result, it is possible to sequentially browse the display images corresponding to the thumbnail images shown in FIG.

  When the return button has not been pressed (No in S318), the display image generation unit 40 determines whether the save button has been pressed (S320). When the save button is pressed (Yes in S320), the display image is saved in the user-specified image holding unit 240 (S322), and the process proceeds to Step S324. If the save button has not been pressed (No in S320), the display image generation unit 40 proceeds to Step S324. As a result, only the display image that the user likes can be stored in the user-specified image holding unit 240, and only the display image that the user likes can be displayed.

  In step S324, the display image generation unit 40 determines whether or not the on-hook button has been pressed (S324). If the on-hook button has not been pressed (No in S324), the display image generation unit 40 considers that the user is browsing the display image and returns to Step S312 to generate a display image.

  When the on-hook button is pressed (Yes in S324), the display image generation unit 40 regards that the browsing of the display image by the user has ended, and ends the display image generation process.

  FIG. 11 is a flowchart illustrating a subroutine of display image generation processing corresponding to the type of effect.

  First, the display image generation unit 40 determines the type of effect (S400). This determination is performed by randomly selecting enlargement / reduction processing, viewpoint movement processing, zoom-in / out processing, and cut-out processing.

  If the determined effect is enlargement / reduction processing, the enlargement / reduction processing unit 420 first determines whether it is the initial stage of display image generation (S402). If it is the initial stage of display image generation (Yes in S402), the enlargement / reduction processing unit 420 determines the object (human face) area and the entire image area in the original image as display areas (S404). Then, “1” is set in the enlargement flag (S406), an enlarged display image is generated from the object (human face) region (S408), and the process returns to the main routine shown in FIG.

  This enlarged display image was calculated by calculating the enlargement / reduction ratio of the object (human face) area from the ratio of the number of pixels of the object (human face) area and the display 50 in the original image. It is generated by performing complementation or thinning based on the ratio. As a result, only the area of the object (human face) is displayed on the display 50 as shown in FIG.

  On the other hand, when it is not the initial stage of display image generation (No in S402), the enlargement / reduction processing unit 420 determines whether the enlargement flag is “1” (S410) and whether the center button is pressed (S412). Are sequentially determined. As a result of the determination, if the enlargement flag is “1” (Yes in S410) and the center button is pressed (Yes in S412), the enlargement / reduction processing unit 420 regards that the enlarged image has been browsed, Is set to "0" (S414), a reduced image is generated from the entire image area (S416), and the process returns to the main routine shown in FIG.

  This reduced display image is generated by calculating the reduction ratio of the original image from the ratio of the number of pixels of the entire area of the original image and the display 50, and performing thinning or the like based on the calculated ratio. As a result, the entire image is displayed on the display 50 as shown in FIG.

  On the other hand, as a result of the determination, if the enlargement flag is “0” (No in S410) and the center button is pressed (Yes in S412), it is regarded that the browsing of the reduced image is completed, and the process returns to step S414. Then, “1” is set in the enlargement flag (S406), and an enlarged display image is generated from the area of the object (human face) (S408).

  If the center button has not been pressed (No in S412), the enlargement / reduction processing unit 420 regards that the enlarged image or the reduced image is being browsed, and returns to the main routine shown in FIG.

  Such an enlarged image makes it possible to make the subject appear larger or show details, to reduce visual image quality degradation, improve visibility, and reduce the user's operation load.

Further, since the reduced display image is also created from the entire original image, it is possible to reduce visual image quality degradation, improve visibility, and reduce the user's operation load.
In step S400, when the determined effect is the viewpoint movement process, the viewpoint movement processing unit 430 first determines whether or not the display image generation is an initial stage (S420). In the initial stage of display image generation (Yes in S420), the viewpoint movement processing unit 430 determines the area of the object (breast shot) in the original image and the area where the viewpoint is moved to the right side as the display area. (S422) A display image is generated from the area moved to the right (S424), and the process returns to the main routine shown in FIG.

  This display image is obtained by calculating the enlargement / reduction ratio of the object area from the ratio of the number of pixels of the area where the viewpoint in the original image is moved to the right and the display 50, and complementing or reducing based on the calculated ratio. , And so on. As a result, as shown in FIG. 13A, the display 50 displays half of the object (breast shot) and the area around the person.

  On the other hand, when the display image generation is not initial (No in S420), the viewpoint movement processing unit 430 waits for a predetermined time (for example, 2 seconds) to elapse (S426). When the predetermined time has elapsed (Yes in S426), the viewpoint movement processing unit 430 considers that viewing of the right lateral image has ended, generates a display image from the area of the object (breast shot) (S428), Return to the main routine shown in FIG.

  In this display image, the enlargement / reduction ratio of the object area is calculated from the ratio of the number of pixels between the object area and the display 50 in the original image, and complementation, thinning out, or the like is performed based on the calculated ratio. Is generated. Thereby, only the image of the object (breast shot) is displayed on the display 50 as shown in FIG.

  With such a display image, the object can be enlarged over a wide range from the object's surroundings to the object, and the details can be shown well, reducing visual image quality degradation, improving visibility, and user operation load Can also be reduced.

  If the determined effect is the zoom-in / out process in step S400, the zoom-in / out processing unit 440 first determines whether or not the display image generation is an initial stage (S430). If it is the initial stage of display image generation (Yes in S430), the zoom-in / out processing unit 440 determines the object (waist shot) area in the original image and the area zoomed in with the viewpoint fixed as display areas. Then, a display image is generated from the zoomed-in area (S434), and the process returns to the main routine shown in FIG.

  In this zoomed-in image, the enlargement / reduction ratio of the object area is calculated from the ratio of the number of pixels of the zoomed-in area in the original image and the display device 50, and complementation or thinning is performed based on the calculated ratio. Is generated. As a result, as shown in FIG. 14A, the display unit 50 displays a zoomed-in area on the object (breast shot) with the viewpoint as the center of the screen.

  On the other hand, when the display image generation is not initial (No in S430), the zoom-in / out processing unit 440 waits for a predetermined time (for example, 2 seconds) to elapse (S436). When the predetermined time has elapsed (Yes in S436), the zoom-in / out processing unit 440 considers that browsing of the zoomed-in image has ended, and generates a display image from the area of the object (breast shot) (S438). Return to the main routine shown in FIG.

  This zoomed-out image calculates the enlargement / reduction ratio of the object area from the ratio of the number of pixels of the zoomed-out area in the original image and the display device 50, and complements, thins out, etc. based on the calculated ratio Is generated. Thereby, as shown in FIG. 14B, the object (waist shot) and the image around it are displayed on the display 50.

  With such zoomed-in and zoomed-out images, you can get close to the object to make it look larger, show details, and move away from the object to make it look good over a wide area. It is possible to reduce deterioration of the image quality, improve visibility, and reduce the operation load on the user.

Note that the viewpoint of the zoom-in / out processing unit 440 is not necessarily fixed, and the viewpoint may be moved during processing.
In step S400, when the determined effect is the clipping process, the clipping processing unit 450 first determines whether or not it is the initial stage of display image generation (S440). When it is the initial stage of display image generation (Yes in S440), the cutout processing unit 450 determines the area of the object (breast shot) in the original image and the area of the person's face cut out as display areas ( In step S442, a display image is generated from the object (breast shot) area (S444), and the process returns to the main routine shown in FIG.

  This display image calculates the enlargement / reduction ratio of the object area from the ratio of the number of pixels of the object (breast shot) area and the display 50 in the original image, and complements based on the calculated ratio, Generated by thinning out and the like. As a result, the area of the object (breast shot) is displayed on the display 50 as shown in FIG.

  On the other hand, when it is not the initial stage of display image generation (No in S440), the cutout processing unit 450 waits for a predetermined time (for example, 2 seconds) to elapse (S446). When the predetermined time has elapsed (Yes in S446), the cutout processing unit 450 regards that the browsing of the breast shot image is finished, and cuts out only the area of the object (human face) to generate a display image (S448). ), The process returns to the main routine shown in FIG.

  This cut-out image is generated by cutting the outside of the area of the object (human face) and drawing black in the cut area. As a result, an image obtained by cutting out only the object (human face) is displayed on the display 50 as shown in FIG.

  With such a cut-out image, an image of an unnecessary area can be masked or decorated with a comment or the like at the time of shooting, reducing visual image quality deterioration, improving visibility, and user operation load Can also be reduced.

  An arbitrary image / text / texture (pattern to be pasted / pattern) can be displayed on the background from which the object is cut out. These data may be stored in advance in the image holding unit 210, may be newly generated by the display image generation unit 40, or may be acquired through the external interface unit 10.

Next, processing executed by the slide show unit 460 will be described.
FIG. 16 is a flowchart showing processing executed by the slide show 460 unit.
The slide show unit 460 repeatedly executes the following processing for each display image for all the display images held in the image holding unit 210 (loop C).

  The slide show unit 460 reads the display image stored in accordance with the user's designation from the user designated image holding unit 240 (S500), displays the display image on the display 50 (S502), and performs a predetermined time (for example, 2 seconds). ) Wait for elapse (S504).

By repeating such processing, the slide show unit 460 ends the slide show when all the display images have been displayed for a certain period of time.
Here, the user-designated image holding unit 240 generally holds images with improved visibility among images that the user likes.

Therefore, even in a slide show, it is possible to reduce visual image quality deterioration, improve visibility, and reduce a user's operation load.
In the first embodiment, a reduced image is generated and displayed when the center button is pressed when an enlarged image is displayed, and the center button is pressed when a reduced image is displayed. At this time, the enlarged image is generated and displayed. However, as shown in FIG. 17, the enlarged image 421 and the reduced image 422 may be automatically displayed alternately. Thereby, it becomes possible to reduce a user's operation load.

  In the first embodiment, only the image is displayed on the display device 50. However, as shown in FIG. 18, when the enlarged image 423 is displayed on the screen, "return", "whole", " When the reduced image 424 is displayed, “return”, “enlarged”, and “save” icons may be displayed. Thereby, the operativity by GUI can be improved and a user's operation load is also reduced.

  In the first embodiment, two images, that is, the entire image and the enlarged image of the human face are generated and displayed. However, as shown in FIG. 19, the enlarged human face is displayed from the entire image 425a. It is also possible to change the magnification gradually up to the image 425d to make a smooth transition. Thereby, the visual effect can be improved.

  Furthermore, in the first embodiment, the viewpoint movement processing unit 430 performs viewpoint transition within a narrow range, but it is also possible to display while adding a predetermined visual effect. A specific example will be described below.

  Here, it is assumed that the original image is a captured image shown in FIG. Although the entire display or the enlarged display of the object can be automatically performed according to the first embodiment described so far, here, a display method in which attention is gradually paid to the ROI while grasping the entire image. Will be explained.

  Here, it is assumed that the object (face) is extracted as the ROI in the captured image by the object determination unit 30, and the region (final display region) to be finally displayed is the left-side square shown in FIG. The area to be enclosed. On the other hand, the final display area and the area surrounded by the square shown in FIG. 20A farthest from other objects are set as the initial display area. In this case, as a matter of course, the size of the display area is the same between the initial display area and the final display area. Then, a display area movement line (spline line that circulates the object) connecting the “marker”, “unknown”, “tie”, and final display area of the object can be drawn from the initial display area. It is also possible to display such that the viewpoint is moved along the spline line and gradually reaches the final display area from the initial display area.

  In this case, the image displayed on the display device 50 changes on the time axis like an initial display area image 426a to a final display area image 426g shown in FIG. As a result, a panoramic image that gradually extends from the initial display area to the final display area is displayed, and the visual effect can be dramatically enhanced.

Of course, the position / size of the initial display, the position / size of the final display area, the method of moving the display area, etc. are not limited to the above, but are all arbitrary.
That is, it is an arbitrary process that can be performed by a combination of the enlargement / reduction processing unit 420, the viewpoint movement processing unit 430, the zoom-in / out processing unit 440, and the cut-out processing unit 450. While combined, zoom-in / out processing can be added to display the entire image and object.

  Further, through the first embodiment, the image to be reduced and the image data to be enlarged can be generated independently from the original image or can be generated dependently. In the case of independent generation, since a reduced image can be enlarged and displayed without being enlarged again, better image quality can be obtained. In the case of subordinate generation, it is possible to reduce the internal memory by using an image that is particularly reduced and displayed as a base.

  In addition, a general electric device having a display device such as a mobile phone, a PDA, a game machine, a digital camera, a video camera, a personal computer (PC), a notebook PC, or a car navigation is regarded as an image display device. be able to.

  Further, the present invention can be applied not only to displaying a single image but also to displaying a plurality of images continuously in a slide show (short film). In this case, the number of image sensors (the number of CCD pixels) is larger than the number of pixels on the terminal display screen, and a large number (a plurality of) of image data is generally stored, such as a camera-equipped mobile phone or a digital camera. When displaying images in a terminal on a natural terminal, the video is edited while adding not only a slide show of the reduced images, but also an enlarged display of the desired location, such as zooming in to the ROI. As you can see, it is possible to display stylishly.

  In addition, it has an external output, and in particular, it can be saved as a moving image or a data file describing the still image and the playback method, and can be output to the outside as a video output. it can.

Furthermore, an arbitrary sound effect can be given at the time of enlargement / reduction, and music can be given at the time of the above-described slide show reproduction.
(Embodiment 2)
Next, the short film production | generation apparatus which concerns on Embodiment 2 of this invention is demonstrated using FIGS.

FIG. 22 is a block diagram showing a functional configuration of the short film generating apparatus according to Embodiment 2 of the present invention.
The short film generation device 500 is a computer device that generates a scenario of a short video (short film) using one or more input still images and reproduces a short film according to the scenario. , An operation input unit 501, an image input unit 502, a character region extraction unit 503, an image storage unit 504, a scenario generation unit 505, an image reproduction unit 506, and a display unit 507.

The operation input unit 501 is an input device such as a keyboard or a mouse that receives an operation input from a user such as an instruction for capturing still image data or selecting a still image.
The image input unit 502 is a processing unit that captures still image data, such as a JPEG compression format, captured by the digital camera 100 or the like into the apparatus, and is a USB (Universal Serial Bus) or IEEE (Institute of Electrical and Electronic Engineers) 1394. Etc. are realized by the interface.

  The character area extraction unit 503 determines whether or not a rectangular character area is included in the still image captured in the apparatus, and extracts character area information related to the character area when the character area is included. A processing unit, which is realized by an OCR (Optical Character Reader) using a pattern matching method, an edge detection method, or the like. Here, the character area information includes the position of the character area in the still image, the shape, size, and color of the character area. The character area extraction unit 503 stores the extracted character area information in the memory in the apparatus. Hold.

The image storage unit 504 is a storage device such as a hard disk that stores still image data such as a JPEG compression format captured by the digital camera 100.
The scenario generation unit 505 is a processing unit that acquires the selection result of the image received from the user from the operation input unit 501, determines the reproduction order and visual effect of the selected image, and generates a scenario that is reproduction information. This is realized by a CPU and a memory.

  The image playback unit 506 is a processing unit that reads out a still image from the image storage unit 504 according to the scenario generated by the scenario generation unit 505, and repeats an operation of playing back at 15 frames / second while applying a visual effect specified by the scenario. It is realized by a CPU and a memory.

The display unit 507 is a CRT display or a liquid crystal display that displays a user interface screen or an image under the control of the scenario generation unit 505 or the image reproduction unit 506.
The processing operation of each part of the short film generating apparatus 500 configured as described above will be described in detail below.

FIG. 23 is a flowchart showing the processing procedure of the character area extraction unit 503.
First, still image data captured by the user with the digital camera 100 is directly taken into the short film generating apparatus 500 via the image input unit 502 (S600).

  At this time, the character region extraction unit 503 extracts a character region from still image data captured from the image input unit 502 (S602). At this time, the character area extraction unit 503 generates a field for describing the file name of the still image data and the character area extraction information indicating the type of the character area, and stores the field in the memory.

  Here, when the character area is not included in the still image data (No in S604), the character area extraction unit 503 adds “no extraction” to the field of the character area extraction information held in the memory, The still image data is stored in the image storage unit 504, and the processing operation is terminated (S606).

  On the other hand, if the still image data includes a character region (Yes in S604), the character region extraction unit 503 refers to the character region information and determines that the character region included in the image is a vertically long region (the aspect ratio is vertical / horizontal). It is determined whether it is 2 to 1 or more (S608).

  When the character region is a vertically long region (Yes in S608), the character region extracting unit 503 adds “vertically long region” to the field of the character region extraction information held in the memory, and the still image data is stored in the image storage unit. Then, the processing operation is terminated (S610).

  If the character area is not a vertically long area (No in S608), the character area extracting unit 503 refers to the character area information, and the character area included in the image is a horizontally long area (the ratio of aspect ratio is 1 to 2 or more). It is determined whether or not there is (S612).

  When the character area is a horizontally long area (Yes in S612), the character area extracting unit 503 adds “horizontal area” to the field of the character area extraction information stored in the memory, The image is stored in the image storage unit 504, and the processing operation is terminated (S614).

  On the other hand, when the character area is not a horizontally long area (No in S612), that is, when the character area is nearly square, the character area extraction unit 503 displays “square” in the character area extraction information field stored in the memory. Region "is added, the still image data is stored in the image storage unit 504, and the processing operation is terminated (S616).

FIG. 24 is a flowchart showing the processing procedure of the scenario generation unit 505.
First, when the operation input unit 501 receives an operation of pressing a short film generation button from the user, the operation input unit 501 outputs a scenario generation instruction to the scenario generation unit 505.

  When the scenario generation unit 505 acquires the scenario generation instruction, the scenario generation unit 505 generates a screen (image selection screen) that prompts the user to select a plurality of images from the images stored in the image storage unit 504 and outputs the screen to the display unit 507. (S650).

  Here, when the operation input unit 501 receives an operation for selecting a still image to be reproduced from images displayed on the display unit 507 from the user (Yes in S652), the scenario generation unit 505 displays the selected image. Output information.

  Then, the scenario generation unit 505 checks the shooting time information of the selected image, and generates a scenario outer frame in which reproduction information of each image is arranged in order of shooting time to form a linked list (S654).

  Here, FIG. 25 shows a scenario outer frame generated when the operation input unit 501 accepts selection of five still images (image 1, image 2, image 3, image 4, and image 5) from the user. Show. It is assumed that the shooting times of the five still images are arranged in time series in the order of image 1, image 2, image 3, image 4, and image 5.

  As shown in FIG. 25, the scenario outer frame includes a scenario header 600, a scenario head list 601, a scenario second list 602, a scenario third list 603, a scenario fourth list 604, and a scenario fifth list 605 in a linked list. It is constituted by.

  At this time, the scenario header 600 is set with a scenario name and the number of images, and each list 601 to 605 of the scenario outer frame contains an image file name, a storage location and a character in the image storage unit 504 in the reproduction information of each image. Area extraction information is set.

The total playback time of the scenario header 600 and the visual effect information and playback time information of each list 601 to 605 of the scenario outer frame are blank at this stage.
Referring to FIG. 24 again, when the description of the processing procedure of the scenario generation unit 505 is continued, the scenario generation unit 505 refers to the generated scenario outer frame scenario first list 601 (S656) and is shown in the scenario first list 601. Determine the visual effects to assign to the image.

  At this time, the scenario generation unit 505 checks the character region extraction information set in the scenario outer frame to determine the visual effect, and the character region is extracted from the image (image 1) shown in the scenario head list 601. It is determined whether the image has been processed (S658).

  When the character region is an extracted image (Yes in S658), the scenario generation unit 505 further refers to the character region extraction information to determine whether the character region included in the image is a vertically long region. (S660).

  If the character region is a vertically long region (Yes in S660), the scenario generation unit 505 adds the Pan effect from the top to the bottom of the character region to the visual effect information in the scenario head list 601 (S662). In the first embodiment, the visual effect of moving the display area in the vertical direction is referred to as “tilt”. However, hereinafter, the visual effect is referred to as the “pan effect in the vertical direction”.

  On the other hand, if the character region is not a vertically long region (No in S660), the scenario generation unit 505 refers to the character region extraction information and determines whether the character region included in the image is a horizontally long region (S664). ).

  When the character area is a horizontally long area (Yes in S664), the scenario generation unit 505 adds the Pan effect from the left to the right for the character area to the visual effect information in the scenario head list 601 (S666).

  When the character region is not a horizontally long region (No in S664), that is, when the character region is a square region, the scenario generation unit 505 adds a zoom-in effect for the character region to the visual effect information in the scenario head list 601 ( S668).

  On the other hand, when the image shown in the scenario head list 601 is not an image from which a character area has been extracted (No in S668), the scenario generation unit 505 generates a random number using a random function and divides by 3 ( S670) The visual effect assigned to the image is determined by obtaining the remainder after division.

  That is, if the remainder is 1 (Yes in S672), the scenario generation unit 505 adds a fade-in effect to the image to the visual effect information in the scenario head list 601 (S674). If the remainder is 2 (Yes in S676). Then, the fade-out effect for the image is added to the visual effect information of the scenario head list 601 (S678). If the remainder is 0 (No in S676), a flag indicating that no visual effect is applied to the visual effect information in the scenario head list 601 is added.

  Furthermore, when adding the visual effect to be assigned to the visual effect information in the scenario head list 601, the scenario generation unit 505 also sets the image playback time and adds it to the playback time information in the scenario head list 601. Here, the playback time of an image to which the vertical or horizontal Pan effect on the character area or the zoom-in effect is applied is set to 3 seconds, and the playback time of other images is set to 2 seconds.

  Thereafter, the scenario generation unit 505 determines whether or not the list to which the visual effect information and the reproduction time information are added is the last list (S680), and if it is the last list (Yes in S680), generates the scenario. When the processing operation is completed and the list is not the last list (No in S680), the next list in the outer frame of the scenario is referred to (S682), and the processing operation for determining the visual effect on the image or the character area included in the image is repeated. .

  In this manner, when the scenario generation unit 505 repeats the processing operation for assigning visual effects from the first image (image 1) to the last image (image 5) in the reproduction order, the scenario shown in FIG. 26 is completed.

FIG. 26 shows a completed scenario.
In this completed scenario, information indicating the total playback time of 12 seconds is described in the scenario header 610, and visual effect information for assigning the fade-out effect is described in the scenario head list 611 in which the playback information of the image 1 is described. , Reproduction time information of 2 seconds of reproduction time is described. Similarly, the visual effect information and the reproduction time information assigned to each of the lists 611 to 615 in which the reproduction information of each image is described.

  That is, according to this completed scenario, images 1, 2, 3, 4, and 5 are reproduced in this order, with image 1 having a fade-out effect, image 2 having a zoom-in effect on a character area, image 3 Shows the Pan effect in the left-right direction with respect to the character area, and the image 4 shows that the visual effect of the fade-in effect is applied to each, and the image 5 shows that the visual effect is not applied. Yes. In addition, it is shown that the reproduction times of the images are 2 seconds, 3 seconds, 3 seconds, 2 seconds, and 2 seconds (total 12 seconds), respectively.

FIG. 27 is a flowchart showing a processing procedure in which the image reproduction unit 506 reproduces an image using the scenario generated in this way.
First, the image reproduction unit 506 receives the address information of the scenario generated by the scenario generation unit 505, refers to the scenario head list (S700), and reproduces the reproduction information of the image (image 1) stored in the scenario head list, That is, an image file name, a storage location in the image storage unit 504, character area extraction information, visual effect information, reproduction time information, and next list information are acquired.

  Next, the image playback unit 506 reads out still image data in a compressed format that matches the image file name from the storage location in the image storage unit 504 based on the acquired playback information (S702). When the image reproduction unit 506 decompresses the read still image data in the compressed format, the image reproduction unit 506 performs the visual management (fade out effect) indicated in the visual effect information while performing time management based on the acquired reproduction time information (number of frames). Then, the image is reproduced and displayed on the display unit 507 (S704).

When the image reproduction ends, the image reproduction unit 506 refers to the acquired next list information and determines whether or not the list is the last list in the scenario (S706).
If it is not the last list (No in S706), the image reproduction unit 506 refers to the next list of scenarios based on the next list information (S708) and acquires the reproduction information of the next image (image 2). Then, the above processing operation is repeated until the reproduction processing of the last image (image 5) is completed.

  On the other hand, if it is the last list in the scenario (Yes in S706), that is, if the next list information is “NULL”, the image reproduction unit 506 ends the image reproduction processing operation.

Here, the visual effect on the character area included in the image will be described with reference to FIG.
FIG. 28 is a diagram illustrating a first example of a screen display in the short film generating apparatus 500.

Here, a case will be described in which a Pan effect in the left-right direction is applied to a character region included in an image.
First, the image reproducing unit 506 of the short film generating apparatus 500 outputs and displays the first display image example 623 on the display unit 507. As shown in FIG. 28, the first display image example 623 includes a first character region 624 that is a horizontally long region, and a Pan effect in the left-right direction is applied to the first character region 624.

  Next, the image reproduction unit 506 of the short film generating apparatus 500 enlarges the first character area 624 included in the first display image example 623 and outputs and displays the second display image example 623a on the display unit 507. Then, the image reproduction unit 506 moves the display portion of the first character region 624 to the right, and sequentially outputs and displays the third display image example 623b and the fourth display image example 623c on the display unit 507. After displaying the fourth display image example 623c, the image playback unit 506 ends the playback of this image and proceeds to the playback processing of the next image.

  By applying such a visual effect to the character area included in the image, it is possible to emphasize the characteristics of the image (such as the shooting location), and to have a great impact on the user viewing the short film. .

  In the scenario generation unit 505, the visual effect applied to the character area included in the image is not limited to the Pan effect or the zoom-in effect in the vertical and horizontal directions. For example, a focus-in effect that focuses on the text area after displaying the entire image, or a focus-out effect that displays the entire image after focusing on the text area, or a text area that has multiple parts. The slide-in effect that slides the parts that make up the character area sequentially from the outside of the display screen, and the slide-out effect that slides each part off the display screen and erases them. Also good.

A case where the focus-out effect is applied to the character area included in the image will be described with reference to FIG.
FIG. 29 is a diagram illustrating a second example of the screen display in the short film generating apparatus 500.

  First, the image reproduction unit 506 of the short film generating apparatus 500 outputs and displays the second display image example 633a on the display unit 507. As shown in FIG. 29, the second display image example 633a includes a first character region 634 that is a horizontally long region, and a focus-out effect is applied to the first character region 634.

  Next, the image reproduction unit 506 of the short film generating apparatus 500 enlarges the focus applied to the first character region 634 and outputs and displays a first display image example 633 indicating the entire image on the display unit 507. Thereafter, the reproduction of this image is terminated, and the process proceeds to the reproduction process of the next image.

  If the arrow indicating the transition of the image display in FIG. 29 is reversed, that is, after the first display image example 633 is displayed, the first character area 634 is displayed in such a manner that the focus is sequentially reduced. If the display image example 633a is displayed, the focus-in effect is applied.

Further, a case where a slide-in effect is applied to a character area included in an image will be described with reference to FIG.
FIG. 30 is a diagram illustrating a third example of screen display in the short film generating apparatus 500.

  First, the image reproduction unit 506 of the short film generation apparatus 500 outputs and displays the second display image example 643a on the display unit 507. As shown in FIG. 30, the second display image example 643a includes a first character region 644 that is a horizontally long region, and a slide-in effect is applied to the first character region 644.

  Next, the image reproducing unit 506 of the short film generating apparatus 500 displays the part of the first character region 644 (the “Nara” portion of “Nara Station”) by sliding it from the outside of the display screen. In 507, the third display image example 643b is displayed and output.

  Subsequently, the image reproduction unit 506 of the short film generation device 500 displays the next part of the first character area 644 (the “good” portion of “Nara Station”) by sliding it from the outside of the display screen. The display unit 507 displays and outputs the fourth display image example 643c.

  Finally, the image reproduction unit 506 of the short film generating apparatus 500 displays the last part of the first character area 644 (the “station” portion of “Nara Station”) by sliding it from the outside of the display screen. The first display image example 643 is displayed on the display unit 507, and then the reproduction of this image is terminated, and the process proceeds to the reproduction process of the next image.

  As described above, according to the short film generating apparatus 500 according to the second embodiment, the character region extracting unit 503 extracts the character region included in the still image, and the scenario generating unit 505 converts the character region included in the still image. On the other hand, since a suitable visual effect is selected, a short film having a higher impact can be generated.

  In the second embodiment, the visual effect to be applied to the character area included in the image is selected based on the shape (aspect ratio) of the character area from the character area information. The visual effect to be applied to the character region included in the image may be selected based on the area or the vertical or horizontal length) or the color (character or background color). Specifically, the text area that is smaller than the predetermined area is enlarged, or the text area that is drawn in white on a blue background, such as the name of a road sign, or white in a green area. For example, it may be possible to apply an enlargement effect to a character area or the like.

  In addition, the character area is emphasized by changing the color of the character area included in the image, or by replacing it with another character prepared in advance. A short film suitable for the above may be generated.

  Further, in the second embodiment, the image captured by the digital camera 100 is directly taken into the apparatus via the image input unit 502. However, the image captured by the digital camera 100 is recorded on a memory card or the like. You may make it take in in an apparatus via a medium.

Furthermore, it goes without saying that the visual effect applied to the image or the character region included in the image in the second embodiment is merely an example, and is not limited to these visual effects.
(Embodiment 3)
Then, the short film production | generation apparatus which concerns on Embodiment 3 of this invention is demonstrated using FIGS.

  The configuration and processing operation of the short film generation device according to the third embodiment are substantially the same as those of the short film generation device 500 according to the second embodiment, but the configuration and processing operation in the scenario generation unit are different.

Hereinafter, this difference will be mainly described. Note that the same components as those in the second embodiment are denoted by the same reference numerals and description thereof is omitted.
FIG. 31 is a block diagram showing a configuration of a short film generating apparatus in the third embodiment.

  As shown in FIG. 31, the short film generation apparatus 510 includes an operation input unit 501, an image input unit 502, a character region extraction unit 503, an image storage unit 504, a scenario generation unit 508, an image reproduction unit 506, and a display unit 507. Prepare.

The operation input unit 501 receives an operation input from the user such as selection of a character area included in the still image.
The scenario generation unit 508 acquires information on the character region selected by the user (character region selection information) from the operation input unit 501, and selects a reproduction image, title image based on the character region information about the selected character region, and the like. , A playback order and a visual effect of the selected image, and a scenario that is playback information, and includes a selected image determination unit 509.

  Based on the shooting time information of the still image (key image) including the selected character area, the selected image determining unit 509 captures a still image shot within a predetermined time before and after the key image (for example, within 3 hours). Is a processing unit that selects an image to be reproduced, and is realized by a CPU and a memory.

The processing operation of each part of the short film generating apparatus 510 having such a configuration will be described in detail below, focusing on the scenario generating unit 508.
FIG. 32 is a flowchart showing the processing procedure of the scenario generation unit 508.

  First, when the operation input unit 501 receives an operation of pressing the short film generation button from the user, the operation input unit 501 outputs a scenario generation instruction to the scenario generation unit 508.

  When the scenario generation unit 508 acquires the scenario generation instruction, the scenario generation unit 508 generates a screen that prompts the user to select a character area (character area selection screen) and outputs the screen to the display unit 507 (S800). At this time, the scenario generation unit 508 includes, for example, by enclosing the character region portion of the image in which the character region is extracted by the character region extraction unit 503 among the images stored in the image storage unit 504 with a broken line. The character area selection screen is generated by specifying the character area portion. When there are a plurality of images from which character areas are extracted, the scenario generation unit 508 generates a character area selection screen including a next button, and the operation input unit 501 performs an operation of pressing the next button from the user. When accepted, the scenario generation unit 508 switches the character area selection screen to an image from which the next character area has been extracted.

  Here, when the operation input unit 501 receives an operation for selecting a character region from the character region selection screen displayed on the display unit 507 from the user (Yes in S802), the operation input unit 501 sets the character region selection information to the scenario generation unit 508. Output to.

  Subsequently, the scenario generation unit 508 reproduces all images shot within a predetermined time before and after the key image based on the shooting time information of the image (key image) including the character area in the selected image determination unit 509. The image to be selected is selected (S804).

Image selection in the selected image determination unit 509 will be described with reference to FIG.
FIG. 33 is a diagram schematically illustrating image selection performed by the selected image determination unit 509.
FIG. 33 shows a state in which captured images (images 7 to 15) are arranged in order of the capturing time.

Here, when the operation input unit 501 receives an operation for selecting a character region included in the image 10 from the user, the selected image determination unit 509 determines the image 10 as the key image 700.
Then, the selected image determination unit 509 selects all images captured within a predetermined time t before and after the capturing time of the key image 700 as images to be reproduced.

  In FIG. 33, since the images 8 and 13 are images captured within a predetermined time t before and after the capturing time of the key image, the selected image determination unit 509 displays the images 8 to 13. Is handled as the selected image 701.

  In this manner, when an image to be reproduced is selected in the selected image determination unit 509, the scenario generation unit 508 checks the shooting time information of the selected image, and arranges the reproduction information of each image in order of the shooting time and connects them. A list of scenario outlines is generated (S806 in FIG. 32).

Here, as shown in FIG. 33, FIG. 34 shows a scenario outer frame generated when six selected still images (image 8 to image 13) are selected by the selected image determination unit 509.
FIG. 34 is a diagram showing a scenario outer frame.

  As shown in FIG. 34, the scenario outline includes a scenario header 710, a scenario head list 711, a scenario second list 712, a scenario third list 713, a scenario fourth list 714, a scenario fifth list 715, and a scenario sixth list 716. The scenario seventh list 717 is formed as a linked list.

  At this time, the scenario header and the number of images are set in the scenario header 710, and a title image (for example, a single image such as black) stored in the image storage unit 504 in advance for the title of the short film is stored in the scenario head list 711. The image file name and the storage location in the image storage unit 504 are set. In addition, in each list 712 to 717 of the remaining scenario outer frames, the image file name and the storage location in the image storage unit 504 are set among the reproduction information of each image.

The total playback time of the scenario header 210 and the visual effect information and playback time information of each list 711 to 717 of the scenario outer frame are blank at this stage.
When the description of the processing procedure of the scenario generation unit 508 is continued with reference to FIG. 32 again, the scenario generation unit 508 refers to the scenario start list 711 of the generated scenario outer frame (S808) and is shown in the scenario start list 711. Determine the visual effects to assign to the image.

At this time, the scenario generation unit 508 checks the image file name and determines whether the image shown in the scenario head list 711 is a title image (S810).
In the case of a title image (Yes in S810), the scenario generation unit 508 uses the character embedding effect representing embedding the character region selected by the user in the center of the image of the title image as the visual effect information in the scenario head list 711. It adds (S812).

  In the case of an image other than the title image (No in S810), the scenario generation unit 508 generates a random number using a random function, divides it by 4 (S814), and obtains the remainder by dividing the image. Determine the visual effects to assign to.

  That is, if the remainder is 1 (Yes in S816), the scenario generation unit 508 adds a fade-in effect to the image to the visual effect information in the scenario head list 711 (S818), and if the remainder is 2 (Yes in S820). Then, the fade-out effect for the image is added to the visual effect information in the scenario head list 711 (S822). If the remainder is 0 (Yes in S824), the zoom-in effect on the central portion of the image is added to the visual effect information in the scenario head list 711 (S826). If the remainder is 3 (No in S824), the scenario head A flag indicating that no visual effect is applied to the visual effect information in the list 711 is added.

  Further, when adding the visual effect to be assigned to the scenario head list 711, the scenario generation unit 508 also sets image playback time information and adds it to the playback time information of the scenario head list 711. Here, the reproduction time of the image to which the zoom-in effect is applied is set to 3 seconds (45 frames), and the reproduction time of the other images is set to 2 seconds (30 frames).

  Thereafter, the scenario generation unit 508 determines whether or not the list to which the visual effect information and the reproduction time information are added is the last list (S828). If the list is the last list (Yes in S828), the scenario generation unit 508 determines whether the scenario is generated. When the processing operation is finished and the list is not the last list (No in S828), the next list in the scenario outer frame is referred to (S830), and the processing operation for determining the visual effect on the image is repeated.

  In this way, when the scenario generation unit 508 repeats the processing operation of assigning visual effects from the first image (title image) to the last image (image 13) in the playback order, the scenario shown in FIG. 35 is completed.

FIG. 35 is a diagram showing a completed scenario.
In this completed scenario, information indicating the total playback time of 16 seconds is described in the scenario header 720, and visual effect information to which the character embedding effect is assigned is described in the scenario head list 721 in which the playback information of the title image is described. Thus, reproduction time information of a reproduction time of 2 seconds is described. Similarly, in the lists 722 to 727 in which the reproduction information of each image is described, the visual effect information and the reproduction time information respectively assigned are described.

  That is, according to this completed scenario, the title image, the image 8, the image 9, the image 10, the image 11, the image 12, and the image 13 are reproduced in this order, and the title image has a character embedding effect and the image 8 has a fade-in effect. It is shown that the zoom-in effect is applied to the image 9, the fade-out effect is applied to the image 10, the zoom-in effect is applied to the image 12, and the visual effect is not applied to the images 11 and 13. Yes. In addition, it is shown that the reproduction times of the images are 2 seconds, 2 seconds, 3 seconds, 2 seconds, 2 seconds, 3 seconds, and 2 seconds (16 seconds in total), respectively.

  Thereafter, the image playback unit 506 receives the address information of the scenario generated by the scenario generation unit 508, refers to the scenario head list, and acquires playback information of the image (title image) stored in the scenario head list. . Then, the image reproduction unit 506 reads out and decompresses still image data in a compressed format from the image storage unit 504 based on the acquired reproduction information, and performs visual management while performing time management based on the reproduction time information (number of frames). The visual effect (character embedding effect) indicated in the effect information is applied, and the image is reproduced and output and displayed on the display unit 507.

  When the image reproduction ends, the image reproduction unit 506 refers to the next list of scenarios, acquires reproduction information of the next image (image 8), and the reproduction processing of the last image (image 13) ends. Until the above processing is repeated.

36 and 37 show examples of screen display of a short film reproduced by the short film generating apparatus 510 through such processing operations.
FIG. 36 is a diagram illustrating a first example of a screen display of the short film generating apparatus 510.

  In FIG. 36, a total of seven display image examples including a first display image example 730, which is a display example of a title image, and a first display image example 731 to a seventh display image example 736, which are display examples of images 8-13. The short film generation device 510 starts reproduction from the first display image example 730 in order.

  Here, the first display image example 730 is a title image reproduced according to the scenario head list 721 of FIG. 36, and is a character region of the image 10 that is a key image, that is, a horizontally long image included in the fourth display image example 733. The visual effect of embedding the first character area 733a ("Broadway") of the area is assigned, and the short film generating apparatus 510 displays the first display image example 730 for 2 seconds, and then displays the first display image example 731 in the second display image example 731. Switch the display.

  The second display image example 731 is an image 8 reproduced in accordance with the scenario second list 722 of FIG. 36, to which a fade-in effect is assigned, and the short film generating apparatus 510 starts from the first display image example 730. When the display is switched to the second display image example 731, the second display image example 731 is displayed by gradually making the image clearer.

  Subsequently, the short film generating apparatus 510 similarly reproduces the images 9 to 13 arranged in the reproduction order according to the scenario list, and sequentially displays the third display image example 732 to the seventh display image example 736. .

  Note that the character region extracted by the character region extraction unit 503 is not limited to kanji or alphabet, but may be hiragana characters or the like. The character area extraction unit 503 holds a pattern matching template for each language, so that it is possible to extract character areas that span multiple languages. The same applies to the object determination unit 300 in the first embodiment and the second embodiment.

FIG. 37 is a diagram illustrating a second example of the screen display of the short film generating apparatus 510.
FIG. 37 also shows a total of seven display image examples, a first display image example 740 that is a display example of a title image, and a first display image example 741 to a seventh display image example 746, as in FIG. It is shown that the short film generation apparatus 510 starts reproduction sequentially from the first display image example 740.

  Here, it is understood that the visual effect for embedding the first character region 743a ("Kamakura Daibutsu") in the vertically long region included in the fourth display image example 743 (key image) is assigned to the first display image example 740. . In this way, it is possible to embed a vertically long character area in a title image.

  Further, in the fifth display image example 744, it can be seen that the second character area 744b ("Omikuji") of Hiragana included therein is extracted. In this way, it is also possible to extract the character area of Hiragana.

  As described above, according to the short film generating apparatus 510 of the third embodiment, since the still image including the character area selected by the user is used as a key image, an image to be reproduced by the scenario generation unit 508 is automatically selected. Can save the trouble of selecting an image to be played.

  In addition, since the character region selected by the user is automatically embedded in the title image, it is possible to eliminate the need for the user to input characters used for the title using a keyboard or the like.

  In the third embodiment, the user selects a character area from the candidates via the operation input unit 501. However, the scenario generation unit 508 automatically determines the character area based on conditions such as the size of the character area. You may make it do. Here, the short film generating apparatus 510 automatically determines a character area to be used for a title image with a default value, and when the operation input unit 501 receives a character input for a title from the user, the title desired by the user is determined. Characters may be used for the title image.

  In addition, the selected image determining unit 509 selects the image captured within a predetermined time period before and after the key image selected by the user as a reference, but the selected image determining unit 509 selects the key image. If there is an interval of a predetermined time or more by checking the shooting time interval with the next image by arranging the images in order of shooting time instead of the time range from, the next image is divided into another group and grouped, An image belonging to the group including the key image may be selected as an image to be reproduced.

  Furthermore, the scenario generation unit 508 arranges the reproduction information of the title image in the scenario top list, but without the arrangement in the scenario, the image reproduction unit 506 adds the title image in which the character area is embedded before the scenario is executed. The output may be displayed on the display unit 507. As for the title image, the key image itself may be used as the title image instead of the black screen.

  Further, as in the second embodiment, the image captured by the digital camera 100 is directly taken into the apparatus via the image input unit 502. However, the image captured by the digital camera 100 is stored in a memory card or the like. You may make it take in in an apparatus via this recording medium.

  Further, as in the second embodiment, the visual effect applied to the image or the character area included in the image in the third embodiment is merely an example, and it goes without saying that the visual effect is not limited to these visual effects. Yes.

  The image display device according to the present invention is useful as an image display device including a display device having a number of pixels smaller than the number of pixels of the original image, such as a portable terminal such as a mobile phone or a PDA.

  The short film generation apparatus according to the present invention can be applied to a mobile phone with an image reproduction function, a personal computer, and the like, and is particularly suitable for a personal computer storing a large amount of still image data.

It is a figure which shows the external appearance structure of the mobile telephone which has a digital camera and an image display apparatus. 1 is a diagram schematically showing a block configuration of an image display device in Embodiment 1. FIG. It is a flowchart which shows the algorithm at the time of the original image input in an external interface part and an object determination part. It is a figure which shows the original image stored in an image holding part. It is a figure which shows the structural example of object information. It is a figure which shows the example of extraction of an object candidate. It is a figure which shows the structural example of an object priority setting table. It is a flowchart which shows the algorithm at the time of the thumbnail image generation in the thumbnail image generation process part of the image generation part for a display. FIG. 9A is a diagram showing an example in which thumbnail images are listed in the image display device according to the present invention, and FIG. 9B is an example in which thumbnail images are listed in the conventional image display device. FIG. It is a flowchart which shows the image generation process for a display. It is a flowchart which shows the subroutine of the display image generation process according to the kind of effect. FIG. 12A is a diagram showing an enlarged image, and FIG. 12B is a diagram showing an entire image. FIG. 13A is a diagram showing a display image before the viewpoint is moved, and FIG. 13B is a diagram showing a display image after the viewpoint is moved. FIG. 14A shows a display image when zoomed in, and FIG. 14B shows a display image when zoomed out. FIG. 15A is a diagram showing a display image before processing when the cut processing is performed, and FIG. 15B is a diagram showing a display image after processing. It is a flowchart which shows the process which a slide show part performs. It is a figure which shows the display image in the case of automatically displaying a magnified image and a reduced image alternately. On the screen, icons for "Back", "Whole", and "Save" are displayed for the enlarged image, and icons for "Back", "Enlarged", and "Save" are displayed for the reduced image. It is a figure which shows the example of a display at the time of displaying. It is a figure which shows the example of a display image in the case of changing a magnification gradually from a whole image to the enlarged image of a person's face, and changing smoothly. FIG. 20A and FIG. 20B are diagrams for explaining processing in the case of displaying while adding a predetermined visual effect. It is a figure which shows the example of a display which changes on a time axis at the time of performing the process shown by Fig.20 (a) and FIG.20 (b). It is a block diagram which shows the structure of the short film production | generation apparatus in Embodiment 2. FIG. FIG. 10 is a flowchart showing a processing procedure of a character area extraction unit in the second embodiment. FIG. 10 is a flowchart showing a processing procedure of a scenario generation unit in the second embodiment. It is a figure which shows the scenario outer frame in Embodiment 2. FIG. It is a figure which shows the completed scenario in Embodiment 2. FIG. FIG. 10 is a flowchart showing a processing procedure of an image reproduction unit in the second embodiment. It is a figure which shows the 1st example of the screen display by the short film production | generation apparatus of Embodiment 2. FIG. It is a figure which shows the 2nd example of the screen display by the short film production | generation apparatus of Embodiment 2. FIG. It is a figure which shows the 3rd example of the screen display by the short film production | generation apparatus of Embodiment 2. FIG. It is a block diagram which shows the structure of the short film production | generation apparatus in Embodiment 3. FIG. FIG. 10 is a flowchart showing a processing procedure of a scenario generation unit in the third embodiment. 10 is a diagram schematically showing image selection in Embodiment 3. FIG. FIG. 10 is a diagram showing a scenario outer frame in the third embodiment. FIG. 10 is a diagram illustrating a completed scenario in the third embodiment. It is a figure which shows the 1st example of the screen display by the short film production | generation apparatus of Embodiment 3. FIG. It is a figure which shows the 2nd example of the screen display by the short film production | generation apparatus of Embodiment 3. FIG. It is a block diagram which shows the structure of the conventional short film production | generation apparatus. It is a flowchart which shows the process sequence of the conventional scenario production | generation part. It is a figure which shows the conventional scenario outer frame. It is a figure which shows the 1st example of the completed scenario in the past. It is a figure which shows the 2nd example of the completed scenario in the past. It is a figure which shows the example of a screen display by the conventional short film production | generation apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image display apparatus 10 External interface part 20 Memory part 30 Object determination part 40 Display image generation part 50 Display 100,910 Digital camera 110 Image pick-up part 120 Image input part 130 User IF part 210 Image holding part 250 Image holding part for display Unit 310 object candidate extraction unit 320 object candidate determination unit 330 object priority determination unit 410 thumbnail image generation processing unit 420 enlargement / reduction processing unit 430 viewpoint movement processing unit 440 zoom in / out processing unit 450 cut out processing unit 460 slide show 500, 510, 900 Short film generation apparatus 501, 901 Operation input unit 502, 902 Image input unit 503 Character area extraction unit 504, 903 Image storage unit 505, 508, 904 Scenario generation unit 506, 90 Image playback unit 507, 906 Display unit 509 Selected image determination unit 600, 610, 710, 720, 920, 930, 940 Scenario header 601, 611, 711, 721, 921, 931, 941 Scenario head list 602, 612, 712, 722, 922, 932, 942 Scenario second list 603, 613, 713, 723, 923, 933, 943 Scenario third list 604, 614, 714, 724, 924, 934, 944 Scenario fourth list 605, 615, 715 , 725, 925, 935, 945 Scenario 5th list 623, 633, 643, 730, 740 First display image example 623a, 633a, 643a, 731, 741 Second display image example 623b, 643b, 732, 742 Third display Image example 623c, 6 3c, 733, 743 Fourth display image example 624, 634, 644, 731a to 736a, 741a to 746a First character area 700 Key image 701 Selected image 716, 726, 946 Scenario sixth list 717, 727 Scenario seventh list 734 744, fifth display image example 744b second character area 735, 745 sixth display image example 736, 746 seventh display image example 950 display image example 951 first display area 952 display transition line 953 second display area

Claims (8)

An image display device comprising display means for displaying an image,
Original image holding means for holding an original image formed with a number of pixels larger than the number of displayable pixels in the display means;
Object determination that determines whether or not an object is included in the original image held in the original image holding means, and generates object information on each object when at least one or more objects are included Means,
Based on the original image designated by the user and at least one or more pieces of the object information corresponding to the original image, at least a part of the object area that is an area including the object in the original image is converted into the original image. A display image generating unit that extracts the image and generates a display image in accordance with the number of displayable pixels in the display unit ;
The object determination means includes
Based on the shape of the object, the type of the object is determined from among a plurality of types of objects including a plurality of types of persons, animals, plants, movables, and real estate. An object type determination unit embedded in the object information;
An object priority setting unit that embeds the priority corresponding to the determined type of the object in the object information;
The display image generating means generates an object region corresponding to the object information indicating that the priority is high among at least one of the object information based on the priority embedded in the object information. An image display device characterized by being extracted from an image. The object determination means includes
An object extraction unit that extracts the position and size of an object included in the original image and embeds the extracted position and size of the object in the object information;
The image display apparatus according to claim 1, wherein the display image generation unit determines a portion to be extracted from the original image based on a position and a size of an object embedded in the object information. The display image generation means further includes:
Based on the object information, thumbnail image generation for extracting at least a part of an object area, which is an area including an object in the original image, from the original image and generating a thumbnail image having a predetermined number of pixels The image display apparatus according to claim 1, further comprising: means. The original image holding means holds a plurality of original images,
4. The image display device according to claim 3 , wherein the display image generating means causes the display means to display a list of thumbnail images of the original images generated by the thumbnail image generating means. The image display device according to claim 4, wherein the predetermined number of pixels is the number of pixels assigned to display one thumbnail image in the display unit. The display image generating means determines a portion to be extracted from the original image so that the number of pixels extracted from the original image is equal to the number of pixels that can be displayed on the display means. Item 6. The image display device according to any one of Items 1 to 5 . An image display method in an image display device comprising display means for displaying an image,
The image display device includes:
An original image holding means for holding an original image formed with a larger number of pixels than the displayable number of pixels in the display means;
Object determination that determines whether or not an object is included in the original image held in the original image holding means, and generates object information about each object when at least one or more objects are included Steps,
Based on the original image designated by the user and at least one or more pieces of the object information corresponding to the original image, at least a part of the object area that is an area including the object in the original image is converted to the original image. extracted from the image, seen including a display image generating step of generating a display image matched to the number of pixels that can be displayed on the display means,
The object determination step includes
Based on the shape of the object, from among a plurality of object types including a plurality of types of person, animal, plant, movable property, and real estate, determine the type of one object, and determine the determined object type An object type determination step embedded in the object information;
An object priority setting step for further embedding the priority corresponding to the determined type of the object in the object information,
In the display image generation step, based on the priority embedded in the object information, an object region corresponding to the object information indicating that the priority is high among at least one or more of the object information. An image display method characterized by extracting from an image. A program for an image display device comprising display means for displaying an image,
The image display device includes:
An original image holding means for holding an original image formed with a larger number of pixels than the displayable number of pixels in the display means;
Object determination that determines whether or not an object is included in the original image held in the original image holding means, and generates object information about each object when at least one or more objects are included Steps,
Based on the original image designated by the user and at least one or more pieces of the object information corresponding to the original image, at least a part of the object area that is an area including the object in the original image is converted to the original image. Extracting from the image, causing the computer to execute a display image generation step for generating a display image in accordance with the number of displayable pixels in the display means ,
The object determination step includes
Based on the shape of the object, from among a plurality of object types including a plurality of types of person, animal, plant, movable property, and real estate, determine the type of one object, and determine the determined object type An object type determination step embedded in the object information;
An object priority setting step for further embedding the priority corresponding to the determined type of the object in the object information,
In the display image generation step, based on the priority embedded in the object information, an object region corresponding to the object information indicating that the priority is high among at least one or more of the object information. A program characterized by extracting from an image .

Images