JP2015130546A - Image processing system, method, program, and data format - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate image data which expresses a series of time flow from plural images, or to reproduce image data which expresses a series of time flow.SOLUTION: An image processing system receives (11) plural images, acquires (12) images related with the images from an image data base (14) according to need, sequences (12) the images, determines (16) whether or not the images can smoothly be interconnected, selects (19) only the images which are important and can smoothly be interconnected in order, and stores (23) information on "a series of images" interconnected smoothly and a reproduction sequence. Then, the system reproduces (25) "a series of images" in accordance with the reproduction sequence.

Description

The present technology relates to an image processing method, apparatus, program, and format that enable a plurality of image data to be displayed as a series of images.

Recently, with the widespread use of digital cameras, for example, it has become common to shoot a large number of still images or shoot a moving image and record it on a single trip. However, while it is easy to shoot, the situation is that it can never be replayed because it can be played back at any time. This is because there are an enormous number of sheets and it is troublesome to see.

Therefore, the applicant of the present application proposes to collect a huge number of images as one image. Since they are grouped together, it is only necessary to see that one data, and there is no trouble of seeing a huge number of sheets. Therefore, for example, it becomes possible to look back on travel records easily.

Conventionally, there have been the following methods or apparatuses for collecting a plurality of images into one. That is, there is a method in which a plurality of images are displayed on a display unit (rectangular region), and a photographer moves an image position to create an image in which a plurality of images are inserted into one rectangular region. (Patent Document 1). In addition, a method of creating a photo collage from a plurality of images according to a certain rule is also disclosed (Patent Document 2). There is also a method in which a plurality of still images that are easy to connect are connected to generate one huge image (see Non-Patent Document 1).

JP 2007-104336 A

JP 2000-215212 A

Josef Sivic, Biliana Kaneva, Antonio Torralba, Shai Avidan, William Freeman, “Creating and Exploring a Large Photorealistic Virtual Space,” First IEEE Workshop on Internet Vision, CVPR 2008

By the way, in Patent Document 1, it is troublesome because it is necessary to manually edit when collecting the image data.

Further, in Patent Document 1 and Patent Document 2, there is no concept of connecting (merging) image data, and a plurality of images are simply arranged, and one finally obtained image data is: I cannot feel the continuity that flows with time. In other words, only one photo at a time seemed to overlap, and one continuous flow of time could not be expressed.

Non-Patent Document 1 shows a method of connecting (merging) image data and combining them into one image, but does not take into account the concept of time or position of shooting. For example, consider creating one continuous image from a plurality of images taken during a trip by the method of Non-Patent Document 1. Even if the viewer appreciates the created image, it is not known in what order the tourists have visited the sightseeing spot. This is because images that change with time are not presented. Note that the images are connected smoothly and you can feel the continuity between images, but the adjacent images are not necessarily images taken continuously in time, so the continuity between images is It does not represent the continuity of time.

In any of Patent Document 1, Patent Document 2, and Non-Patent Document 1, a still image is a premise process, and what kind of processing should be performed on a moving image is not disclosed.

In any of Patent Document 1, Patent Document 2, and Non-Patent Document 1, it is not considered in what order the collected images should be displayed.

To reiterate, the prior art cannot generate image data expressing one continuous time flow from a plurality of images. In addition, there is no data format necessary for reproduction of “image data expressing one continuous flow of time”.

The present technology has been made in view of such a situation, and makes it possible to create or reproduce image data for display that allows the user to feel the continuity that flows with time.

An image processing apparatus according to an aspect of the present technology is an image processing apparatus that processes an input image data group and outputs display image data. The first image data group is input to the first image data group. A first input unit having the input image data group as an input image, an image alignment unit for sorting the image data in the input image data group, and an importance for determining the importance of each image data in the input image data group The degree determination unit, the connection possibility determination unit for determining the connection possibility between the image data in the input image data group, and the image data in the input image data group are selected and connected by the connection determination unit The display image data is created by connecting the image data determined to be important by the importance determination unit in the order sorted by the image alignment unit. Connection and the above connection And an output unit for outputting the display image data created in parts.

In the image processing apparatus, a database unit including a plurality of image data and image data related to the first image data group input in the first input unit are extracted from the database unit, and the input image A second input unit to be added to the data group can be further provided.

In the image processing apparatus, among the two pieces of image data connected by the connection unit, image data having a lower rank in the order sorted by the image alignment unit is referred to as first image data, and the other, that is, If the image data of the order that is not young is called the second image data, the connection between the image data of the input image data group in the connection unit is
(Case 1) When the first image data is a still image and the second image data is a moving image, the first image data that is a still image and the second image that is a moving image. Make a connection to the first frame of data,
(Case 2) When the first image data is a moving image and the second image data is a still image, the last frame of the first image data that is a moving image and the above-mentioned image that is a still image Make a connection with the second image data,
(Case 3) When the first image data is a moving image and the second image data is a moving image, the last frame of the first image data that is a moving image and the second image that is a moving image. It is possible to connect to the first frame of the image data.

The display image data is generated in the order in which the display image data is displayed over time in the order connected by the connection unit, that is, in ascending order in the order sorted by the image alignment unit. A generation unit is further provided, and the output unit can output the reproduction order information together with the display image data.

A data format according to an aspect of the present technology includes a plurality of image data, a positional relationship between the image data, and data indicating an order to be displayed.

An image processing method or program according to an aspect of the present technology is an image processing method or program that processes an input image data group and outputs display image data. The first image data group is input and the first image data group is input. A first input step in which the input image data group is the input image data group, an image alignment step of sorting the image data in the input image data group, and the importance of the image data in the input image data group An importance level determining step, a connection possibility determination step for determining the connection possibility between the image data in the input image data group, and image data in the input image data group are selected and the connection possibility determination is performed. Image data determined to be connectable in the step and determined to be important in the importance determination step are sorted in the image alignment step. By going to connect in turn, comprises a connecting step of generating image data for the display, and an output step of outputting the display image data created in the connection step.

In one aspect of the present technology, in image processing for processing an input image data group and outputting display image data, the first image data group is input, and the first image data group is input to the input image data group. And sorting the image data in the input image data group, determining the importance of the image data in the input image data group, and determining the possibility of connection between the image data in the input image data group. The image data in the input image data group is selected, the image data determined to be connectable and the image data determined to be important are connected in the sorted order, Display image data is created, and the created display image data is output.

An image processing apparatus according to an aspect of the present technology is an image processing apparatus that displays a first still image and a first moving image, and the first still image and the first frame of the first moving image intersect. After the first still image is mainly displayed, the display area is moved with time to display the first frame of the first moving image, and after the next time. The second and subsequent frames of the first moving image are sequentially displayed or arranged so that the last frame of the first moving image and the first still image intersect each other. Display sequentially from the first frame of the first video to the last frame of the first video, and at the next time, mainly display the last frame of the first video, Move the display area over time So it comprises mainly or going to display the first still image, at least it performs display either a.

An image processing method or program according to one aspect of the present technology is an image processing method or program for displaying a first still image and a first moving image, wherein the first still image and the first moving image of the first moving image are displayed. Arrange the frames so that they intersect with each other, display the first still image mainly, then move the display area over time to display the first frame of the first moving image mainly, and then After that time, the second and subsequent frames of the first moving image are sequentially displayed, or the last frame of the first moving image and the first still image intersect. And sequentially displaying from the first frame of the first moving image to the second frame from the end of the first moving image, and at the next time, mainly the last frame of the first moving image. To display To, by moving the display area with time, mainly including the first or will display the still image, at least it performs display processing step either in the.

In one aspect of the present technology, in the image processing for displaying the first still image and the first moving image, the first still image and the first frame of the first moving image are arranged so as to intersect each other. After displaying the first still image mainly, the display area is moved with time to display mainly the first frame of the first moving image, and after the next time, the first moving image is displayed. The second and subsequent frames are sequentially displayed, or arranged so that the last frame of the first moving image and the first still image intersect each other, and the first moving image From the last frame to the second frame from the end of the first moving image sequentially, and at the next time, the last frame of the first moving image is mainly displayed, and the display area is also displayed with time. Move the main To perform the first or will display the still image, at least it performs display processing either of.

According to one aspect of the present technology, it is possible to create and reproduce image data for display that can feel continuity flowing with time.

It is a figure which shows the general example of the image image | photographed with the digital camera. It is explanatory drawing regarding the smooth connection between the images in this invention. It is explanatory drawing regarding the smooth connection between the images in this invention. It is explanatory drawing regarding the smooth connection between the images in this invention. It is explanatory drawing regarding the smooth connection between the images in this invention. It is a figure which shows the example of the "continuous image" produced by this invention. It is explanatory drawing at the time of reproducing | regenerating the "continuous image" produced by this invention. It is explanatory drawing regarding the "continuous image" created by this invention, and its reproduction | regeneration. It is explanatory drawing regarding the "continuous image" created by this invention, and its reproduction | regeneration. It is explanatory drawing regarding the "continuous image" created by this invention, and its reproduction | regeneration. It is explanatory drawing at the time of reproducing | regenerating the "continuous image" produced by this invention. It is explanatory drawing at the time of reproducing | regenerating the "continuous image" produced by this invention. It is explanatory drawing of the positional relationship between the images by the smooth connection in this invention. It is a flowchart explaining the process for performing the smooth connection of two still images in this invention. It is a block diagram which shows the structure of the 1st Example of this invention. It is a flowchart explaining the process performed in the connection possibility judgment part in 1st Example of this invention which judges the possibility of a connection possibility. 5 is a flowchart for explaining processing for selecting a material (image) for constituting “continuous images”, which is performed by the directed graph creation unit, the route search unit, and the image selection unit according to the first embodiment of the present invention. is there. It is a figure which shows an example of the directed graph created in the directed graph creation part in 1st Example of this invention. It is a flowchart explaining the process which calculates | requires the positional relationship between adjacent images, and the process which produces a mask performed in the image connection part in 1st Example of this invention. It is a flowchart explaining the process which determines the rendering position of each image performed in the rendering information preparation part in 1st Example of this invention. It is a flowchart explaining the process which determines the rendering position of each image performed in the rendering information preparation part in 1st Example of this invention. It is a flowchart explaining the process which determines the position displayed at the time of reproduction | regeneration performed by the display information preparation part in 1st Example of this invention. It is a flowchart explaining the process which determines the position displayed at the time of reproduction | regeneration performed by the display information preparation part in 1st Example of this invention. It is explanatory drawing of the position displayed at the time of reproduction | regeneration in 1st Example of this invention. It is explanatory drawing of the position displayed at the time of reproduction | regeneration in 1st Example of this invention. It is a flowchart explaining the process of reproduction | regeneration performed in the reproducing | regenerating part in 1st Example of this invention. It is a figure which shows the general example of the image image | photographed with the digital camera. It is a block diagram which shows the structure of the 2nd Example of this invention. It is explanatory drawing about the trimming performed in the left trimming part in the 2nd Example of this invention, and the right trimming part. It is explanatory drawing regarding the "continuous image" produced by the 2nd Example of this invention, and its reproduction | regeneration. A block diagram illustrating an example of the configuration of a computer device used to generate and play a "continuous image"

First, a typical example to which the present invention is applied will be described. The photographer goes on a trip, travels around a plurality of sightseeing spots while taking photographs and videos with a digital camera, and returns home. After returning home, images (still images, moving images, and cinema graphs) recorded in the memory in the digital camera are taken into a PC (personal computer). Then, by executing a program embodying the present invention on a PC, a “continuous image (picture scroll)” that flows with time is created. The cinema graph will be described later. In the present invention, an image refers to a still image, a moving image, or a cinema graph. By displaying this “continuous image (picture scroll)” on the PC, you can indulge in memories of your trip. The “continuous image (picture scroll)” is not a general term but one product in the present invention, and will be described in detail later.

A cinema graph is a kind of moving image composed of a plurality of frames (still images), and has the following characteristics. That is, the cinema graph is a moving image that does not feel uncomfortable even if the movie is played back to the last frame and then played back (loop playback). For example, after shooting a normal video, “Kwatra et al.,“ Graphcut Textures: Image
and Video Synthesis Using Graph Cuts, "Proceedings
Cinemagraphs can be created by performing the processing described in Chapter 7 of “of SIGGRAPH 2003”. Sony Mobile Communications Co., Ltd. distributes "Xperia (TM) Motion Snap" as an application for smartphones. With this application, you can automatically create a cinema graph when shooting. The cinema graph can express that the subject keeps moving indefinitely.

In the present invention, a cinema graph can also be used as a material. In the present invention, a moving image that can be loop-reproduced is referred to as a cinema graph, and a moving image that cannot be loop-reproduced (moving image that cannot be smoothly shifted from the last frame to the first frame) is referred to as a “moving image”.

Next, the “continuous image (picture scroll)” which is a product of the present invention will be described. For example, it is assumed that a still image A0, a still image A1, and a still image A2 shown in FIG. The order of photographing (order of photographing time) is A0, A1, A2. That is, it is assumed that the traveler (photographer) has photographed the automobile 100 at the beginning of the trip, then photographed the building 101, and finally photographed the person 102. After the trip is over, the traveler (photographer) wants to recall that while he was traveling, he saw the car 100 and then saw the building 101 and met the person 102. Until now, it was only possible to display three still images (A0, A1, A2) in order. In other words, I couldn't feel how each scene changed and I couldn't feel the continuity of travel. The present invention provides a video expression for “feeling the continuity of travel”.

Therefore, first, consider that two still images A0 and A1 are overlapped as shown in FIG. 2 and connected by a dividing line 200 indicated by a thick line in the drawing. The dividing line 200 is a boundary line at which two still images can be smoothly connected. For example, two images may be smoothly connected by the method described in “Non-Patent Document 1”. In A1, a region on the right side of the dividing line (in FIG. 3, a region 300 indicated by a diagonal line surrounded by a thick line including the dividing line 200) is set as a mask image of A1. In other words, the mask image of A1 is a region including “the center of the image of A1” among which the A1 is divided into two by the dividing line 200.

Next, it is considered that two still images A1 and A2 are overlapped as shown in FIG. 4 and connected by a dividing line 400 indicated by a thick line in the drawing. The dividing line 400 is a boundary line that can smoothly connect two still images. In A2, a region on the lower right side of the dividing line (in FIG. 5, a region 500 indicated by a diagonal line surrounded by a thick line including the dividing line 400) is set as a mask image of A2. In other words, the mask image of A2 is a region including “the center of the image of A2” among the divided portions, although A2 is divided into two by the dividing line 400.

After the mask image is determined, A0 is written on the canvas (600 in FIG. 6). Then, A1 is written only in the mask image portion of A1. At this time, a part of A0 written on the canvas 600 is overwritten by A1. Finally, A2 is written only in the portion of the mask image of A2. At this time, a part of A1 written on the canvas 600 is overwritten by A2. By doing so, an image as indicated by reference numeral 601 (referred to as a “continuous image (picture scroll)” in the present invention) is completed on the canvas 600.

The “continuous image (picture scroll) 601” on the campus 600 created in this way is reproduced (displayed) as follows. That is, in FIG. 7, the display frame area indicated by 700 is displayed first, the display frame area indicated by 701 is displayed at the next time, and the display frame area indicated by 702 is displayed at the next time. I will let you. Thereafter, the display frame is sequentially moved to the right until the display frame area 703 is reached. Then, at the next time, the display frame area indicated by 704 is displayed. Further, at the next time, a display frame area indicated by 705 is displayed. Thereafter, the display frame is sequentially displayed while moving to the lower right until the display frame area indicated by 706 is reached. Finally, the display frame area indicated by 706 is displayed, and a series of displays is completed.

By displaying in this way, the traveler (photographer) recalls in order, from the display order, “I saw the car 100 and then saw the building 101 and met the person 102 during the trip”. In addition, each still image is connected smoothly, so you can “feel the continuity of travel”.

In the above, the word “partition line” is used. As shown in FIG. 3 and FIG. 5, this line means a line for dividing a still image (A1 or A2) into a part for displaying and a part for not displaying the still image (A1 or A2). The word was used. Since still images are connected with this line as a boundary, they may be expressed as “connection lines”.

Further, although three still images (A0, A1, A2) have been described as an example here, it is assumed that usually several tens to hundreds of images are connected.

Now, in order to create a “continuous image (picture scroll)”, in the above example, it has been described that the connection is always made in the right or lower right direction. However, the present invention can be applied to other directions. An example thereof will be described with reference to FIG. As will be described later, since there are overlapping portions and it becomes complicated to illustrate in two dimensions, the images are arranged in a three-dimensional space and shown in an overhead view. That is, the X-axis and the Y-axis represent two-dimensional data of a still image, and the overlapping portion is illustrated by changing the position of the Z-axis.

It is assumed that a still image B0, a still image B1, a still image B2, a still image B3, a still image B4, and a still image B5 are taken by a traveler (photographer). The order of photographing (order of photographing time) is B0, B1, B2, B3, B4, and B5.

As shown in FIG. 8, B1 is connected to the lower side with respect to B0. In the dividing line 800, B0 and B1 are smoothly connected.

B2 is connected to the right side with respect to B1. In the dividing line 801, B1 and B2 are smoothly connected.

B3 is connected to the lower right side with respect to B2. In the dividing line 802, B2 and B3 are smoothly connected.

B4 is connected to the left side with respect to B3. In the dividing line 803, B3 and B4 are smoothly connected. In the figure, “B2 on the upper left of B3” and “B4 on the left of B3” overlap each other when illustrated in a two-dimensional plane (XY plane), so the Z-axis value is shifted. It is shown. Although two B3 are shown in the figure, they are exactly the same. Also, two display frames 823, which will be described later, are shown, which are exactly the same.

B5 is connected to the left side with respect to B4. In the dividing line 804, B4 and B5 are smoothly connected.

The “continuous images (picture scrolls) 810” connected by the dividing lines in this way are reproduced (displayed) as follows.

That is, in FIG. 8, the display frame area 820 is displayed first, and the display frame is displayed while moving the display frame in the direction of the arrow 830 as the time advances.

Then, after reaching the area of the display frame indicated by 821, next, the display frame is displayed while moving in the direction of the arrow 831.

Then, after reaching the area of the display frame indicated by 822, next, the display frame is displayed while being moved in the direction of the arrow 832.

Then, after reaching the area of the display frame indicated by 823, next, the display frame is displayed while moving in the direction of the arrow 833.

Then, after reaching the area of the display frame indicated by 824, next, the display frame is displayed while moving in the direction of the arrow 834.

Finally, the display frame area indicated by 825 is displayed, and the series of displays is terminated.

In the above description, the case where only a still image is used as a material for creating a “continuous image (picture scroll)” has been described, but a moving image or a cinema graph may be used.

An example of a moving image will be described with reference to FIG. It is assumed that a still image C0, a still image C1, a moving image C2, a moving image C3, a still image C4, and a still image C5 are photographed by a traveler (photographer). The order of photographing (order of photographing time) is C0, C1, C2, C3, C4, and C5. The moving image C2 includes four frames (C20, C21, C22, and C23). The moving picture C3 is composed of three frames (C30, C31, C32). In general, since a moving image has several tens of seconds to one minute, the number of frames constituting the moving image is several hundred to several thousand. Here, in order to simplify the explanation, it is assumed that a moving image is composed of four and three frames.

As shown in FIG. 9, C1 is connected to the upper right side with respect to C0. In the dividing line 900, C0 and C1 are smoothly connected.

For C1, C20 (the first frame of the moving image C2) is connected to the right side. In the dividing line 901, C1 and C20 are smoothly connected.

C30 (first frame of moving image C3) is connected to the lower right side with respect to C23 (last frame of moving image C2). In the dividing line 902, C23 and C30 are smoothly connected.

For C32 (the last frame of the moving picture C3), C4 is connected to the left side. In the dividing line 903, C32 and C4 are smoothly connected.

C5 is connected to the left side with respect to C4. In the dividing line 904, C4 and C5 are smoothly connected.

The “continuous images (picture scrolls) 910” connected by the respective dividing lines in this way are reproduced (displayed) as follows.

That is, in FIG. 9, the display frame area indicated by 920 is displayed first, and the display frame is displayed while moving the display frame in the direction of the arrow 940 as the time advances.

Then, after reaching the display frame area 921, the display frame is displayed while moving the display frame in the direction of the arrow 941. Then, a display frame area indicated by 922 is displayed.

The still images used so far are C0, C1, and C20. C20 is data of one frame in the moving image and can be said to be a still image. Similarly, each frame of the moving image can be said to be a still image.

At the next time, the display frame area indicated by 923 is displayed. The data displayed here is a part of the still image C21.

At the next time, the display frame area indicated by 924 is displayed. The data displayed here is a part of the still image C22.

At the next time, the display frame area indicated by 925 is displayed. The data displayed here is a part of the still image C23.

After the display frame area indicated by 925 is displayed, the display frame is displayed while moving the display frame in the direction of the arrow 942 as the time advances. Then, a display frame area indicated by 926 is displayed. The still images used here are C23 and C30.

At the next time, the display frame area indicated by 927 is displayed. The data displayed here is a part of the still image C31.

At the next time, the display frame area 928 is displayed. The data displayed here is a part of the still image C32.

After the display frame area indicated by 928 is displayed, the display frame is displayed while moving the display frame in the direction of the arrow 943 as the time advances. The still images used here are C32 and C4.

Then, after reaching the area of the display frame indicated by 929, next, the display frame is displayed while moving in the direction of the arrow 944.

Finally, the display frame area indicated by 930 is displayed, and the series of displays is terminated.

A traveler (photographer) who appreciates such a series of displays recalls a series of events and can feel the continuity of events because the still images are smoothly connected. Furthermore, in this series of displays, the moving image C2 and the moving image C3 are reproduced as moving images. That is, C20, C21, C22, and C23 are continuously displayed, and C30, C31, and C32 are also continuously displayed. Strictly speaking, only the area within the display frame (a part of the moving image) is reproduced in the moving image.

Further, in the process of displaying while moving the display frame in the direction of the arrow 941, it is possible to continuously shift from the still image C1 to the moving image C2 (more precisely, the first frame C20 of the moving image). Similarly, in the process of displaying while moving the display frame in the direction of the arrow 943, the moving image C3 (more precisely, the last frame C32 of the moving image) can be continuously transferred to the still image C4. In this way, even if still images and moving images are mixed, it is possible to display such that the continuity of events is felt.

An example of a cinema graph will be described with reference to FIG. It is assumed that a still image D0, a still image D1, a cinema graph D2, a still image D3, and a still image D4 are taken by a traveler (photographer). The order of photographing (order of photographing time) is D0, D1, D2, D3, and D4. The cinema graph D2 includes four frames (D20, D21, D22, D23). In general, since a cinema graph has several seconds, the number of frames constituting the cinema graph is several tens to several hundreds. Here, for simplicity of explanation, it is assumed that a cinema graph is composed of four frames.

Since D2 is a cinema graph, it is a moving image that does not feel uncomfortable even if loop playback is continued endlessly as D20, D21, D22, D23, D20, D21, D22, D23, D20,.

As shown in FIG. 10, D1 is connected to the right side with respect to D0. In the dividing line 1000, D0 and D1 are smoothly connected.

D20 (first frame of cinema graph D2) is connected to the lower right side with respect to D1. In the dividing line 1001, D1 and D20 are smoothly connected.

D3 is connected to the upper right side with respect to D20 (first frame of the cinema graph D2). In the dividing line 1002, D20 and D3 are smoothly connected.

In the figure, the positions of the dividing line 1001 and the dividing line 1002 are indicated on D21, D22, and D23 as well as on D20.

D4 is connected to the lower right side with respect to D3. In the dividing line 1003, D3 and D4 are smoothly connected.

The “continuous image (picture scroll) 1010” connected at each dividing line in this way is reproduced (displayed) as follows.

In the case of a “continuous image (picture scroll)” including a cinema graph, the concept of two times is introduced when reproducing (displaying). One is the absolute time and the other is the playback time. As will be described later, during playback, the viewer chooses to continue playback (generally called play mode) or pause (generally called pause mode). I can do it. Absolute time advances in both play mode and pause mode. On the other hand, the playback time advances only in the play mode, and in the pause mode, the value of the time continues to be held and does not advance.

In FIG. 10, first, the display frame area indicated by 1020 is displayed. In the play mode, as the playback time advances, the display frame is moved in the direction of the arrow 1030 and displayed. Go.

Then, after reaching the display frame area indicated by 1021, in the play mode, the display frame is moved and displayed in the direction of the arrow 1031 as the playback time advances. That is, the display frame indicated by 1022 and the display frame indicated by 1023 are sequentially displayed.

Then, after reaching the display frame area indicated by 1023, in the play mode, the display frame is displayed while moving in the direction of the arrow 1032 as the playback time advances.

Then, after reaching the display frame area indicated by 1024, in the play mode, as the playback time advances, the display frame is moved in the direction of the arrow 1033 and displayed.

Finally, the display frame area indicated by 1025 is displayed, and the series of displays is terminated.

In the pause mode, the area of the display frame displayed immediately before is kept displayed. When the mode is changed from the pause mode to the play mode, the display frame is moved from the point in the direction of the arrow and displayed.

Now, the cinema graph D2 used for display is specifically one of the still image D20, the still image D21, the still image D22, or the still image D23. Which of D20, D21, D22, and D23 to use depends on the absolute time. Each of D20, D21, D22, and D23 is one frame of data in the cinema graph, and can be said to be a still image. Specifically, a frame corresponding to the remainder obtained by dividing the absolute time by the number of frames of the cinema graph (in this example, “4”) is used. That is, when the absolute time is a multiple of 4, D20 is used, when the remainder when the absolute time is divided by 4, D21 is used, and when the absolute time is divided by 4, the remainder is 2 D22 is used when it is, and D23 is used when the remainder when the absolute time is divided by 4 is 3.

For example, let us consider a case where the display frame area 1022 in FIG. 10 is displayed and the viewer selects the pause mode. Since the pause mode is selected, the reproduction time is maintained at the current value, and the display frame area indicated by 1022 is continuously displayed. That is, in the display frame indicated by 1022, for the upper left part of the dividing line 1001, the corresponding part of the still image D1 (the hatched part indicated by 1100 in FIG. 11A) is continuously displayed. On the other hand, in the display frame indicated by 1022, the absolute time always advances in the lower right part of the dividing line 1001, even in the pause mode, so that the still image D20 corresponds to the absolute time. Corresponding portion (shaded portion indicated by 1101 in FIG. 11 (a)), corresponding portion of still image D21 (shaded portion indicated by 1102 in FIG. 11 (c)), and corresponding portion of still image D22 (of FIG. 11 (d)). The hatched portion indicated by 1103) or the corresponding portion of the still image D23 (the shaded portion indicated by 1104 in FIG. 11 (o)) is displayed alternately. That is, FIG. 11 (f), (ki), (ku), and (g) are displayed corresponding to the remainder values 0, 1, 2, 3 when the absolute time is divided by 4, respectively. It will be.

Similarly, for example, when the area of the display frame indicated by 1023 in FIG. 10 is displayed, the corresponding portion of the still image D20 (the hatched portion indicated by 1200 in FIG. 12A) corresponding to the absolute time, Corresponding portion of the still image D21 (shaded portion indicated by 1201 in FIG. 12 (a)), corresponding portion of the still image D22 (shaded portion indicated by 1202 in FIG. 12 (c)), or corresponding portion of the still image D23 (FIG. 12 (d), the hatched portion indicated by 1203) is displayed alternately. That is, when the absolute time is divided by 4, the remainder values 0, 1, 2, and 3 are displayed as shown in FIGS. 12 (o), (f), (ki), and (ku), respectively. It will be.

A traveler (photographer) who appreciates such a series of displays recalls a series of events and can feel the continuity of events because the still images are smoothly connected. Further, in this series of displays, the cinema graph D2 is loop-reproduced even if it is in the pause mode. That is, D20, D21, D22, and D23 are continuously displayed in an endless manner. Strictly speaking, in the cinema graph, only a region within the display frame (a part of the cinema graph) is loop-reproduced.

In the play mode, the still image D1 can be continuously transferred to the cinema graph D2 in the process of displaying while moving the display frame in the direction of the arrow 1031. Similarly, in the process of displaying while moving the display frame in the direction of the arrow 1032, it is possible to continuously shift from the cinema graph D 2 to the still image D 3. Thus, even if a still image and a cinema graph are mixed, a display that makes the continuity of the event feel is possible. Even in the pause mode, the expression that the cinemagraph-specific subject continues to move indefinitely is not impaired.

As can be seen from the above example, the still images used to determine the dividing line connecting the images are as shown in Table 1.

In Table 1, the image with the younger order means the image with the earlier shooting time, and the image with the lower order means the image with the later shooting time. For example, when still images and moving images are taken in order, as can be seen from the table, connection between a still image and “the first frame of a moving image (which can be regarded as a still image because it is one frame)” may be considered. This coincides with the connection between the still image C1 and the moving image C2 in FIG. Also, for example, when shooting in the order of moving image and cinema graph, as can be seen from the table, “the last frame of the moving image (which can be regarded as a still image because it is one frame)” and “the first frame of the cinema graph (one It can be considered as a still image because it is a frame). Other combinations are as shown in the table.

Next, a procedure for determining a dividing line for smooth connection when two still images are given will be described.

First, the fact that the overlapping portion of two still images need not be small and not large will be described with reference to FIG. FIG. 13 is a diagram for explaining the connection between the still image A0 and the still image A1. A0 is indicated by a thick line and A1 is indicated by a thin line so that the overlapping portion can be easily understood.

As shown in FIG. 13A, consider a case where the overlapping portion of A0 and A1 is large. For the overlapped portion, either A0 or A1 is deleted by the mask process. Therefore, when a “continuous image (picture scroll)” is reproduced, even though it has been taken, much of the area of either A0 or A1 is not displayed. This is not good.

As shown in FIG. 13A, consider a case where the overlapping portion of A0 and A1 is small. The display area starts at 1300 and moves to 1301. When 1302 is displayed on the way, there is no data in the portion indicated by 1310 or 1311, and this portion is displayed in black. It is not good that a lot of black is displayed during appreciation. Of course, it is also possible to change the size of the display frame while moving. That is, during the movement of the display frame, the display frame can be reduced so that the black portion is reduced. In this case, enlargement processing is performed in order to make the display size on the display device (display device for viewing) constant, and the resolution becomes coarse.

Therefore, the overlapping portion of the two still images needs to be an appropriate size.

Taking this into account, the procedure for determining the dividing line is as shown in FIG. In FIG. 14, when two still images S1 and S2 (assumed to have been photographed in the order of S1 and S2) are already given, determination of a dividing line for smoothly connecting the two still images is performed. The processing flow is shown. It is assumed that the still image S1 and the still image S2 have already been given before this processing is performed.

First, in step S101, a large value is set to AV0. Then, the process proceeds to step S102.

In step S102, it is determined whether all the positional relationships between S1 and S2 that satisfy the “overlap condition” have been processed. Here, the “overlapping condition” means that the overlapping portion of S1 and S2 has an appropriate size and is not in the positional relationship as shown in (a) and (b) of FIG. Specifically, for example, “the overlapping portion (rectangular region) of S1 and S2 has 100 or more pixels in the horizontal direction, has 100 or more pixels in the vertical direction, and has pixels in the horizontal direction. One of the number and the number of pixels in the vertical direction is 200 pixels or less ”. If all the positional relationships between S1 and S2 satisfying the “overlap condition” have been processed, the process proceeds to step S107, and if not, the process proceeds to step S103.

When the process proceeds to step S103, there is a positional relationship between S1 and S2 that satisfies the “overlap condition” that has not been processed yet. Therefore, in step S103, S1 and S2 are set so that the positional relationship between S1 and S2 satisfying the “overlap condition” has not yet been processed. Then, the process proceeds to step S104.

In step S104, an optimum line (breaking line) for dividing the overlapping portion is determined. That is, the line having the smallest difference average value is obtained from the lines having the start point and the end point of the intersection (there are two) of the edge of S1 (the edge of the image) and the edge of S2 (the edge of the image). L1 for the determined breaking line
The average difference value at that time is AV1. The difference average value is a value defined by the following. That is, the absolute difference value between each pixel of S1 and each pixel of S2 is summed up for the pixel position where the line passes. A value obtained by dividing the total value by the length of the line (the number of passing pixels) is the difference average value.

The dividing line in step S104 is determined by creating a graph with the start point and the end point of the intersection (there are two edges) of the edge of S1 (the edge of the image) and the edge of S2 (the edge of the image). It can be solved by planning methods and graph cuts. This determination method (dynamic programming or graph cut) is a known method, and its description is omitted.

After step S104, the process proceeds to step S105.

In step S105, it is determined whether AV1 is smaller than AV0. If it is determined that the value is smaller, the process proceeds to step S106, and if not, the process proceeds to step S102.

When the process proceeds to step S106, if the still images of S1 and S2 are connected by the dividing line L1 in the current positional relationship between S1 and S2, a better connection than the positional relationship between S1 and S2 searched so far can be achieved. That is. Therefore, in step S106, the value AV1 is set to AV0. Then, information on the dividing line of L1 is set in L0. MV0
Is set to “a vector having the center position of S1 as the start point and the center position of S2 as the end point”. Then, the process proceeds to S102.

When the process proceeds to step S107, it means that a relationship capable of optimal connection is obtained among all the positional relationships of S1 and S2 that satisfy the “overlap condition”. Therefore, in step S107, the inverse of AV0 is output as the degree of connection. The degree of connection means that the larger the value is, the more smoothly two still images are connected. This is apparent from the fact that the value of AV0 is the average value of the absolute differences between the pixels on the dividing line. Then, the process proceeds to step S108.

In step S108, MV0 is output as the positional relationship between the two images. Then, the process proceeds to step S109.

In step S109, an area surrounded by the edge of S2 (the edge of the image) and L0 and including the center of S2 is obtained. A mask image using this region as a mask is output. Then, a series of processing is finished.

In this way, all the positional relationships between S1 and S2 are searched and the positional relationship when connected as smoothly as possible (output in S108), the degree of connection when connected as smoothly as possible (smoothly (Whether or not connection is possible) (output of S107), it is possible to obtain the region (mask) of S2 (output in S109) that is adopted when connecting as smoothly as possible.

<First Embodiment>
Now, the first embodiment of the present invention will be described. FIG. 15 shows a configuration example of the image processing apparatus 10 according to the first embodiment of the present invention. In the following, D1 and D2 are used as symbols representing images, which are different from D1 and D2 in FIG.

In the image processing apparatus 10, an input unit 11, an image collection unit 12, a database management unit 13, an input image data storage unit 15, a connectability determination unit 16, a directed graph creation unit 17, a route search unit 18, and an image selection unit 19 , An image connection unit 20, a rendering information creation unit 21, a display information creation unit 22, a "continuous image (picture scroll) data storage unit 23", a user interface unit 24, a playback unit 25, and an image display unit 26 Further, the database management unit 13 includes a general-purpose image database 14.

A plurality of images taken by a traveler (photographer) are input from the input unit 11. The “image” referred to here indicates either a still image, a moving image, or a cinema graph. Assume that these images are given shooting time and shooting position as accompanying information. Furthermore, it is assumed that an important image is given as accompanying information to an important image by a traveler (photographer) as necessary.

The image input from the input unit 11 is sent to the image collection unit 12. The image collection unit 12 performs a process of discarding unnecessary images from the input images and storing only useful images in the subsequent input image data storage unit 15. Further, an image taken at the same position as the input image is taken out from the general-purpose image database 14 and this image is also stored in the input image data storage unit 15 at the subsequent stage. Note that the database management unit 13 may be installed at a position physically separated from the image collection unit 12 and exchange data via the Internet.

Conventionally, there are known a face recognition technique for automatically determining whether a face is included in an image and a Visual Attention technique for automatically determining whether a noticeable object has been photographed. These techniques are known techniques, and a description thereof is omitted.

For each of the images input from the input unit 11, the image collecting unit 12 uses the existing face recognition technology or Visual Attention technology to capture a face or to capture an object to be noticed. Only such an image is stored as a useful image in the subsequent input image data storage unit 15. In general, since an image showing a face is an important image taken as a commemorative photo, it is useful to determine whether the face is shown using face recognition technology. At this time, even if the result of judgment using existing face recognition technology or Visual Attention technology is “not a useful image”, it is given to the traveler (photographer) that it is important as accompanying information. Save the image. Even if the result is that the image is obsolete using the existing technology, there may be an image that the traveler (photographer) has a passion for. Such an image can be prevented from being discarded by giving the traveler (photographer) explicitly important information as accompanying information.

In addition, in the image input from the input unit 11, a plurality of images taken at substantially the same time are stored in the subsequent input image data storage unit 15 as any one of them as useful images, and the remaining images are stored. The image may be discarded. This process is performed using information on the photographing time that is accompanying information of the input image. The reason for discarding is that a traveler (photographer) generally takes a plurality of images of the same composition at the same location, so only one such image needs to be saved. is there.

Note that when the image is stored in the input image data storage unit 15, the degree of “an object to be noticed” determined by the existing Visual Attention technique is also used as the importance of this image. Save to storage. “Importance” means that the larger the image, the more important the image. Regardless of the degree of “noticeable object being photographed” as judged by Visual Attention technology, images that are given importance by the traveler (photographer) as accompanying information are always important. Set a very large value for the degree.

In addition, the image collection unit 12 notifies the database management unit 13 of position information (a type of accompanying information) (when captured) of an image (image A) input from the input unit 11.

Within the database management unit 13 is a general-purpose image database 14. It is assumed that a large number of beautiful images for viewing are stored in the general-purpose image database 14 in advance. Then, it is assumed that position information is attached to these images. The database management unit 13 extracts an image (image B) at the same position as the position information of the image A from the general-purpose image database 14 and sends it to the image collection unit 12.

In the image collection unit 12, the accompanying information (shooting time) of the image B is set such that “the same time as the shooting time of the image A (or the time immediately after the shooting time of the image A) is set as the shooting time of the image B”. Add a note. This is because it seems as if the traveler (photographer) has also photographed a beautiful image for viewing (image B) at the same time as when the image A was photographed. Then, the image B is stored in the input image data storage unit 15. Also, the importance of the image B is increased.

As will be described later, in the subsequent processing, both the image A and the image B are processed on the assumption that the images are taken at the time indicated in the accompanying information.

In the following, it is assumed that the number of images stored in the input image data storage unit 15 is N. Then, it is assumed that numbers are assigned from the 0th to the (N-1) th in the order of shooting times, and the story is advanced. The larger the number, the later the image was taken. That is, when storing each image from the image collection unit 12 to the input image data storage unit 15, the image collection unit 12 checks the shooting time, adds an appropriate number, and inputs the input image data storage unit. 15 stores each image.

Further, as described above, importance is given as accompanying information to each image stored in the input image data storage unit 15.

Next, an image to be actually used for a “continuous image (picture scroll)” is selected from the N images stored in the input image data storage unit 15. This process is performed by the connectability determination unit 16, the directed graph creation unit 17, the route search unit 18, and the image selection unit 19. Actually, assuming that M images are used for a “continuous image (picture scroll)”, M images are selected from N images, and (N−M) images are discarded. Images with high importance are selected as much as possible, and images with low importance are discarded. In addition, image pairs that can be smoothly connected between images are selected as much as possible, and image pairs that cannot be smoothly connected are discarded. Furthermore, for example, when an image group photographed when going on a trip for one week is input, several of the images photographed on each day are selected to be adopted. In other words, no wrong selection is performed so that no image taken on a certain day is adopted, or an image with high importance is not adopted. Details of the processing will be described later.

In this way, the selected M images are actually obtained by obtaining the positional relationship for smooth connection and an appropriate mask image by the image connecting unit 20 and the rendering information creating unit 21. Are stored in the “continuous image (picture scroll) data storage unit 23”. In addition, when displaying, as the time advances, information indicating which position is displayed (information relating to the transition of the “display frame” described above) is created by the display information creating unit 22, and the result is “continue” Are stored in the data storage unit 23 ”of the image (picture scroll). Details of the processing will be described later.

Through the above processing, a “continuous image (picture scroll)” is created and stored in the “continuous image (picture scroll) data storage unit 23”.

Next, a process of reproducing (displaying) this will be described. The “continuous image (picture scroll)” stored in the “continuous image (picture scroll) data storage unit 23” is input to the reproduction unit 25. From the user interface unit 24, the viewer designates the play mode or the pause mode. In response to the designation of this mode, the playback unit 25 cuts out an appropriate image for each time from the “continuous image (picture scroll)” and sends it to the image display unit 26. The image display unit 26 displays the sent image. Details of the processing will be described later.

With the above processing, “continuous images (picture scrolls)” are created and reproduced.

As a matter of course, the portions 24, 25, and 26 are unnecessary if only a “continuous image (picture scroll)” is created. On the other hand, if the “continuous image (picture scroll)” that has already been created is to be reproduced (displayed), the parts 11 to 22 are unnecessary.

Next, details of the process of actually selecting an image to be used for a “continuous image (picture scroll)” from the N images stored in the input image data storage unit 15 will be described.

This process is performed by creating a directed graph and searching for a path (path) that minimizes energy on the directed graph.

Note that the N images are “0th to (N−1) th images” that have already been numbered, and are not “1st to Nth images”.

First, processing (see FIG. 16) for obtaining energy (more precisely, edge energy) required for creating a directed graph is performed as follows. This process is performed by the connection possibility determination unit 16.

First, in step S201 in FIG. 16, 0 is set to the variable i. Then, the process proceeds to step S202.

In step S202, it is determined whether the value of i is N-1. If it is N-1, since the processing has been completed for all image pairs, a series of processing ends. Otherwise, the process proceeds to step S203.

In step S203, i + 1 is set to the variable j. Then, the process proceeds to step S204.

In step S204, it is determined whether the value of j is N. If it is N, processing for all image pairs has been completed for the currently set value of i, and the process proceeds to step S207. Otherwise, the process proceeds to step S205.

In step S205, the i-th image is set to D1. Let the jth image be D2. Then, according to Table 2, the still image S1 is determined. According to Table 3, the still image S2 is determined. Then, the process of FIG. 14 is performed. The connection degree obtained by the processing of FIG. 14 is output as “connection degree between the i-th image and the j-th image”. This output is input to the directed graph creation unit 17.

When the process of step S205 is performed, i <j. Therefore, the image D1 has a shooting time earlier than the image D2. Therefore, if the still image S1 and the still image S2 are determined for D1 and D2 according to Tables 2 and 3, respectively, the still image is selected according to the same rule as in Table 1 (already described). For example, if the i-th image is a still image and the j-th image is a cinema graph, from Table 2 and Table 3, the i-th image (still image) is S1 and j is S2. The first frame of the cinema graph that is the image of the eye is selected (since it is a single frame, it can be regarded as a still image). Also, for example, if the i-th image is a moving image and the j-th image is a still image, from Table 2 and Table 3, the last frame (1 Since it is one frame, it can be regarded as a still image), but the j-th image (still image) is selected as S2. 14 is performed, the positional relationship when connecting as smoothly as possible, the connection degree when connecting as smoothly as possible (whether it can be connected smoothly), and as smoothly as possible It is possible to obtain the region (mask) of S2 that is adopted when connecting. Here, only the degree of connection is necessary, and the positional relationship and the region (mask) of S2 are determined by the processing of FIG.

After step S205, the process proceeds to step S206.

In step S206, the value of j is increased by 1 to select the next image pair. Then, the process proceeds to step S204.

In step S207, the value of i is increased by 1 to select the next image pair. Then, the process proceeds to step S202.

As will be described later, the “degree of connection between the i-th image and the j-th image” output in S205 is used as the energy of each edge of the directed graph.

This is the end of the description of the process for obtaining the energy required to create the directed graph (more precisely, the edge energy). Next, creation of a directed graph will be described.

First, in step S301 of FIG. 17, N + 2 ordered nodes are created. That is, N + 2 nodes of the 0th node, 1st node, 2nd node,..., N + 1th node. The 0th node is called a source node. The N + 1th node that is the last node is called a sink node. Each node has energy. Then, the process proceeds to step S302.

In step S302, 0 is set as the energy of the 0th node (source node) and the N + 1th node (sink node). Then, the process proceeds to step S303.

In step 303, attention is focused on the i-1th image (i is an integer from 1 to N) in the N images stored in the input image data storage unit 15. The inverse of the importance of this image is set as the energy of the i-th node. Then, the process proceeds to step S304.

In step S304, an i-th node (i is an integer from 0 to N) is used as a starting point, and a node lower in rank than the i-th node (j-th node, where j is an integer from i + 1 to N + 1). Create an edge (a directional edge) as the end point. However, only the edge starting from the 0th node (source node) and ending at the (N + 1) th node (sink node) is not created. Each edge has energy. Then, the process proceeds to step S305.

In step S305, a monotonically increasing value is set as the energy of the edge from the 0th node (source node) to the i-th node (i is an integer from 1 to N). That is, the energy of the edge from the 0th node (source node) to the i-th node is set to a larger value as the value of i increases. For example, the energy of the edge from the 0th node (source node) to the i th node is set to a value of “i squared”. Then, the process proceeds to step S306.

In step S306, a value monotonously decreasing with respect to the value of i is set as the energy of the edge from the i-th node (i is an integer from 1 to N) to the N + 1-th node (sink node). That is, the energy of the edge from the i-th node toward the (N + 1) -th node (sink node) is set to a smaller value as the value of i increases. For example, the energy of the edge from the i-th node toward the (N + 1) -th node (sink node) is set to a value of “(N + 1−i) square”. Then, the process proceeds to step S307.

In step S307, the energy of the edge from the i-th node (i is an integer from 1 to N-1) to the j-th node (j is an integer from i + 1 to N) is set. In other words, in the N images, attention is paid to the i−1th image and the j−1th image. An edge directed from the i-th node to the j-th node by a value obtained by multiplying the reciprocal of the connection degree of these two images (value obtained by the processing of FIG. 16) by the square of (ji). Set as energy.

Now, by creating nodes and edges in this way, a directed graph can be created by N + 2 nodes and edges (directional edges) connecting these nodes. Each node and each edge is given energy.

FIG. 18 shows a directed graph when N = 7. A circle indicated by S in the figure is the 0th node (source node), and a circle indicated by T is the N + 1th node (sink node). Circles indicated by 0 to 6 in the figure are the first to seventh nodes, respectively. In addition, an arrow in the figure is an edge (an edge having a direction).

After step S307, the process proceeds to step S308.

In step S308, a path (path) that starts from the 0th node (source node) and reaches the N + 1th node (sink node) is the minimum energy. That is, in this directed graph, consider a path (path) that starts from one node, passes through one or more edges and nodes, and arrives at another node. Edges can only advance in the direction of the specified orientation, not backwards. A value obtained by summing (adding) energy of nodes and edges that have passed in an arbitrary route (path) is defined as energy in the route (path). By defining “the energy of the path (path)” in this way, it starts from the 0th node (source node) and arrives at the (N + 1) th node (sink node) by dynamic programming (dynamic programming). Among all the paths (paths), the path (path) having the minimum energy can be obtained. The dynamic programming method is a known method, and its description is omitted. To restate, in step S308, the minimum energy and the minimum energy in all paths (paths) starting from the 0th node (source node) and arriving at the N + 1th node (sink node) are determined by dynamic programming. Find a route. Then, the process proceeds to step S309.

In step S309, only the optimum image is selected from the N images in correspondence with the path (path) having the minimum energy. That is, if the i-th (i is an integer from 1 to N) node is included in the path (path) having the minimum energy, the i-1th image is held. On the other hand, if the i-th (i is an integer from 1 to N) node is not included in the path having the minimum energy, the i-1th image is discarded. Then, a series of processing is finished.

Image selection is performed by the above-described series of processing (the processing shown in FIG. 17).

In the processing shown in FIG. 17, the remaining image (selected image) without being discarded is an optimal image as a material for creating a “continuous image (picture scroll)”. The reason for being optimal is as follows.

Reason 1: The energy of the edge from the 0th node (source node) to the i-th node (i is an integer of 1 to N) is set to a monotonically increasing energy with respect to the value of i. Therefore, when the value of i is large, the energy of the edge from the 0th node (source node) to the i-th node becomes a large value, and such an edge is not easily included in the path (path) having the minimum energy. Become. That is, when attention is paid to the edge from the 0th node (source node) to the i-th node included in the path (path) having the minimum energy, the value of i is a small value. That is, of the N images, at least one of the first images is held.

Reason 2: Focusing on the edge from the i-th node to the (N + 1) -th node (sink node) included in the path (path) having the minimum energy, the value of i is a large value. This is clear when considered in the same manner as in Reason 1. Therefore, at least one of the last images is held in N images.

Reason 3: Focus on the edge from the i-th node (i is an integer from 1 to N-1) to the j-th node (j is an integer from i + 1 to N) included in the path having the minimum energy. . In this case, the (i-1) th and (j-1) th images are retained, and the i-th to j-2th images are discarded. Since the energy of the edge from the i-th node to the j-th node is small, it means that the connection degree between the (i−1) -th image and the j−1-th image is good. Further, clearly from the definition, considering that the energy of the corresponding edge increases as the difference between i and j increases, a subscript indicating the start point and end point of each edge included in the path (path) having the minimum energy. The difference between i and j should be small. That is, the held i−1th and j−1th images are neighboring images in the order of N ordered images.

Reason 4: Pay attention to the i-th node (i is an integer from 1 to N) included in the path having the minimum energy. The energy of this node is a small value. That is, the importance of the held (i−1) -th image is a large value. Conversely, an image with a low importance is not included in a path (path) having the minimum energy and is discarded.

As described above, when reason 1 to reason 4 are put together, at least one of the first images is retained in N images (reason 1). For example, when inputting a group of images taken when traveling for a week, no image taken on the first day is used. Among the N images, at least one of the last images in the sorted order is retained (reason 2). For example, when inputting a group of images taken when traveling for a week, there is no possibility that no image taken on the last day is adopted. In N images, a series of images is not completely adopted (reason 3). For example, when inputting a group of images taken when traveling for a week, not all images taken on an intermediate day are adopted. Also, an important image is adopted (reason 4). In addition, a smooth connection is possible between the images (reason 3). For the above reasons, an optimal image is held as a material for creating a “continuous image (picture scroll)”, and an unnecessary image is discarded.

In addition, for an image given importance by a traveler (photographer) as accompanying information, a very large value is set as the importance, so that it is almost certainly adopted.

Note that the directed graph creation unit 17 in FIG. 15 performs processing from step S301 to step S307 described above. Since the information of the directed graph from the directed graph creation unit 17 is transmitted to the route search unit 18, step S 308 is processed by the route search unit 18. Then, since the information on the path (path) that is the minimum energy from the path search unit 18 is transmitted to the image selection unit 19, the image selection unit 19 performs processing in step S309. That is, since the image selection unit 19 receives the images (N) in the input image data storage unit 15, only the selected image is sent to the subsequent image connection unit 20.

Further, since the importance level information of each image is transmitted from the input image data storage unit 15 to the directed graph creation unit 17, processing is performed using this information in step S303. Since the information on the degree of connection between the images is transmitted from the connectability determination unit 16 to the directed graph creation unit 17, in step S307, processing is performed using this information.

In the process of FIG. 17, a small random number may be added to each node and each edge before performing step S308. Thereby, the path | route (path) used as the minimum energy will differ slightly. That is, even if the same image group is input to the image processing apparatus 10, the image adopted every time it is processed is slightly different. Every time it is processed, a slightly different “continuous image (picture scroll)” can be created, so if it is processed once again by the image processing apparatus 10, any “continuous image (picture scroll)” is generated. The user can enjoy it.

Next, a process of creating a “continuous image (picture scroll)” using the images selected by the image selection unit 19 will be described.

As described above, the image selection unit 19 performs selection from N images (0th to (N-1) th images). Here, the number of selected images (remaining without being discarded) is M. The M images are renumbered from the 0th to the M−1th, in ascending order of the numbers, and sent to the image connection unit 20 at the subsequent stage. Accordingly, the image connection unit 20 receives images numbered from 0th to M−1th. The M ordered images are in order of shooting time.

Processing performed in the image connection unit 20 will be described below with reference to FIG.

First, in step S401 in FIG. 19, 0 is set to k as a variable representing an image to be processed at present. Then, the process proceeds to step S402.

In step S402, the k-th image is set as D2. Mask (k) is a mask having the same size as that of the kth image. Here, since k is 0, it is equivalent to the processing of “0th image is D2. And Mask (0) is a mask having the same size as the 0th image”.

The definition of the word mask will be described. That is, the pixels of the corresponding image are displayed for the area specified by the mask (or mask image), and the pixels of the corresponding image are not displayed for the area not specified by the mask (or mask image).

After step S402, the process proceeds to step S403.

In step S403, a mask is attached to the k-th image, and the k-th image is stored in the “continuous image (picture scroll) data storage unit 23” (see Table 4). At this time, as shown in Table 4, when the kth image is a moving image, it becomes a serial still image with f as an argument, and when it is a cinema graph, it becomes a serial still image with f as an argument. That is, all the image data stored in the “continuous image (picture scroll) data storage unit 23” is a still image with a mask.

The data stored in the “continuous image (picture scroll) data storage unit 23” in step S403 will be described again.

That is, as shown in Table 4, when the image D2 is a still image, since D2 is composed of one still image, the data obtained by adding Mask (k) as a mask to D2 (one still image) is used. Save as image (k, 0).

When the image D2 is a moving image, assuming that the number of frames is F, data obtained by adding Mask (k) as a mask to the first frame of D2 (that is, the 0th frame) is called image (k, 0). Save with a name. Further, data obtained by adding a mask having the same size as the image to the f-th frame of D2 (f is an integer from 1 to F-1) is stored as image (k, f). Therefore, when the image D2 is a moving image, the number of still images to be stored is the same number (F) as the number of frames of the moving image. Note that Mask (k) is added only to the first frame (that is, the 0th frame), and a mask having the same size as the image is added to the other frames. For example, in FIG. 9, C20 is in agreement with the fact that only the right side of the dividing line 901 is adopted in order to connect C20 smoothly with the dividing line 901. C21, C22, and C23 coincide with the fact that all areas of the image are adopted. Although it is an aside, it has been described that all areas of the image are adopted for C23, but in reality, when C30 in the lower right part of the dividing line 902 is adopted, the lower right part of C23 is used. Is overwritten.

When the image D2 is a cinema graph, assuming that the number of frames is F, data obtained by adding Mask (k) as a mask to each of the f-th frames (f is an integer from 0 to F-1) of D2. Save as image (k, f). Therefore, when the image D2 is a cinema graph, the number of still images to be stored is the same number (F) as the number of frames of the cinema graph. Unlike the case of moving images, the addition of Mask (k) in all frames is consistent with what has been described with reference to FIG.

After step S403, the process proceeds to step S404.

In step S404, it is determined whether the value of k is M-1. If it is M-1, since the processing has been completed for all the images, a series of processing is completed. Otherwise, the process proceeds to step S405.

In step S405, the value of k is increased by 1 to process the next image. Then, the process proceeds to step S406.

In step S406, the (k−1) -th image is set as D1. Let the kth image be D2. Then, the still image S1 is determined according to Table 2 (existing). The still image S2 is determined according to Table 3 (existing). Then, the processing flow of FIG. 14 is performed. The positional relationship (two-dimensional vector) of the two images obtained by the processing of FIG. 14 is output as MV (k−1, k). (The output MV (k−1, k) is input to the rendering information creating unit 21.) Further, the mask image obtained by the process of FIG. 14 is masked (k).

In step S406, the positional relationship when the k-1th image and the kth image are smoothly connected and the mask of S2 are obtained. The reason is the same as that described in step S205 in FIG. Here, only the positional relationship and the region (mask) of S2 are necessary, and the degree of connection is obtained by the processing of FIG.

After step S406, the process proceeds to step S403.

Through the series of processes described above, each of the 0th to M−1th images is converted into a still image if necessary according to the type of image, and an appropriate mask (a mask that can be smoothly connected between adjacent images). And stored in the “continuous image (picture scroll) data storage unit 23”. In other words, the masked still image image (k, f) is stored in the “continuous image (picture scroll) data storage unit 23”. Here, k is an integer from 0 to M-1. If the kth image is a still image, f is only 0. If the kth image is a moving image or a cinema graph, f is an integer from 0 to F-1. Here, F is the number of frames included in the kth image.

Next, processing performed by the rendering information creation unit 21 will be described with reference to FIGS. By executing this processing, position information (rendering information file: a plurality of renderingTextFile (s) files with s as a variable) indicating what position image (k, f) should be rendered in can be created. .

Before explaining the processing flow shown in FIG. 20 and FIG.

As described with reference to FIG. 8, at the time when the display area (display frame) is 820 to 823, only the still images B0, B1, B2, and B3 are rendered at appropriate positions. At this time, the still image B4 should not be rendered. As described above, if B4 is rendered, B2 at the same position is overwritten and disappears. Then, at the time when the display area (display frame) is 823 to 825, only the still images B3, B4, and B5 are rendered at appropriate positions. That is, as rendering information, “to render only the still images B0, B1, B2, and B3 at appropriate positions” and “to render only the still images B3, B4, and B5 at appropriate positions. It is necessary to divide into two things. In short, when the positional relationship overlaps, it is necessary to consider the rendering information separately.

Further, as described with reference to FIG. 9, only the still images C0, C1, and C20 are rendered at appropriate positions at the time when the display area (display frame) is 920 to 922. At this time, the still image C21 must not be rendered. This is because, as described above, if C21 is rendered, C20 is overwritten and disappears. In FIG. 9, the same can be said for the time when the area (display frame) to be displayed is 923 or later. That is, as rendering information, “only the still images C0, C1, and C20 are rendered at an appropriate position”, “only the still image C21 is rendered at an appropriate position”, and “still image "Render only C22 at an appropriate position", "Render only still images C23 and C30 at an appropriate position", "Render only still image C31 at an appropriate position", It is necessary to divide it into “only the still images C32, C4, and C5 are rendered at appropriate positions”. In short, when the material is a moving image, it is necessary to consider rendering information separately.

As described with reference to FIG. 10, when the material is a cinema graph, “D20, D21, D22, or D23 is selected as a still image to be rendered according to the absolute time”. Information needs to be included in the rendering information.

Now, FIG. 20 and FIG. 21 will be described.

First, in step S501 in FIG. 20, a buffer for character input having an empty content is prepared. Then, 0 is set to the variable s indicating the file number in which the rendering information is described. 0 is set to the variable k indicating the number of the image to be processed. (0, 0) is set to the two-dimensional vector mv indicating the position of the image.

In the buffer, as will be described later, “two-dimensional vector representing the position to render the image”, “number of still images”, and “name of stored still image (one or , Multiple) ”.

After step S501, the process proceeds to step S502.

In step S502, when the kth image is arranged at the position of mv, it is determined whether there is an intersection with any of the images in the buffer (excluding the image written in the last line). Advances to step S503. Otherwise, the process proceeds to step S504.

Here, a description of the “intersection” is added. If nothing is written in the buffer, there is no object to be determined, so there is no interaction. When only one line is written in the buffer, since the determination is made except for the last line, there is no target for substantial determination, so there is no intersection. When two or more lines are written in the buffer, the “intersection” with the image excluding the last line is determined. Specifically, an arbitrary line (excluding the last line) written in the buffer includes “a two-dimensional vector (mv0) representing a position where an image is to be rendered” and “the number of still images”. ”And“ name of saved still image (one or plural) (image0) ”. Therefore, image0 is read out from the data stored in the “continuous image (picture scroll) data storage unit 23”, and the size of image0 (the number of vertical pixels is image0H and the number of horizontal pixels is image0W) Investigate). Consider a rectangular area RECT0 having a vertical image0H and a horizontal image0W centered on mv0 on a two-dimensional plane. Further, the k-th image is read out from the image selection unit 19 and the size of the image (the number of vertical pixels is KH and the number of horizontal pixels is KW) is examined. Then, a rectangular area RECT1 having vertical KH and horizontal KW around mv on the two-dimensional plane is considered. If there is an intersection between these two rectangular areas RECT0 and RECT1, it is determined that there is an intersection. For all lines written in the buffer (excluding the last line), RECT0 is obtained, and the intersection with RECT1 is determined. If there is even one intersection, it is determined that there is an intersection. If there is no interaction, it is determined that there is no interaction.

In step S503, the contents of the buffer are output as a file to the "data storage unit 23 for a series of images (picture scrolls)" with the name renderingTextFile (s). All the other lines are erased, leaving the last line in the buffer. By this deletion, the last line becomes the first line. Then, the value of s is increased by 1. Then, the process proceeds to step S504.

For example, in the situation shown in FIG. 8, if B4 is the kth image, there is an intersection with B2, so it can be determined that there is an intersection in step S502, and in step S503, “still images B0, B1, B2”. And a rendering information file having information that “only B3 is rendered at an appropriate position” can be created. Thereafter, by proceeding with the processing, information related to B3 is maintained in the buffer in order to create information that “only the still images B3, B4, and B5 are rendered at appropriate positions”. That is, in step S503, the process of leaving the last line of the buffer (in this case, information regarding B3) is necessary.

In step S504, the k-th image is read from the image selection unit 19 and the type of this image is examined. If the kth image is a still image, the process proceeds to step S505. In the case of a cinema graph, the process proceeds to step S506. In the case of a moving image, the process proceeds to step S507 (FIG. 21).

In step S505, the one-line text “mv, 1, {image (k, 0)}” is added to the end of the buffer. Here, mv represents “a two-dimensional vector representing a position where an image is to be rendered”, and the next numerical value “1” represents “the number of still images”, which is surrounded by {and} at the end. The name in the box represents “name of one or more stored still images”. The k-th image (one still image) is stored as “image (k, 0)” in the “continuous image (picture scroll) data storage unit 23” by the processing in the image connection unit 20. ing. That is, in step S505, rendering information that the kth image (one still image) is arranged at the position of mv is written in the buffer.

After step S505, the process proceeds to step S513.

In step S506, the single-line text “mv, F, {image (k, 0), image (k, 1), image (k, 2),..., Image (k, F−1)}” is added to the end of the buffer. Add to Here, F is the number of frames of the kth image (cinema graph). The value of F can be determined by reading the kth image from the image selection unit 19 and examining the number of frames of this image (cinema graph). Here, mv represents “a two-dimensional vector representing a position where an image is to be rendered”, and the next numerical value “F” represents “the number of still images”, which is surrounded by {and} at the end. The name in the box represents “name of one or more stored still images”. The k-th image (cinema graph composed of F still images) is processed by the image connection unit 20, and image (k, 0) is stored in the “continuous image (picture scroll) data storage unit 23”. , Image (k, 1), image (k, 2),..., Image (k, F-1). That is, in step S505, rendering information that the kth image (F still images) is arranged at the position of mv is written in the buffer.

After step S506, the process proceeds to step S513.

In step S507 in FIG. 21, the number of frames of the kth image (moving image) is set in F0. Then, the process proceeds to step S508.

In step S508, 0 is set to the variable f representing the target frame to be processed in the moving image. Then, the process proceeds to step S509.

In step S509, the one-line text “mv, 1, {image (k, f)}” is added to the end of the buffer. Then, the contents of the buffer are output as a file to the “data storage unit 23 of a series of images (picture scrolls)” with the name renderingTextFile (s). Then, all the contents of the buffer are cleared. Then, the value of s is increased by 1. Then, the process proceeds to step S510.

In step S510, the value of f is increased by 1. Then, the process proceeds to step S511.

In step S511, it is determined whether the value of f is F0-1. If it is F0-1, processing has been completed for all the still images constituting the moving image (except for the still image that is the last frame), and the process proceeds to step S512. Otherwise, the process proceeds to step S509.

In step S512, the one-line text “mv, 1, {image (k, f)}” is added to the end of the buffer. Then, the process proceeds to step S513.

For example, in the situation shown in FIG. 9, C2 is the k-th image (consisting of four images C20, C21, C22, and C23). In this case, F0 = 4. If f = 0, a rendering information file having information “Render only still images C0, C1, and C20 to appropriate positions” can be created in step S509. If f = 1, a rendering information file having information “Render only the still image C21 to an appropriate position” can be created in step S509. If f = 2, then in step S509, a rendering information file having information “Render only the still image C22 to an appropriate position” can be created. Then, if f = 3 (that is, F0-1), by proceeding with the processing thereafter, in order to create information that “only the still images C23 and C30 are rendered at appropriate positions”, C23 A process of writing information on the buffer into the buffer (step S512) is performed.

In step S513, it is determined whether the value of k is M-1. If it is M−1, since processing has been completed for all M images, the process proceeds to step S516. Otherwise, the process proceeds to step S514.

In step S514, the value of k is increased by 1 to process the next image. Then, the process proceeds to step S515.

In step S515, the position for rendering the kth image is set to mv. Specifically, a vector MV (k−1, k) indicating the k−1 and kth positional relationship may be added to the position of the k−1th image. That is, mv + MV (k-1, k) is newly set as mv. Note that MV (k−1, k) is given from the image connection unit 20. Then, the process proceeds to step S502.

In step S516, the data remaining in the buffer last is output. That is, the contents of the buffer are output as a file to the “data storage unit 23 for a series of images (picture scrolls)” with the name renderingTextFile (s). Then, a series of processing is finished.

Thus, in the rendering information creation unit 21, information on what position the still image with mask stored in the "data storage unit 23 of the continuous image (picture scroll)" should be rendered. Can be created.

Next, processing performed by the display information creation unit 22 will be described with reference to FIGS. By executing this process, it is possible to create information indicating which position is displayed as the time advances during reproduction (information relating to the transition of the “display frame” described above).

First, in step S601 of FIG. 22, a buffer for character input having an empty content is prepared. Then, the process proceeds to step S602.

In the buffer, as will be described later, “number for specifying the rendering information file” and “position information indicating which position to display when displaying (position of the aforementioned“ display frame ”) ) ”And“ Information of size when displaying (size of the above-mentioned “display frame”) ”.

In step S602, 0 is set to a variable s indicating a rendering information file number. Then, the process proceeds to step S603.

In step S603, it is determined whether there is a file called renderingTextFile (s) in the “continuous image (picture scroll) data storage unit 23”. If a file exists, the process advances to step S604. Otherwise, the process proceeds to step S618.

In step S604, 0 is set to a variable k indicating the number of lines of interest in the file called renderingTextFile (s). The first line is not the first line but the 0th line. Then, the process proceeds to step S605.

In step S605, attention is paid to the k-th row (ie, the 0th row) of renderingTextFile (s). The X coordinate value of the position information (two-dimensional vector) mv written in this line is set to X2, and the Y coordinate value is set to Y2. The still image written in this line is read out from the “continuous image (picture scroll) data storage unit 23”, and the value obtained by multiplying the number of pixels in the width direction of the still image by 0.6 is W2. A value obtained by multiplying the number of pixels in the direction by 0.6 is set to H2. Then, the process proceeds to step S606.

As will be described later, W2 and H2 are the sizes of the display frames. When obtaining W2 and H2, the reason why the horizontal and vertical sizes are multiplied by 0.6 is because the central 60% portion is displayed instead of the entire still image. By setting the ratio to about 60%, it is difficult to cause a portion where no image data exists in the process of moving the display frame. Of course, the present invention is not limited to 60%, and other magnifications may be used.

In step S606, the value of k is incremented by 1 to read the next line. Then, the process proceeds to step S607.

In step S607, it is determined whether or not the renderingTextFile (s) has the k-th row. If there is, the process proceeds to step S608 (FIG. 23). Otherwise, the process proceeds to step S616.

In step S608 of FIG. 23, the value of X2 is substituted for X1, the value of Y2 is substituted for Y1, the value of W2 is substituted for W1, and the value of H2 is substituted for H1. Then, the process proceeds to step S609.

In step S609, attention is focused on the kth row of renderingTextFile (s). The X coordinate value of the position information (two-dimensional vector) mv written in this line is set to X2, and the Y coordinate value is set to Y2. The still image written in this line is read out from the “continuous image (picture scroll) data storage unit 23”, and the value obtained by multiplying the number of pixels in the width direction of the still image by 0.6 is W2. A value obtained by multiplying the number of pixels in the direction by 0.6 is set to H2. Then, the process proceeds to step S610.

In step S610, the distance between the position (X1, Y1) on the two-dimensional plane and (X2, Y2) is set as Length. Then, the process proceeds to step S611.

In step S611, 0 is set to a variable currentPosition indicating the current position. Then, the process proceeds to step S612.

In step S612, it is determined whether the value of currentPosition is smaller than Length. If so, the process proceeds to step S613. Otherwise, the process proceeds to step S606.

In step S613, a value of “currentPosition / Length” is set as the weight weight2 for performing the weighted average. Then, a value of “1-weight2” is set to weight1, which is another weight. Furthermore, using these two weights, four weighted averages as follows are obtained. That is, X = weight1 * X1 + weight2 * X2, Y = weight1 * Y1 + weight2 * Y2, W = weight1 * W1 + weight2 * W2, H = weight1 * H1 + weight2 * H2. Then, the process proceeds to step S614.

In step S614, the one-line text “s, (X, Y), (W, H)” is added to the end of the buffer. Here, s is a number for specifying the rendering information file used for display, (X, Y) is the position of “display frame” (more precisely, the center position of “display frame”), ( W, H) is the size of the “display frame”.

After step S614, the process proceeds to step S615.

In step S615, the value of currentPosition is increased by 10 to advance the display position. Here, the display position is shifted with time, but the shift amount is 10 in terms of the number of pixels. Of course, the present invention is not limited to 10, and other amounts may be used. Then, the process proceeds to step S612.

The information written in the buffer so far will be described with reference to FIG.

In FIG. 24, the still image written in the k−1 line of the renderingTextFile (s) focused in step S605 and its position (still image center position) (X1, Y1) A display frame (W1, H1) having a size of 60% is illustrated. Note that since the value of k is increased by 1 in step S606, the number of lines read before that time is not k, but is the (k-1) th line. In step S608, the value of the variable whose subscript is 2 is assigned to the variable whose subscript is 1.

Further, in FIG. 24, the still image written in the kth line of the renderingTextFile (s) focused in step S609 and its position (still image center position) (X2, Y2) A display frame (W2, H2) having a size of 60% is illustrated.

According to the calculation in step S613, (X, Y) advances in increments of 10 pixels on the arrow starting from (X1, Y1) and ending at (X2, Y2) in the figure. Also, (W, H) changes from (W1, H1) to (W2, H2) as the value of currentPosition increases. That is, in the buffer, “the position of the still image written in the (k−1) -th line (X1, Y1) is 60% of the size of the still image written in the (k−1) -th line (W1 , H1) to display the display frame ", and further, shifting the display frame position by 10 pixels to (X2, Y2) which is the position of the still image written in the k-th line, The description that the size of the display frame is close to (W2, H2), which is 60% of the still image written in the k-th line, is saved.

Now, the description returns to FIG.

The process reaches step S616 when the process proceeds to the last line of renderingTextFile (s). In step S616, a description for displaying the last display frame related to renderingTextFile (s) is added to the buffer. Specifically, “s, (X2, Y2), (W2, H2)
Is added to the end of the buffer. As a result, “(X2, Y2) which is the position of the still image (the center position of the still image) written in the last line of the renderingTextFile (s) shown in FIG. 25 is set as the position of the display frame. A description “display (W2, H2) as the size of the display frame, which is 60% of the size of the still image” is stored in the buffer.

After step S616, the process proceeds to step S617.

In step S617, since the display information related to renderingTextFile (s) has ended, the value of s is incremented by 1 in order to execute processing related to the next file. Of course, after this, the numerical value (s) at the beginning of the one-line text written in the buffer is a value increased by one. Then, the process proceeds to step S603.

In step S618, all the display information has been written in the buffer, so the contents of the buffer are output as a file to the "data storage unit 23 for a series of images (picture scrolls)" with the name playListFile. Then, a series of processing is finished.

Now, the processing described so far is performed by each unit of the image processing apparatus 10 in FIG. 15, so that the “continuous image (picture scroll) data storage unit 23” is related to “continuous image (picture scroll)”. Data will be saved. That is, a still image with a mask selected as a material for a series of images (picture scrolls) is stored with a file name of image (k, f) with k and f as variables. Also, information indicating where to place each still image is stored as a file name renderingTextFile (s) with s as a variable. In each renderingTextFile (s), one or more lines of text are written. In each line, “a two-dimensional vector representing the position where the image is to be rendered”, “the number of still images”, and “saved” are stored. The name of the still image (one or more) ”is written. Further, when displaying, as time advances, information indicating which position is to be displayed (information regarding the transition of the “display frame” described above) is stored under the name playListFile (one file). Each line of playListFile includes “the number of a file in which rendering information is written, that is, information for specifying renderingTextFile (s)” and “position information indicating which position is to be displayed (the above-described“ display frame ”). "Position)" and "size information for display (size of the above-mentioned" display frame ")". These files are collected and stored in the “continuous image (picture scroll) data storage unit 23”.

It should be noted that the playListFile by the display information creation unit 22 only needs to be performed immediately before the reproduction described later. Therefore, only image (k, f) and renderingTextFile (s) are created and stored in the “continuous image (picture scroll) data storage unit 23”, and the process is temporarily terminated. Then, just before playback, a playListFile may be created using image (k, f) and renderingTextFile (s) stored in the “continuous image (picture scroll) data storage unit 23”.

Next, the reproduction of the “continuous image (picture scroll)” stored in the “continuous image (picture scroll) data storage unit 23” will be described.

In the user interface unit 24 in the image processing apparatus 10 of FIG. 15, an instruction from the viewer is input. Specifically, an instruction for play mode or pause mode is input. This mode information is sent to the playback unit 25.

When there is an instruction for the play mode, the playback unit 25 executes the processing flow of FIG.

Since an appropriate still image is cut out at each time by the playback unit 25, the image display unit 26 displays the cut out still image so that the viewer can appreciate it.

Now, processing when the playback unit 25 is instructed to play mode will be described.

When there is a play mode instruction, first, in step S701 in FIG. 26, 0 is set to LA indicating the number of lines of interest in the playListFile. The first line is not the first line but the 0th line. LA is the same as the “reproduction time” when the reproduction of the cinema graph is described with reference to FIG. Then, the process proceeds to step S702.

In step S702, 0 is set to a variable T indicating the absolute time. The absolute time T is the same as the “absolute time” when the reproduction of the cinema graph is described with reference to FIG. Then, the process proceeds to step S703.

In step S703, the rendering list file written in the LA line of the playListFile stored in the “continuous image (picture scroll) data storage unit 23” is read. Specifically, the renderingTextFile (s) corresponding to the numerical value (s) at the beginning of the LA line is read from the “continuous image (picture scroll) data storage unit 23”. This file is a rendering list file. Then, the process proceeds to step S704.

In step S704, the campus data is cleared before each still image is rendered. Then, the process proceeds to step S705.

In step S705, 0 is set to LB indicating the number of lines of interest in the rendering list file currently being read. The first line is not the first line but the 0th line. Then, the process proceeds to step S706.

In step S706, attention is focused on the LB line of the read rendering list file. In this line, the two-dimensional vector (mv) is written at the top, the number of still images (F) is written second, and the file names of F still images are written last. Therefore, among the F still images, the “T mod F (that is, the remainder when T is divided by F)” still image is read from the “continuous image (picture scroll) data storage unit 23”. Then, this still image is overwritten at the position of mv on the canvas. At this time, only the area designated by the mask attached to the still image is overwritten.

When F = 1, one still image written at the end of the LB line is always selected regardless of the value of T. When F is 2 or more, it is equivalent to the material being a cinema graph, and the selected still image is different depending on the absolute time T.

After step S706, the process proceeds to step S707.

In step S707, it is determined whether there is a next line in the rendering list file. If there is, the process proceeds to step S708. If not, the process proceeds to step S709.

In step S708, the value of LB is increased by 1 in order to render the next line in the rendering list file. Then, the process proceeds to step S706.

When the process moves to step S709, all still images written in the rendering list file currently read are rendered at appropriate positions on the campus. Therefore, in step S709, a rectangular area having a size of (W, H) written in the third centered on the position of (X, Y) written in the second line of the LA line of playListFile is set. The image is taken out from the canvas and displayed on the image display unit 26. Then, the process proceeds to step S710.

In step S710, the process is interrupted for 1/30 second before displaying at the next time. This is because the information is displayed every 1/30 seconds. Then, after 1/30 second, the process proceeds to step S711.

In step S711, it is determined whether there is a next line in the playListFile. If there is, the process proceeds to step S712. If not, display has been completed according to all display instructions in the playListFile, and the series of processing ends.

In step S712, the current mode is checked. Even during playback, the viewer can change the mode via the user interface unit 24. Therefore, at each time, it is necessary to perform appropriate processing according to the mode at that time. That is, when the current mode is the play mode, the process proceeds to step S713. In the pause mode, the process proceeds to step S714.

When the process proceeds to step S713, since it is the play mode, it is necessary to display the next display frame. Therefore, in step S713, the value of LA is increased by 1. Then, the process proceeds to step S714.

In step S714, the value of T is increased by 1. This is because the absolute time is always advanced regardless of the type of mode. The reason has already been described with reference to FIG. Then, the process proceeds to step S703.

Before finishing the description of the first embodiment of the present invention, important processing and data will be described below.

Input an image taken by a traveler (photographer). If described in correspondence with the claims, it corresponds to the first input unit.

If necessary, an image related to the image is also input from the general-purpose image database. This will correspond to the second input section if described in correspondence with the claims.

These images are ordered according to a certain rule (in this case, depending on the shooting time). This will correspond to the image aligning section if described in correspondence with the claims.

Determine the importance of each image. This may be determined by a known technique, or may be explicitly determined by a traveler (photographer). If this is described in correspondence with the claims, it corresponds to the importance determination section.

Then, the degree of connection between the images (degree of whether smooth connection is possible) is obtained. If this is described in correspondence with the claims, it corresponds to the connection determination unit.

Based on the degree of connection and the importance of each image, an image necessary as a material for creating a “continuous image (picture scroll)” is selected. Create a "continuous image (picture scroll)" by smoothly connecting the selected images. This corresponds to the connecting portion if described in correspondence with the claims.

Furthermore, the order in which “continuous images (picture scrolls)” are reproduced is determined. If this is described in correspondence with the claims, it corresponds to the reproduction order instruction generating unit.

As described above, a still image called image (k, f) having k and f as arguments is created. These still images correspond to the “plurality of image data” in claim 5 if described in correspondence with the claims.

A file indicating the positional relationship of each image, called renderingTextFile (s) with s as an argument, is created. These files correspond to the “positional relationship between the image data” in claim 5 if described in correspondence with the claims.

A file indicating a display position called “playListFile” is created. This file corresponds to “order to be displayed” in claim 5 if described in correspondence with the claims.

image (k, f), renderingTextFile (s), and playListFile are combined into one and stored. If this is described in correspondence with claims, it corresponds to “a data format having a plurality of image data, a positional relationship between the image data, and data indicating an order to be displayed”.

The stored image (k, f), rendering TextFile (s), and playListFile can be used for reproduction.

This is the end of the description of the processing contents in the first embodiment of the present invention.

<Second Embodiment>
Next, a second embodiment of the present invention will be described.

In the second embodiment of the present invention, the following scene is assumed. That is, it is a scene in which two friends (referred to as P1 and P2) make separate trips and finally join together. Suppose that P1 has photographed A10 (still image with a car) and A11 (still image with a house) shown in FIG. 27 in order during travel. Then, P2 shoots A20 (still image with trees) and A21 (still image with ships) in order during travel. These still images are taken at different locations by P1 and P2. Then, after taking these pictures, P1 and P2 meet at the meeting place, and finally P1 and P2 take pictures of themselves (still picture A30).

After traveling, P1 and P2 will appreciate the “continuous image (picture scroll)” together. At this time, "P1 saw a car and saw a house while traveling" and "P2 saw a tree and saw a ship while traveling" , “P1 and P2 met at the end of the trip” can be recalled. Such a “continuous image (picture scroll)” can be created by the second embodiment of the present invention.

Again, according to the second embodiment of the present invention, the flow of a plurality of events (in the above example, “P1 saw a car and then saw a house while traveling” , “P2 saw the tree and then saw the ship while traveling” joined and branched (in the above example, “P1 and P2 met at the end of the trip”) ) Can also be expressed.

FIG. 28 shows a configuration example of the image processing apparatus 30 according to the second embodiment of the present invention.

11a to 23a and 25a in the figure perform the same processing as 11 to 23 and 25 of FIG. 15, respectively. 11b to 23b and 25b perform the same processing as 11 to 23 and 25 of FIG. 15, respectively. Since description of these processes has already been described, it is omitted here.

In addition, 14a and 14b in the figure are simple databases, and therefore, only one may be used in common.

In FIG. 28, unlike FIG. 15, the image from the input unit 11a is not directly given to the image collection unit 12a but is given via the left trimming unit 31a. Similarly, the image from the input unit 11b is given to the image collection unit 12b via the right trimming unit 31b.

The user interface 24c is the same as 24 in FIG. That is, an instruction from the viewer is input on the user interface 24c. Specifically, an instruction for play mode or pause mode is input. This mode information is sent to both the playback unit 25a and the playback unit 25b.

The image display part (left part) 26a and the image display part (right part) 26b are both the same as 26 in FIG. However, physically, 26a and 26b are disposed at adjacent positions. 26a is arranged on the left side and 26b is arranged on the right side. With this configuration, the viewer is in the “continuous image (picture scroll) data storage unit 23b” and the “continuous image (picture scroll) data storage unit 23b” in the “continuous image (picture scroll) data storage unit 23a”. You can view another “continuous image (picture scroll)” side by side at the same time.

In the second embodiment of the present invention, unlike the first embodiment of the present invention, there are a left trimming portion 31a and a right trimming portion 31b.

Images A10, A11, and A30 captured by P1 are input from the input unit 11a. At this time, it is assumed that A30, which is the last image, is given as accompanying information that it is important.

Similarly, images A20, A21, and A30 captured by P2 are input from the input unit 11b. At this time, it is assumed that A30, which is the last image, is given as accompanying information that it is important.

The left trimming unit 31a does not process any of A10 and A11, and sends the processed image to the subsequent image collection unit 12a. Only for A30, only the area “5/8” is left from the left side, the area “3/8” on the right side is deleted, and then the image is sent to the subsequent image collection unit 12a. That is, as shown in FIG. 29A, an image A31 in which the black portion on the right side of A30 is deleted is created and sent to the subsequent image collection unit 12a.

Similarly, the right trimming unit 31b does not perform any processing on A20 and A21, and sends it to the subsequent image collecting unit 12b as it is. Only for A30, only the area “5/8” is left from the right side, and the area “3/8” on the left side is deleted, and then sent to the subsequent image collection unit 12b. That is, as shown in FIG. 29A, an image A32 in which the black part on the left side of A30 is deleted is created and sent to the subsequent image collection unit 12b.

Here, deleting the “3/8” area and leaving only the “5/8” area will be described. In (a) and (b) of FIG. 29, rectangular areas 2900 and 2901 are indicated by dotted lines. The rectangular area 2900 is an area that is 60% larger than A31 both vertically and horizontally. Further, the image center of A30 is on the right side of 2900. Actually, the amount to be deleted is determined so that the right side of the 60% rectangular area is the center of the A30 image. This amount is “3/8”. Similarly, the rectangular area 2901 is an area having a size of 60% both vertically and horizontally with respect to A32. Further, the image center of A30 is on the left side of 2901. Actually, the amount to be deleted is determined so that the left side of the 60% rectangular area is the center of the A30 image. This amount is “3/8”.

As will be described later, when data is reproduced (displayed), the image data that is finally displayed on the image display unit (left part) 26a is image data in the region 2900. The image data finally displayed on the image display part (right part) 26b is image data in the area 2901. Since the image display part (left part) 26a and the image display part (right part) 26b are physically adjacent to each other, the viewer finally has a rectangular area 2902 shown in FIG. I will appreciate it. Here, the rectangular area 2902 is an area obtained by combining the rectangular area 2901 and the rectangular area 2902.

By means of 11a, the left trimming unit 31a, and 12a to 22a, for example, a “continuous image (picture scroll)” as shown in FIG. 23a ". Here, it is assumed that all images of A10, A11, and A31 (A30 trimmed image) are adopted without being discarded. Further, since A30 (that is, A31) is given as accompanying information that it is important, it is not discarded in a series of processes. In the playListFile, display the area of the display frame indicated by “3000 (see (a) in FIG. 30) first, and then move the display frame in the direction of the arrow 3010 as the time advances and display it. Then, after reaching the display frame area indicated by 3001, the display frame is displayed while moving the display frame in the direction of the arrow 3011. Finally, the display frame area indicated by 3002 is displayed. Information is displayed ", and a series of displays are finished." As described in the first embodiment, the display frame is 60% of the image size. Therefore, the area 3002 in FIG. 30A matches 2900 in FIG.

Similarly, 11b, right trimming unit 31b, and 12b to 22b can store, for example, a “continuous image (picture scroll)” as shown in FIG. Stored in the section 23b ". Here, it is assumed that all images A20, A21, and A32 (A30 trimmed image) are adopted without being discarded. Further, since A30 (that is, A32) is given as accompanying information that it is important, it is not discarded in a series of processes. In the playListFile, display the area of the display frame indicated by “3020 (see (a) in FIG. 30) first”, and move the display frame in the direction of the arrow 3030 as the time advances. Then, after reaching the display frame area indicated by 3021, the display frame is displayed while moving the display frame in the direction of arrow 3031. Finally, the display frame area indicated by 3022 is displayed. Information is displayed ", and a series of displays are finished." As described in the first embodiment, the display frame is 60% of the image size. Therefore, the area 3022 in FIG. 30A corresponds to 2901 in FIG.

Although FIG. 8, FIG. 9 and FIG. 10 show the parting lines, FIG. 30 does not show the parting lines in order to avoid complications.

Next, when an instruction for the play mode is input from the user interface 24c, playback is performed according to the playListFile in the “data storage unit 23a for a series of images (picture scrolls)”, and the image display unit (left part) 26a. An appropriate image is displayed. At the same time, playback is performed in accordance with playListFile in the “continuous image (picture scroll) data storage unit 23b”, and an appropriate image is displayed on the image display unit (right part) 26b. Finally, a display frame area 3002 is displayed on the image display section (left section) 26a, and a display frame area 3022 is displayed on the image display section (right section) 26b.

Now, let's consider the process of reproduction as an appreciator. By looking at the left side, you can recall in turn that “I saw a car, looked at my house, and finally P1 and P2 met while traveling.” And by looking at the right side, you can recall in turn that “I saw the trees during the trip, and even the boat, and finally P1 and P2 met.” Since each image is connected smoothly, you can “feel the continuity of travel”. Further, the last displayed image is an area indicated by 2902 (see FIG. 29C). That is, since a series of images smoothly continued on the left display unit and a series of images smoothly continued on the right display unit are presented as one still image (2902), 2 You can also feel that two trips have merged. Actually, P1 and P2 match at the end, so the expression method that you can feel that P1 and P2 travel merged at the end is very wonderful.

Now, the merging of two image sequences has been described, but it is of course possible to branch. For example, contrary to the above description, it is assumed that P1 and P2 first meet and P1 and P2 photograph themselves (still image A30 in FIG. 27). Then, suppose that P1 and P2 are separated and have made separate trips. That is, P1 is assumed to have taken A11 and A10 in order during the trip. Assume that P2 has taken A21 and A20 in order during the trip. By inputting the images A30, A11 and A10 taken by P1 from the input unit 11a and the images A30, A21 and A20 taken by P2 from the input unit 11b, the arrows shown in FIG. 30 are reversed. It is possible to display in the direction of. In this case, first, one still image (2902) is presented, and an image using A11 or A10 is displayed on the left side of the display unit. Then, images using A21 and A20 are displayed on the right side of the display unit.

As shown in the second embodiment, by using a plurality of the first embodiments of the present invention in combination, a plurality of events can be displayed in parallel for each display area. Furthermore, it is also possible to express that a plurality of events are merged or branched.

This is the end of the description of the processing contents in the second embodiment of the present invention.

In the above description, the shooting time is used as the order of images. This is because the order in which viewers generally recall is time series. However, the present invention is not limited to the shooting time. For example, assume that a zoo is photographed while walking along a route defined by the zoo. Suppose that there was an animal that failed to shoot, so after taking a round along the route, go straight to the animal that failed to shoot and shoot. In this case, if “continuous images (picture scrolls)” are created in the order of the photographing times, the last display is an image of “animals that have failed to be photographed”. In such a case, it is better to create “continuous images (picture scrolls)” in the order according to the route. Therefore, the order may be determined not by the shooting time but by the shooting position. This is because the route specified by the zoo is usually ordered in terms of position, and therefore “ordering by route” is equivalent to “ordering at the shooting position”.

In the above description, it has been described that the incidental information includes the photographing time and the photographing position. Normally, when shooting with a digital camera, images are automatically numbered in ascending order and saved as a file. Therefore, the file name number of each image may be used instead of the shooting time. In addition, as the accompanying information, when there is no shooting position, it is not possible to obtain from the database management unit 13 a “beautiful image for viewing” shot from the same position as the position where the photographer shot. In this case, the user may explicitly search manually from the general-purpose image database 14 and store the image in the input image data storage unit 15.

In addition, the viewer not only displays a series of images (picture scrolls), but also of course the original image input to the input section (the image before editing taken by the traveler (photographer)) Alternatively, the image may be displayed on the image display unit. Some recent displays (image display devices) have a touch panel, and such a display may be used as the image display unit of the present invention. Then, while the “continuous image (picture scroll)” is being displayed, the viewer touches the screen so that the touched image (“continuous image (picture scroll) data storage unit 23”) is displayed. It is also possible to switch to and display an original image (an image before editing that was taken by a traveler (photographer)) having a shooting time substantially the same as that of the still image stored in (1). In other words, if the viewer is interested, the user can view the original image at that time (and the time in the vicinity) by touching the screen, and the traveler's scene at that time can be seen in more detail. You can appreciate it.

<Other Examples of the Present Invention>
In the above-described embodiment, an example in which the “continuous image (picture scroll)” is created and reproduced by the dedicated image processing apparatus 10 or 30 has been described. However, in addition to this, it is also possible to send a plurality of images taken by a traveler (photographer) to a computer device, and generate and reproduce the image in the computer device. In this case, although a detailed description is omitted, a “continuous image (picture scroll)” can be created and reproduced in the computer apparatus by the same process as that in the image processing apparatus 10 or 30 described above. I can do it.

FIG. 31 shows a configuration example of a computer device 40 that creates a “continuous image (picture scroll)” and reproduces it.

The computer apparatus 40 includes a CPU (Central Processing Unit) 4011, a memory 4012, a display controller 4013, an input device interface 4014, and a network interface 4015. Further, the computer device 40 includes an external device interface 4016 and a digital camera interface 4018. These are connected to the bus 4017.

The display 4019 is connected to the bus 4017 via the display controller 4013. A keyboard (KBD) 4020 and a mouse 4021 are connected to the bus 4017 via the input device interface 4014. Further, the hard disk drive (HDD) 4022 and the media drive 4023 are connected to the bus 4017 via the external device interface 4016. The digital camera is connected to the bus 4017 via the digital camera interface 4018. The computer device 40 is connected to a network such as the Internet via a network interface 4015.

The CPU 4011 is a main controller of the computer device 40. The CPU 4011 executes various applications under the control of an operating system (OS). For example, the CPU 4011 can execute an application program for processing images (a plurality of images taken by a traveler (photographer)) once downloaded from the digital camera to the hard disk drive 4022. In the application program, a series of processing programs for creating the above-described “continuous image (picture scroll)” and reproducing it are implemented. The CPU 4011 is interconnected with other devices by a bus 4017.

A memory 4012 is a storage device used for storing program codes executed by the CPU 4011 and temporarily storing work data being executed. The memory 4012 includes, for example, a nonvolatile memory such as a ROM and a volatile memory such as a DRAM.

A display controller 4013 is a dedicated controller for actually processing a drawing command issued by the CPU 4011. The drawing data processed in the display controller 4013 is once written in a frame buffer (not shown), for example, and then output on the screen by the display 4019. For example, an image read from the hard disk drive 4022 is displayed on the screen of the display 4019 so that the user can see and enjoy it.

The input device interface 4014 is a device for connecting user input devices such as a keyboard 4020 and a mouse 4021 to the computer device 40. A user can input a command for reproducing an image or the like via the keyboard 4020 or the mouse 4021.

The network interface 4015 connects the computer device 40 to a local network such as a LAN (Local Area Network) and further to a wide area network such as the Internet according to a predetermined communication protocol such as Ethernet (registered trademark).

On the network, a plurality of host terminals and servers (not shown) are connected in a transparent state, and a distributed computing environment is constructed. On the network, distribution services such as software programs and data contents can be provided. For example, image data can be downloaded to the hard disk drive 4022 via a network from another server in which images taken by others are stored.

The digital camera interface 4018 is a device for capturing an image supplied from the digital camera. The external device interface 4016 is a device for connecting external devices such as the hard disk drive 4022 and the media drive 4023 to the computer device 40.

As conventionally known, the hard disk drive 4022 is an external storage device in which a magnetic disk as a storage carrier is fixedly mounted, and is superior to other external storage devices in terms of storage capacity and data transfer speed. The hard disk drive 4022 can also be randomly accessed. Placing the software program on the hard disk drive 4022 in an executable state is called “installation” of the program in the system. Usually, the hard disk drive 4022 stores program codes of an operating system to be executed by the CPU 4011, application programs, device drivers, and the like in a nonvolatile manner. For example, a program for performing a series of processes of creating the above-described “continuous image (picture scroll)” and reproducing it can be installed on the hard disk drive 4022.

The media drive 4023 is a device for loading portable media such as CD (Compact Disc), MO (Magneto-Optical disc), DVD (Digital Versatile Disc) and the like and accessing the data recording surface. The portable media is mainly used for the purpose of backing up software programs, data files, and the like as data in a computer-readable format, and for moving them between systems (that is, including sales, distribution, and distribution). Application programs for performing image processing can be physically distributed and distributed among a plurality of devices by using these portable media.

As mentioned above, although embodiment of this invention was explained in full detail, the concrete structure is not restricted to this embodiment, The design change etc. of the range which does not deviate from the summary of this invention are included.

Furthermore, this technique can also be set as the following structures.

[1]
An image processing apparatus that processes input image data groups and outputs display image data,
A first input unit that inputs a first image data group and uses the first image data group as the input image data group;
An image alignment unit for sorting the image data in the input image data group;
An importance determination unit for determining the importance of each image data in the input image data group;
A connectability determination unit that determines the connectability of image data in the input image data group;
The image data in the input image data group is selected, and the image data determined to be connectable by the connectability determining unit and determined to be important by the importance determining unit are image-aligned. A connection unit for creating the display image data by connecting in the order sorted in the unit,
An image processing apparatus comprising: an output unit that outputs the display image data generated by the connection unit.
[2]
A database part including a plurality of image data;
The image processing apparatus further includes a second input unit that extracts image data related to the first image data group input in the first input unit from the database unit and adds the image data to the input image data group [1]. An image processing apparatus according to 1.
[3]
Of the two pieces of image data connected by the connection unit, the image data having a lower rank in the order sorted by the image alignment unit is referred to as first image data, and the other, that is, an image having a lower rank. If the data is called second image data,
Connection between the image data of the input image data group at the connection unit is as follows.
When the first image data is a still image and the second image data is a moving image, the first image data that is a still image and the first of the second image data that is a moving image Make a connection with the frame,
When the first image data is a moving image and the second image data is a still image, the last frame of the first image data that is a moving image and the second image that is a still image Make a connection to the data,
When the first image data is a moving image and the second image data is a moving image, the last frame of the first image data that is a moving image and the second image data that is a moving image The image processing apparatus according to [1] or [2], wherein connection with the first frame is performed.
[4]
The display image data is generated in the order in which the display image data is displayed over time in the order connected by the connection unit, that is, in ascending order in the order sorted by the image alignment unit. A generation unit;
The image processing apparatus according to any one of [1] to [3], wherein the output unit outputs the reproduction order information together with the display image data.
[5]
A data format having a plurality of image data, a positional relationship between the image data, and data indicating an order to be displayed.
[6]
An image processing method for processing an input image data group and outputting display image data,
A first input step of inputting a first image data group and using the first image data group as the input image data group;
An image alignment step of sorting each image data in the input image data group;
An importance determination step for determining the importance of each image data in the input image data group;
A connectability determination step for determining connectability between image data in the input image data group;
The image data in the input image data group is selected and the image data determined to be connectable in the connectability determination step and the image data determined to be important in the importance determination step are arranged in the image alignment. A connection step of creating the display image data by connecting in the order sorted in the step;
And an output step of outputting the display image data created in the connection step.
[7]
An image processing program that processes an input image data group and outputs display image data,
A first input step of inputting a first image data group and using the first image data group as the input image data group;
An image alignment step of sorting each image data in the input image data group;
An importance determination step for determining the importance of each image data in the input image data group;
A connectability determination step for determining connectability between image data in the input image data group;
The image data in the input image data group is selected and the image data determined to be connectable in the connectability determination step and the image data determined to be important in the importance determination step are arranged in the image alignment. A connection step of creating the display image data by connecting in the order sorted in the step;
A program that causes a computer to execute processing including an output step of outputting the display image data created in the connection step.
[8]
An image processing apparatus for displaying a first still image and a first moving image,
The first still image and the first frame of the first moving image are arranged so as to intersect with each other, and after the first still image is mainly displayed, the display area is moved with time, Display the first frame of the first video, and then display the second and subsequent frames of the first video in sequence after the next time,
Or
The last frame of the first moving image and the first still image are arranged so that there is an intersection, and from the first frame of the first moving image to the second frame from the end of the first moving image. The images are sequentially displayed, and at the next time, the last frame of the first moving image is mainly displayed, and the display area is moved with time to mainly display the first still image. Or
An image processing apparatus [9] including a display unit that performs at least one of the above [9]
An image processing method for displaying a first still image and a first moving image,
The first still image and the first frame of the first moving image are arranged so as to intersect with each other, and after the first still image is mainly displayed, the display area is moved with time, Display the first frame of the first video, and then display the second and subsequent frames of the first video in sequence after the next time,
Or
The last frame of the first moving image and the first still image are arranged so that there is an intersection, and from the first frame of the first moving image to the second frame from the end of the first moving image. The images are sequentially displayed, and at the next time, the last frame of the first moving image is mainly displayed, and the display area is moved with time to mainly display the first still image. Or
An image processing method [10] comprising a display processing step for performing at least one of the above
An image processing program for displaying a first still image and a first moving image,
The first still image and the first frame of the first moving image are arranged so as to intersect with each other, and after the first still image is mainly displayed, the display area is moved with time, Display the first frame of the first video, and then display the second and subsequent frames of the first video in sequence after the next time,
Or
The last frame of the first moving image and the first still image are arranged so that there is an intersection, and from the first frame of the first moving image to the second frame from the end of the first moving image. The images are sequentially displayed, and at the next time, the last frame of the first moving image is mainly displayed, and the display area is moved with time to mainly display the first still image. Or
A program for causing a computer to execute processing including at least a display processing step for performing any one of the above.

DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus 11, 11a, 11b ... Input part 12, 12a, 12b ... Image collection part 13, 13a, 13b ... Database management part 14, 14a, 14b ... General-purpose image database 15, 15a, 15b ... Input image data storage 16, 16 a, 16 b... Connectability determination unit 17, 17 a, 17 b... Directed graph creation unit 18, 18 a, 18 b... Route search unit 19, 19 a, 19 b. 21a, 21b ... Rendering information creation units 22, 22a, 22b ... Display information creation units 23, 23a, 23b ... Data storage units 24, 24c for continuous images (picture scrolls) ... User interface units 25, 25a, 25b ... Playback units 26, 26a, 26b ... image display unit 30 ... image processing devices 31a, 31b ... trimming unit 40 ... Computer equipment

Claims (10)

An image processing apparatus that processes input image data groups and outputs display image data,
A first input unit that inputs a first image data group and uses the first image data group as the input image data group;
An image alignment unit for sorting the image data in the input image data group;
An importance determination unit for determining the importance of each image data in the input image data group;
A connectability determination unit that determines the connectability of image data in the input image data group;
The image data in the input image data group is selected, and the image data determined to be connectable by the connectability determining unit and determined to be important by the importance determining unit are image-aligned. A connection unit for creating the display image data by connecting in the order sorted in the unit,
An image processing apparatus comprising: an output unit that outputs the display image data generated by the connection unit. A database part including a plurality of image data;
The image processing apparatus further comprises: a second input unit that takes out image data related to the first image data group input in the first input unit from the database unit and adds the image data to the input image data group. An image processing apparatus according to 1. Of the two pieces of image data connected by the connection unit, the image data having a lower rank in the order sorted by the image alignment unit is referred to as first image data, and the other, that is, an image having a lower rank. If the data is called second image data,
Connection between the image data of the input image data group at the connection unit is as follows.
When the first image data is a still image and the second image data is a moving image, the first image data that is a still image and the first of the second image data that is a moving image Make a connection with the frame,
When the first image data is a moving image and the second image data is a still image, the last frame of the first image data that is a moving image and the second image that is a still image Make a connection to the data,
When the first image data is a moving image and the second image data is a moving image, the last frame of the first image data that is a moving image and the second image data that is a moving image The image processing apparatus according to claim 1, wherein connection with the first frame is performed. The display image data is generated in the order in which the display image data is displayed over time in the order connected by the connection unit, that is, in ascending order in the order sorted by the image alignment unit. A generation unit;
The image processing apparatus according to claim 1, wherein the output unit outputs the reproduction order information together with the display image data. A data format having a plurality of image data, a positional relationship between the image data, and data indicating an order to be displayed. An image processing method for processing an input image data group and outputting display image data,
A first input step of inputting a first image data group and using the first image data group as the input image data group;
An image alignment step of sorting each image data in the input image data group;
An importance determination step for determining the importance of each image data in the input image data group;
A connectability determination step for determining connectability between image data in the input image data group;
The image data in the input image data group is selected and the image data determined to be connectable in the connectability determination step and the image data determined to be important in the importance determination step are arranged in the image alignment. A connection step of creating the display image data by connecting in the order sorted in the step;
And an output step of outputting the display image data created in the connection step. An image processing program that processes an input image data group and outputs display image data,
A first input step of inputting a first image data group and using the first image data group as the input image data group;
An image alignment step of sorting each image data in the input image data group;
An importance determination step for determining the importance of each image data in the input image data group;
A connectability determination step for determining connectability between image data in the input image data group;
The image data in the input image data group is selected and the image data determined to be connectable in the connectability determination step and the image data determined to be important in the importance determination step are arranged in the image alignment. A connection step of creating the display image data by connecting in the order sorted in the step;
A program that causes a computer to execute processing including an output step of outputting the display image data created in the connection step. An image processing apparatus for displaying a first still image and a first moving image,
The first still image and the first frame of the first moving image are arranged so as to intersect with each other, and after the first still image is mainly displayed, the display area is moved with time, Display the first frame of the first video, and then display the second and subsequent frames of the first video in sequence after the next time,
Or
The last frame of the first moving image and the first still image are arranged so that there is an intersection, and from the first frame of the first moving image to the second frame from the end of the first moving image. The images are sequentially displayed, and at the next time, the last frame of the first moving image is mainly displayed, and the display area is moved with time to mainly display the first still image. Or
An image processing apparatus comprising a display unit that performs at least one of the above An image processing method for displaying a first still image and a first moving image,
The first still image and the first frame of the first moving image are arranged so as to intersect with each other, and after the first still image is mainly displayed, the display area is moved with time, Display the first frame of the first video, and then display the second and subsequent frames of the first video in sequence after the next time,
Or
The last frame of the first moving image and the first still image are arranged so that there is an intersection, and from the first frame of the first moving image to the second frame from the end of the first moving image. The images are sequentially displayed, and at the next time, the last frame of the first moving image is mainly displayed, and the display area is moved with time to mainly display the first still image. Or
An image processing method comprising a display processing step for performing at least one of the above An image processing program for displaying a first still image and a first moving image,
The first still image and the first frame of the first moving image are arranged so as to intersect with each other, and after the first still image is mainly displayed, the display area is moved with time, Display the first frame of the first video, and then display the second and subsequent frames of the first video in sequence after the next time,
Or
The last frame of the first moving image and the first still image are arranged so that there is an intersection, and from the first frame of the first moving image to the second frame from the end of the first moving image. The images are sequentially displayed, and at the next time, the last frame of the first moving image is mainly displayed, and the display area is moved with time to mainly display the first still image. Or
A program for causing a computer to execute processing including at least a display processing step for performing any one of the above.

Images