WO2015146242A1 - Image processing device - Google Patents

Abstract

Provided is an image processing device for classifying each frame into a channel when frames belonging to different channels are mixed within a single video. The image processing device is provided with an automatic classification unit. The automatic classification unit calculates the similarities between a plurality of frames in the video on the basis of local color trends and/or overall color trends by applying image processing to the frames and classifies the plurality of frames into a plurality of channels on the basis of the similarities.

Description

Image processing device

The present invention relates to an image processing apparatus and an image processing program, and more particularly to an image processing apparatus and an image processing program for classifying these frames for each channel when frames belonging to different channels are mixed in one moving image. .

An intermittent recording method called time lapse is known as a method for recording moving images. This is a method of recording at a frame rate lower than normal, that is, dropping frames, and enables recording on the same recording medium for a longer time than usual. Also in security cameras, this time lapse method is often employed, and further, images from other security cameras may be inserted in the gap time formed by lowering the frame rate. In this case, a moving image is recorded in which videos from a plurality of cameras, that is, videos from a plurality of channels are alternately switched along one timeline.

¡Videos that are recorded from multiple channels on a single timeline can be displayed on a monitor for each camera, that is, for each channel, using a dedicated device. In the dedicated device of Patent Document 1, channel number data is assigned to each frame of the video, and at the time of playback, by referring to this, a video of an arbitrary channel can be selected from videos in which videos of a plurality of channels are mixed. Can only be taken out.

JP 2002-319210 A

However, when a video in which videos of multiple channels are mixed is played on a general playback device, the video for each channel cannot be viewed. This is because the specifications for recognizing the channel of each frame differ for each manufacturer or for each model, and general playback devices do not support these specifications.

The present invention relates to an image processing apparatus that enables reproduction for each channel, even if frames belonging to different channels are mixed in one moving image, no matter what specifications the moving image is recorded in. An object is to provide an image processing program.

An image processing apparatus according to a first aspect of the present invention is an image processing apparatus for classifying the frames into the respective channels when frames belonging to different channels are mixed in one moving image, and an automatic classification unit Is provided. The automatic classification unit calculates a similarity between frames based on a local color tendency and / or an overall color tendency by performing image processing on a plurality of frames included in the moving image, and the similarity The plurality of frames are classified into a plurality of channels.

Here, when frames belonging to different channels are mixed in one moving image, a function for classifying these frames for each channel is provided. Specifically, first, image processing is performed on a plurality of frames included in a moving image to calculate the similarity between frames, and the plurality of frames are automatically classified into a plurality of channels according to this. . Here, the similarity between frames is determined based on local and / or overall color tendency in the frame screen. Therefore, according to this function, a video in which frames belonging to different channels are mixed into a video for each channel based on the local and / or overall color tendency in the frame screen regardless of the video specifications. And can be separated. That is, in the case where frames belonging to different channels are mixed in one moving image, it is possible to reproduce for each channel regardless of the specifications of the moving image recorded.

An image processing apparatus according to a second aspect of the present invention is the image processing apparatus according to the first aspect, and the automatic classification unit includes a setting unit, a calculation unit, and a determination unit. The setting unit sets a specific frame included in the moving image or a frame obtained by combining two or more specific frames included in the moving image as a reference frame, and sets another specific frame included in the moving image. Set as a comparison frame. The calculation unit divides the comparison frame and the reference frame into a plurality of partial areas, calculates a first color index for each partial area, and based on the first color index, the comparison frame and the reference frame And the degree of similarity between the composition of the comparison frame and the composition of the reference frame is calculated. The determination unit determines whether the comparison frame belongs to the same channel as the reference frame or a frame from which the comparison frame is combined based on the similarity.

Note that “dividing a frame into a plurality of partial areas” includes not only dividing the entire frame into a plurality of partial areas without overlapping, but also a part of the entire frame (for example, excluding the outer frame area). And the like are divided into a plurality of partial areas without overlapping, and the entire frame or part of the frame is divided into a plurality of partial areas partially overlapping.

Here, the composition of each of the comparison frame and the reference frame is derived based on the color tendency, and it is determined whether or not both frames are from the same channel according to the similarity of both compositions. The composition is an arrangement of each element (for example, a target object and a background) that appears in the frame screen. Therefore, based on the composition of the frame, it can be determined whether the compared frames belong to the same channel.

An image processing apparatus according to a third aspect of the present invention is the image processing apparatus according to the first aspect, and the automatic classification unit includes a setting unit, a calculation unit, and a determination unit. The setting unit sets a specific frame included in the moving image or a frame obtained by combining two or more specific frames included in the moving image as a reference frame, and sets another specific frame included in the moving image. Set as a comparison frame. The calculation unit divides the comparison frame and the reference frame into a plurality of partial areas, calculates a first color index for each partial area, and based on the first color index, the comparison frame and the reference frame And comparing the comparison frame and the reference frame into a plurality of new partial areas based on the composition, and at least one of the new partial areas for each of the comparison frame and the reference frame. Then, a second color index is calculated, and the similarity between the second color index of the comparison frame and the second color index of the reference frame is calculated. The determination unit determines whether the comparison frame belongs to the same channel as the reference frame or a frame from which the comparison frame is combined based on the similarity.

Here, the composition of each of the comparison frame and the reference frame is derived based on the color tendency. Further, the comparison frame and the reference frame are divided into a plurality of partial areas based on the composition. Then, whether or not both frames are derived from the same channel is determined according to the similarity between the color tendency in the partial area of the comparison frame and the color tendency in the partial area of the reference frame. Therefore, it can be determined whether or not the compared frames belong to the same channel based on the color tendency considering the composition of the frames.

An image processing apparatus according to a fourth aspect of the present invention is the image processing apparatus according to the first aspect, and the automatic classification unit includes a setting unit, a calculation unit, and a determination unit. The setting unit sets a specific frame included in the moving image or a frame obtained by combining two or more specific frames included in the moving image as a reference frame, and sets another specific frame included in the moving image. Set as a comparison frame. The calculation unit divides the comparison frame and the reference frame into a plurality of partial areas, calculates a first color index for each partial area, and based on the first color index, the comparison frame and the reference frame And comparing the comparison frame and the reference frame into a plurality of new partial areas based on the composition, and for each of the comparison frame and the reference frame, at least 2 of the new partial areas. A second color index is calculated, a third color index that is a difference between the second color indices is calculated for each of the comparison frame and the reference frame, and the third color index of the comparison frame The similarity with the third color index of the reference frame is calculated. The determination unit determines whether the comparison frame belongs to the same channel as the reference frame or a frame from which the comparison frame is combined based on the similarity.

Here, the composition of each of the comparison frame and the reference frame is derived based on the local color tendency. Further, the comparison frame and the reference frame are divided into a plurality of partial areas based on the composition. Whether or not both frames come from the same channel according to the similarity between the difference in color index between the plurality of partial areas in the comparison frame and the difference in color index between the plurality of partial areas in the reference frame Is judged. Note that the color index difference between the plurality of partial areas is, for example, a saturation difference or a density difference in the plurality of partial areas. Therefore, it can be determined whether or not the compared frames belong to the same channel based on the color tendency considering the composition of the frames.

An image processing apparatus according to a fifth aspect of the present invention is the image processing apparatus according to any one of the first to fourth aspects, wherein the automatic classification unit includes at least one of density, brightness, luminance, saturation, and hue. Based on one, the color tendency is evaluated.

Here, the color tendency of the frame can be determined based on at least one of density, saturation, and hue.

An image processing program according to a sixth aspect of the present invention is an image processing program for classifying the frames into the channels when frames belonging to different channels are mixed in one moving image. By performing image processing on a plurality of included frames, a similarity between frames regarding a local color tendency and / or an overall color tendency is calculated, and the plurality of frames are converted into a plurality of frames according to the similarity. Let the computer perform the step of classifying into channels. Here, the same effect as the first aspect can be achieved.

According to the present invention, regardless of the specifications of a moving image, a moving image in which frames belonging to different channels are mixed is separated into moving images for each channel based on the local and / or overall color tendency in the frame screen. It becomes possible to do. That is, in the case where frames belonging to different channels are mixed in one moving image, it is possible to reproduce for each channel regardless of the specifications of the moving image recorded.

1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention. The figure of the basic screen before image data is taken in. The figure of a basic screen after image data was taken in. The figure which shows the still image group which belongs to 1 timeline. The figure which shows an area setting window. The figure which shows a reclassification window. Another figure showing a reclassification window. The figure which shows a confirmation window. The flowchart which shows the flow of the automatic classification process which concerns on 1st Embodiment of this invention. The conceptual diagram explaining the method of calculating the similarity which concerns on 1st Embodiment of this invention. The flowchart which shows the flow of the automatic classification process which concerns on 2nd Embodiment of this invention. The conceptual diagram explaining the method of calculating the similarity which concerns on 2nd Embodiment of this invention. The flowchart which shows the flow of the automatic classification | category process which concerns on the 3rd Embodiment of this invention. The conceptual diagram explaining the method of calculating the similarity which concerns on 3rd Embodiment of this invention. The conceptual diagram explaining the calculation method of another similarity based on 3rd Embodiment of this invention. The conceptual diagram explaining the method of the calculation of another similarity based on 3rd Embodiment of this invention. The flowchart which shows the flow of the process which evaluates the color tendency which concerns on the 3rd Embodiment of this invention. The figure explaining the information which defines an impression degree.

Hereinafter, an image processing apparatus and an image processing program according to some embodiments of the present invention will be described with reference to the drawings.

<1. First Embodiment>
<1-1. Overview of Image Processing Device>
An image processing apparatus 1 shown in FIG. 1 is an embodiment of an image processing apparatus according to the present invention. The image processing apparatus 1 is a general-purpose personal computer. In the image processing apparatus 1, an image processing program 2 which is an embodiment of the image processing program according to the present invention is stored in a computer-readable recording medium 60 such as a CD-ROM, a DVD-ROM, or a USB memory. Etc. are provided and installed by etc. The image processing program 2 is application software for supporting image processing for moving images and still images. The image processing program 2 causes the image processing apparatus 1 to execute steps included in the operations described later.

The image processing apparatus 1 includes a display 10, an input unit 20, a storage unit 30, and a control unit 40. These units 10 to 40 are connected to each other via a bus line, a cable 5 or the like and can communicate appropriately. The display 10 is composed of a liquid crystal display or the like, and displays a screen or the like described later to the user. The input unit 20 includes a mouse, a keyboard, and the like, and accepts an operation from the user for the image processing apparatus 1. The storage unit 30 is a non-volatile storage area configured from a hard disk or the like. The control unit 40 includes a CPU, a ROM, a RAM, and the like.

The image processing program 2 is stored in the storage unit 30. A software management area 50 is secured in the storage unit 30. The software management area 50 is an area used by the image processing program 2. In the software management area 50, an original image area 51, a processed file area 52, and an impression degree definition area 53 are secured. The role of each of the areas 51 to 53 will be described later.

The control unit 40 virtually operates as the automatic classification unit 41 and the reclassification unit 45 by reading and executing the image processing program 2 stored in the storage unit 30. The automatic classification unit also operates as the setting unit 42, the calculation unit 43, and the determination unit 44. The operation of each part 41 to 45 will be described later.

<1-2. Details of Configuration and Operation of Image Processing Device>
When the control unit 40 detects that the user has performed a predetermined operation via the input unit 20, the control unit 40 activates the image processing program 2. When the image processing program 2 is activated, a basic screen W1 (see FIG. 2) is displayed on the display 10. Note that the display of all elements such as a screen, a window, a button, and the like displayed on the display 10 is controlled by the control unit 40.

<1-2-1. Importing image data>
The basic screen W1 accepts an instruction for taking image data into the original image area 51 from the user. The image data taken into the original image area 51 is a target of reproduction processing, image processing, and channel separation processing described later. The control unit 40 captures image data from the still image file or the moving image file into the original image area 51. In this specification, a still image file is a data file in a still image format, and a moving image file is a data file in a moving image format.

When capturing image data from a still image file, the user operates the input unit 20 to specify one still image file or one folder. In the former case, the control unit 40 causes the user to input the address path and file name in the storage unit 30 of the still image file. In the latter case, the control unit 40 causes the user to input the address path and folder name in the storage unit 30 of the folder. Thereafter, the control unit 40 stores the specified still image file or all the still image files in the specified folder as a still image file group in the original image area 51. In the present specification, when referring to a “group”, the number of elements is not limited to a plurality, and may be one.

On the other hand, when capturing image data from a moving image file, the user operates the input unit 20 to input the address path and file name in the storage unit 30 of one moving image file. When the control unit 40 detects that the user has specified a moving image file, the control unit 40 displays a moving image capturing window (not shown) in an overlapping manner on the basic screen W1. The moving image capture window accepts selection of an arbitrary section from the user among all the sections of the timeline of the specified moving image file. When the control unit 40 detects that the user has selected a specific section via the input unit 20, the control unit 40 generates a still image file group corresponding to the frame group included in the section of the designated moving image file on a one-to-one basis. To do. Thereafter, the control unit 40 stores the still image file group in the original image area 51. Therefore, in the present embodiment, the image data to be subjected to playback processing, image processing, and channel separation processing, which will be described later, is not a moving image file but a still image file.

Note that the control unit 40 sets the still image file group to one time not only when the still image file group captured in the original image area 51 is derived from the moving image file but also from the still image file. Recognize that they are arranged along a line. The array is automatically given from the attribute of the file.

<1-2-2. Playback processing>
When the still image file group is taken into the original image area 51, the control unit 40 displays the display window W2 (see FIG. 3) on the basic screen W1. The display windows W2 are created by the number of timelines of the still image file group taken into the original image area 51.

In the display window W2, first, one still image file (for example, a still image file of the first frame on the timeline) included in the still image file group captured in the original image area 51 is displayed. Thereafter, as will be described later, the frame displayed in the display window W2 is switched in response to a user operation.

The control unit 40 can reproduce a frame group belonging to the timeline corresponding to the display window W2 as a moving image in the display window W2. Here, as shown in FIG. 3, a window selection pull-down menu T1, a playback button T2, a frame advance button T3, a frame return button T4, and a timeline bar T5 are arranged on the basic screen W1.

Even if there are a plurality of display windows W2, there is only one active display window W2. The window selection pull-down menu T1 accepts selection of which display window W2 is active from the user. Hereinafter, a timeline corresponding to the active display window W2 is referred to as an active timeline, and a frame group belonging to the active timeline is referred to as an active frame group. A frame currently displayed in the active display window W2 is referred to as an active display frame.

The playback button T2 receives a playback command from the user as a movie of the active frame group. When the control unit 40 detects that the user has pressed the play button T2 via the input unit 20, the frame included in the active frame group is sequentially framed along the timeline in the active display window W2. Display in format. Note that the reproduction starts from the active display frame at the time when the reproduction button T2 is pressed. The playback button T2 accepts a playback stop command from the user. When the control unit 40 detects that the user has pressed the playback button T2 via the input unit 20 during playback, the control unit 40 fixes the display in the active display window W2 to the active display frame at that time.

The frame advance button T3 and the frame return button T4 each receive a command to switch the active display frame to the previous frame one by one along the active timeline.

The timeline bar T5 is an object that schematically shows the active timeline. The timeline bar T5 is equally divided by the number of frames of the active frame group in the direction in which the bar extends. The nth divided region from the left on the timeline bar T5 corresponds to the nth frame on the active timeline (n is a natural number).

As shown in FIG. 3, the timeline bar T5 displays the divided area A1 corresponding to the selected frame group and the divided area A2 corresponding to the non-selected frame group in different modes. The selected frame group is a frame group corresponding to the currently selected section on the active timeline. The non-selected frame group is a frame group corresponding to a section that is not currently selected on the active timeline.

The timeline bar T5 accepts selection of an arbitrary section on the active timeline from the user. In other words, the user can select any number of arbitrary frames from the active frame group by operating the divided areas on the timeline bar T5 via the input unit 20. The control unit 40 recognizes the selected frame group as an object of image processing and channel separation processing described later. Each time the divided area on the timeline bar T5 is selected by the user, the active display frame is switched to a frame corresponding to the most recently selected divided area.

<1-2-3. Image processing>
Hereinafter, image processing for the selected frame group will be described. The control unit 40 can execute a plurality of image processing modules such as noise removal, sharpness, brightness / contrast / saturation adjustment, image resolution, rotation, addition of characters / arrows / mosaic, and image averaging. The image processing module is incorporated in the image processing program 2.

The user can select any one of the image processing modules in any order and any number of times by operating the basic screen W1 via the input unit 20. Each time the control unit 40 detects that the user has selected an image processing module, the control unit 40 executes the image processing module on the selected frame group.

As the image processing module is sequentially executed once, twice, three times,... With respect to the frame, the frame is sequentially sorted into the first order, the second order, the third order,. It will be processed. The 0th frame corresponds to the still image file stored in the original image area 51. The (m + 1) th frame corresponds to the still image file after the image processing module is executed once for the still image file corresponding to the mth frame (m is an integer of 0 or more). The control unit 40 sequentially generates still image files corresponding to the first and subsequent frames, and separately stores these still image files in the processed file area 52.

FIG. 4 is a conceptual diagram showing how still image groups belonging to one timeline are managed by the image processing program 2. In FIG. 4, the N axis on the horizontal axis indicates the order of frames on the timeline, and the M axis on the vertical axis indicates the order of processing. A square corresponding to the coordinates (n, m) in the NM space in FIG. 4 represents the still image Q (n, m). The still image Q (n, m) is an mth-order still image of the nth frame on the timeline (n is a natural number and m is an integer of 0 or more).

The control unit 40 manages the value of the currently selected coordinate m as the parameter m _s for each frame. Immediately after the still image file group is taken into the original image area 51, the coordinate m _s has an initial value of 0. Thereafter, each time the image processing module is executed once, the coordinate m _s of the frame is incremented by one. Further, the user can freely change the coordinate m _s of the selected frame group by performing a predetermined operation via the input unit 20. Note that to execute the image processing module to the frame, it is to perform an image processing module to the m _s next still image of the frame. Therefore, changing the coordinate m _s means changing the execution target of the image processing module. In addition, displaying a frame means displaying a still image at the coordinate m _s of the frame. Therefore, changing the coordinate m _s also means changing the object displayed in the active display window W2.

<1-3. Channel separation processing>
Hereinafter, channel separation processing implemented in the image processing program 2 will be described. The channel separation process is a process of separating a moving image in which videos belonging to different channels are mixed into moving images for each channel. In other words, a moving image to be subjected to channel separation processing typically includes a plurality of moving images recorded in a time-lapse manner, and one time in a format in which video of another moving image is embedded in a gap time of each moving image. A moving image mixed on a line (hereinafter referred to as a mixed time-lapse moving image). A mixed time-lapse moving image is a moving image in which videos of a plurality of channels are alternately switched along a timeline.

The channel separation process is executed for the selected frame group, but a plurality of frames are required to execute the channel separation process. Therefore, in a state where only one frame is selected as the selected frame group, it is assumed that the operation button for starting the channel separation process is invalid and the channel separation process cannot be started. Alternatively, even if only one frame is selected as the selected frame group, if there are a plurality of frames on the active timeline, all the frames on the active timeline are subject to channel separation processing. You can also Hereinafter, a frame group to be subjected to channel separation processing is referred to as a target frame group. In addition, a moving image composed of target frame groups is referred to as a target moving image.

The channel separation process includes an automatic classification process that automatically classifies a plurality of frames included in the target moving image into a plurality of channels, and a reclassification process in which the user corrects the classification result by the automatic classification process by manual operation.

The automatic classification process is executed by the automatic classification unit 41. The automatic classification process is a process of automatically classifying these frames into a plurality of channels by performing image processing on a plurality of frames included in the target moving image. Specifically, the automatic classification unit 41 calculates the similarity between frames, and classifies a plurality of frames included in the target moving image into a plurality of channels according to the similarity. The similarity between frames is calculated as an indicator that the compared frames belong to the same channel. Although details of the automatic classification process will be described later, a plurality of channels are detected as a result of the automatic classification process. A plurality of frames belong to each channel. In addition, as long as a mixed time-lapse moving image is targeted, a plurality of channels are usually detected. However, only one channel may be detected as a result of the automatic classification process. Further, in the automatic classification process, a label that is classified as “unclassified” is assigned to a frame that is determined not to belong to any channel. Therefore, as a result of the automatic classification process, each frame included in the target moving image is given a channel name of one of the channels or one of “unclassified” labeling.

The reclassification process is executed by the reclassification unit 45. The reclassification process is a process of individually classifying arbitrary frames included in the target moving image into arbitrary channels in accordance with a user operation. Specifically, in the reclassification process, unclassified frames that have not been classified into any channel by the automatic classification process can be individually classified into specific channels. In addition, a frame that has been classified into an incorrect channel by the automatic classification process can be classified into another correct channel. Therefore, when the user feels that the result of the automatic classification process is strange, the user can correct the channel to which the problematic frame belongs in units of frames. In the reclassification process, it is also possible to create a new channel or integrate a plurality of channels into one channel according to a user operation. That is, the reclassification process is used for manual correction of the result of the automatic classification process.

When the automatic classification unit 41 detects that the user has performed a predetermined operation via the input unit 20, the automatic classification unit 41 starts the channel separation process. When the channel separation process is started, the setting unit 42 first displays an area setting window W3 (see FIG. 5) for displaying the selected frame on the basic screen W1. The area setting window W3 is a screen that accepts designation of an area serving as a reference for calculating similarity between frames in the automatic classification process within the selected frame. The selected frame is one frame included in the target frame group. For example, the selected frame can be a top frame along the timeline in the target frame group or an active display frame. As is clear from what has already been described, the active display frame is the frame most recently selected on the timeline bar T5, and thus is always included in the target frame group.

When the target video is a security camera video or the like, a date and time display area is often provided in the frame screen (lower left portion in the example of FIG. 5). If the similarity between frames is determined using information on the entire frame including the date / time display area, the similarity may be determined to be low even for frames belonging to the same channel. In some cases, a black edge is provided in the frame screen. On the contrary, in such a case, if the similarity between frames is determined using information on the entire frame including the black edge, there is a possibility that the similarity is determined to be high even for frames belonging to different channels. . In addition, even when the target moving image is a mixed time-lapse moving image composed of images of security cameras installed in different elevators, the similarity between frames belonging to different channels is easily determined. This is because the scenery in the elevator is similar. The area setting window W3 avoids such a situation, and an area that may cause an erroneous determination is displayed for the similarity degree so that the similarity degree between frames calculated by the automatic classification process can be an accurate indicator of belonging to the same channel. Useful for exclusion from areas that are the basis for calculations. Specifically, the user designates an area that characterizes the background of each channel in the selected frame of the area setting window W3. In the example of an elevator, an area in which a poster or the like characterizing the scenery in each elevator is shown may be specified. When the user designates an arbitrary area in the selected frame by operating the mouse or the like on the area setting window W3, a surrounding line 71 indicating the area designated by the user is displayed in the selected frame. Thereby, the user can confirm whether or not a correct area can be designated.

When the start button 72 on the area setting window W3 is pressed by the user, this is detected by the automatic classification unit 41, and automatic classification processing described later is started. At the same time, the area setting window W3 is closed. If the start button 72 is pressed when no area is specified in the selected frame, the entire frame screen is used as a basis for calculating the similarity in the automatic classification process. The cancel button 73 is a button for canceling area designation via the area setting window W3.

* While the automatic classification process is in progress, a dialog box showing the progress of the automatic classification process is displayed. When the automatic classification process ends, the dialog box is closed and the reclassification unit 45 starts the reclassification process. Details of the automatic classification process will be described later. First, the reclassification unit 45 causes the reclassification window W4 shown in FIG. 6 to be superimposed and displayed on the basic screen W1 as a user interface for performing reclassification after the automatic classification process.

On the reclassification window W4, a channel list area C1, a frame list area C2, a playback area C3, and a timeline bar C4 are arranged. In the channel list area C1, channel objects 91 respectively corresponding to the channels detected by the automatic classification process are arranged. In the example of FIG. 6, four channel objects 91 corresponding to the four channels CH01, CH02, CH03, and CH04 are arranged. In the present embodiment, the channel object 91 is in the form of an icon.

As shown in FIG. 6, the channel name 91 of the corresponding channel is displayed on the channel object 91. The channel object 91 displays thumbnail images of representative frames that represent a plurality of frames belonging to the corresponding channel. The representative frame is switched as appropriate in response to a user operation described later. The default is, for example, the first frame along the timeline among the frames belonging to the corresponding channel. Although not shown in FIG. 6, when the number of channels increases and channel objects 91 corresponding to all channels cannot be arranged in the channel list area C1, a scroll bar appears in the area C1. However, the area C1 is substantially enlarged. The channel object 91 also displays the number of frames belonging to the corresponding channel. The number of such frames changes in response to a user operation described later, but the display of the number of frames in the channel object 91 is also switched in real time in accordance with the change.

Further, in addition to the channel object 91, an unclassified object 92 is also arranged in the channel list area C1. The unclassified object 92 is an object corresponding to a classification for collecting unclassified frames that do not belong to any channel in the target frame group. Note that the classification of grouping unclassified frames does not correspond to an actual channel, but in the sense that the frames are grouped, one channel is virtually configured. Therefore, the unclassified object 92 has a form similar to the channel object 91, and is an icon of the same size in this embodiment. Specifically, in the unclassified object 92, “unclassified” is displayed instead of the channel name, and the number of unclassified frames is displayed instead of the number of frames belonging to the channel. The display of the number of images in the unclassified object 92 is also switched in real time in accordance with the change in the number of unclassified frames. The unclassified object 92 also displays thumbnail images of representative frames representing “unclassified” frames. The representative frame is also switched as appropriate in response to a user operation to be described later. The default is, for example, the first frame along the timeline among the frames belonging to the “unclassified” category. In the present embodiment, a boundary line 93 is drawn between the unclassified object 92 and the channel object 91.

The reclassifying unit 45 accepts a selection operation for selecting any one object from all the channel objects 91 and the unclassified objects 92 in the channel list area C1 from the user. Of all the objects 91 and 92, the currently selected object is displayed in a different manner from the other objects. In the example of FIG. 6, the object 91 of the channel CH01 is displayed in a different color from the other objects 91 and 92. Hereinafter, a channel or classification corresponding to one object 91, 92 that is currently selected is referred to as a selected channel.

In the frame list area C2, thumbnail images 94 of all frames belonging to the selected channel are displayed in a list. When the number of frames belonging to the selected channel is large and not all thumbnail images 94 can be arranged in the frame list area C2, a scroll bar 95 appears in the area C2, and the area C2 is substantially subtracted. Expanded to Therefore, the user can selectively display all unclassified frames or all frames belonging to one of the channels in the frame list area C2 by switching the selection state of the objects 91 and 92. it can. Therefore, all the unclassified and classified frames included in the target frame group can be confirmed while effectively using the limited space in the frame list area C2. In particular, according to this list display mode, even if an erroneous frame is classified in each channel, it is easy to find it. In this embodiment, the thumbnail images 94 in the channel list area C1 can be arranged in a multistage manner, and from the left to the right and from the top to the bottom as the order n on the timeline of the original frame increases. Arranged in chronological order.

The reclassifying unit 45 manages a specific single frame belonging to each channel as an active frame, and also manages a specific single unclassified frame as an active frame. In the frame list area C2, a surrounding frame 96 is attached to the thumbnail image 94 of the active frame as a sign for distinguishing it from the thumbnail images 94 of other frames. The thumbnail images 94 in the frame list area C2 are each recognized as an object, and the user can select any one thumbnail image 94 from all the thumbnail images 94 in the frame list area C2 by a click operation or the like. Can be selected. It is also possible to simultaneously select a plurality of thumbnail images 94 in the frame list area C2 by an operation of repeating a click operation while pressing a specific key on the keyboard. Each time the thumbnail image 94 is selected in the frame list area C <b> 2, the reclassifying unit 45 switches the active frame of the selected channel to a frame corresponding to the latest selected thumbnail image 94. At this time, the position of the surrounding frame 96 is also moved. The active frame is linked with the thumbnail images displayed on the channel object 91 and the unclassified object 92, and each time the active frame of each channel or classification is changed, the objects 91, 92 thumbnail images are also switched. That is, in this embodiment, the representative frame of each channel and classification matches the active frame.

The thumbnail image 94 of the active frame in the frame list area C2 can be moved individually so as to be superimposed on one of the objects 91 and 92 in the channel list area C1 by a drag and drop operation. Further, when a plurality of thumbnail images 94 are simultaneously selected in the frame list area C2, the plurality of thumbnail images 94 are collectively put on any one of the objects 91 and 92 in the channel list area C1. It can be moved to overlap. (However, even if it is “move”, the thumbnail image 94 disappears as if it was inserted into the folder when it is released on the target objects 91 and 92.) In other words, the reclassification unit 45 uses the frame list area. An operation of associating an arbitrary thumbnail image 94 in C2 with an arbitrary channel or classification is accepted from the user. When this association operation is performed, the frame corresponding to the thumbnail image 94 moved by the operation is reclassified into a channel or classification corresponding to the objects 91 and 92 to which the thumbnail image 94 is moved. If the frame corresponding to the thumbnail image 94 related to the operation is originally classified into the channel or the classification corresponding to the objects 91 and 92 related to the operation, the operation is ignored and reclassification is not performed. I will not.

As shown in FIG. 6, a newly created object 97 is arranged after the arrangement of the objects 91 and 92 in the channel list area C1. The newly created object 97 is an object for newly creating a channel, and is in the form of an icon in this embodiment. Then, the same operation as the above association operation can be performed not only on the objects 91 and 92 but also on the newly created object 97. Specifically, the reclassification unit 45 accepts an operation of moving the thumbnail image 94 of the active frame in the frame list area C2 individually so as to overlap the newly created object 97 by a drag and drop operation from the user. . Further, when a plurality of thumbnail images 94 are simultaneously selected in the frame list area C2, the plurality of thumbnail images 94 can be moved together so as to overlap the newly created object 97. (However, even if it is “move”, when the thumbnail image 94 is released on the newly created object 97, it disappears as if it was inserted into the folder.) When this association operation is performed, the newly created object 97 is displayed. A channel object 91 is newly created at the position where the.

The newly created channel object 91 is an object representing a new channel whose elements are frames corresponding to one or a plurality of thumbnail images 94 moved by the association operation. In other words, the frame corresponding to one or more thumbnail images 94 moved by the association operation is reclassified into a new channel. Therefore, immediately after the channel object 91 is newly created, if only one thumbnail image 94 has been moved, the thumbnail image 94 is displayed on the object 91. On the other hand, when a plurality of thumbnail images 94 are moved, one of the thumbnail images 94 (for example, the last selected one) is displayed on the newly created channel object 91. . Channel names are also given and displayed as appropriate. As the channel object 91 is newly created, the newly created object 97 moves after the arrangement of the objects 91 and 92 including the newly created object 91.

When the frame is reclassified by the above various association operations, the reclassified frame is labeled with the channel name CH01, CH02,... Or “unclassified” according to the reclassified channel or classification. Will be granted again. Further, the thumbnail image 94 of the reclassified frame is deleted from the frame list area C2. At this time, in the frame list area C2, the remaining thumbnail images 94 are aligned so as to fill the space where the deleted thumbnail images 94 were placed. The reclassified frame always includes an active frame. Therefore, after the reclassification, the active frame of the selected channel is changed to the closest frame among the following frames along the timeline, or the closest frame among the previous frames along the timeline. The change of the active frame of the selected channel is appropriately reflected in the position of the surrounding frame 96 and the display of the thumbnail images on the objects 91 and 92. Further, the number of frames belonging to the channel and / or classification corresponding to the objects 91 and 92 of the reclassified frame before and after the movement is recalculated and reflected in the display of the objects 91 and 92.

In addition, the above various association operations can be realized in modes other than the drag and drop operation. Specifically, as shown in FIG. 6, the objects 91, 92, and 97 in the frame list area C2 display character strings indicating specific keys on the keyboard on the lower right. When the reclassifying unit 45 detects that a specific key displayed on the objects 91, 92, 97 is pressed, the current frame 96 in the active frame of the selected channel at that time, that is, in the frame list area C2. It is determined that the frame marked with is associated with the objects 91, 92, and 97 corresponding to the specific key. Subsequent processing is the same as when the association operation is performed in the form of a drag and drop operation.

Further, the reclassification unit 45 can also integrate a plurality of channels (including classification of “unclassified” in this paragraph and the next paragraph) in response to a user operation. Specifically, in the channel list area C1, any one of the objects 91 and 92 can be individually moved so as to be superimposed on another arbitrary one of the objects 91 and 92 by a drag and drop operation. . Alternatively, a plurality of objects 91 and 92 may be selected at the same time, and the plurality of objects 91 and 92 may be collectively moved onto another arbitrary one of objects 91 and 92. (However, even if it is “move”, the moved objects 91 and 92 disappear when they are released on the destination objects 91 and 92.) In other words, the reclassification unit 45 accepts an operation for associating an arbitrary channel in the channel list area C1 with another arbitrary channel from the user. When this association operation is performed, the channels corresponding to the objects 91 and 92 moved by the operation are integrated with the channels corresponding to the other objects 91 and 92 to which the objects 91 and 92 are moved. .

When the channels are integrated by the above association operation, the channel name “unclassified” of the channel corresponding to the destination object 91, 92 is displayed in all frames belonging to the channel corresponding to the moved object 91, 92. ”, The labeling of CH01, CH02,. As a result, the destination objects 91 and 92 become objects that represent a new integrated channel. As the active frame of the integrated channel, the active frames of the objects 91 and 92 that are the movement destinations are continuously used. In addition, the reclassifying unit 45 calculates the number of frames belonging to the integrated channel and reflects it on the display on the objects 91 and 92. Also, in the channel list area C1, when the channel object 91 is moved, the moved channel object 91 disappears, and the remaining channel object 91 is filled so as to fill the space where the lost channel object 91 is arranged. Objects 91 and 97 are aligned. When the unclassified object 92 is moved, the number of frames is 0, but the unclassified object 92 remains as it is.

Further, when there are a plurality of channel objects 91, the order of arrangement of these objects 91 in the channel list area C1 can be freely changed by a drag and drop operation. For example, when it is desired to move the channel object 91 of the channel CH01 to the right side of the channel object 91 of the channel CH02 in the state of FIG. 6, the channel object 91 of the channel CH01 is placed between the channel objects 91 of the channels CH02 and CH03. Move it.

Next, the reproduction area C3 will be described. In the reproduction area C3, a frame group belonging to the selected channel can be reproduced as a moving image. As shown in FIG. 6, a play button 76, a frame advance button 77, and a frame return button 78 are arranged on the reclassification window W4.

The playback button 76 receives a playback command from the user as a moving image of a frame group belonging to the selected channel. When the reclassifying unit 45 detects that the playback button 76 has been pressed, the reclassifying unit 45 sequentially displays the frames included in the frame group belonging to the selected channel in the playback area C3 in the frame advance format along the timeline. Playback starts from the active frame of the selected channel when the playback button 76 is pressed. Reproduction is performed at a frame rate corresponding to the selected channel. During playback, the frames displayed in the playback area C3 are sequentially switched. At each switching, the active frame managed by the reclassifying unit 45 is updated to the frame after the switching in real time. Is done. The playback button 76 receives a command to stop playback from the user. When the reclassifying unit 45 detects that the playback button 76 is pressed during playback, the reclassifying unit 45 fixes the display in the playback area C3 to the active frame at that time. Even during stoppage, the active frame of the selected channel is always displayed in the reproduction area C3. Accordingly, when the active frame is changed by the selection operation of the thumbnail image 94 in the frame list area C2 during the stop, the display in the reproduction area C3 is also changed in real time.

Also, during playback, the active frame is updated as needed, but the surrounding frame 96 in the frame list area C2 also moves in real time to a position surrounding the thumbnail image 94 of the latest active frame. At this time, the thumbnail images of the objects 91 and 92 corresponding to the selected channel are also updated in real time. That is, the display in the reproduction area C3 is synchronized with the position of the surrounding frame 96 and the display of the thumbnail images of the objects 91 and 92. However, in other embodiments, such a synchronization may not be taken during playback. In this case, for example, the position of the surrounding frame 96 may be moved and / or the thumbnail images of the objects 91 and 92 may be updated only when playback is stopped.

Each of the frame advance button 77 and the frame return button 78 issues a command from the user to switch the display in the reproduction area C3 to the next frame one after the next in the frame group belonging to the selected channel along the timeline. Accept. The change of the display in the reproduction area C3 by the operation of the buttons 77 and 78 is also linked to the change of the active frame.

The above playback functions are useful for finding frames that are misclassified within an existing channel. That is, when a frame suddenly appears in a different channel during playback of a moving image, a person can immediately find it. In such a case, the reproduction is immediately stopped, the thumbnail image 94 of the corresponding frame is searched for in the frame list area C2, and the thumbnail image 94 is moved to the correct channel by the above association operation. .

Below the playback area C3 on the reclassification window W4, a frame rate area C5 for displaying the frame rate of the selected channel is displayed. The frame rate can be calculated by various methods. For example, the frame rate is calculated by dividing the number of frames belonging to the selected channel by the difference between the time of the first frame belonging to the selected channel and the time of the last frame. be able to. Alternatively, the frame rate of the selected channel can be calculated based on the following equation.
(Frame rate of video corresponding to the active timeline) x (number of frames belonging to the selected channel) ÷ (number of frames belonging to the active timeline)

By the way, as described above, the frames belonging to the selected channel change due to the reclassification of the frames. Therefore, the reclassification unit 45 recalculates the frame rate of the selected channel every time reclassification is performed, and changes the display of the frame rate area C5 in real time. Further, in the vicinity of the frame rate area C5, a channel name area C6 in which the channel name or classification name of the selected channel is displayed is arranged. Therefore, the user must understand exactly which channel the video displayed in the reproduction area C3 belongs to and which channel the frame rate displayed in the frame rate area C5 is. Can do.

Next, the timeline bar C4 will be described. The timeline bar C4 is an object that schematically shows the active timeline, like the timeline bar T5 on the basic screen W1. The target frame group that is the target of the channel separation process constitutes part or all of the active frame group. Therefore, since it is important that the active timeline can be managed also on the reclassification window W4, a timeline bar C4 similar to the timeline bar T5 on the basic screen W1 is arranged. The timeline bar C4 extends in the left-right direction and is equally divided in the left-right direction by the number of frames of the active frame group. The nth divided area from the left on the timeline bar C4 corresponds to the nth frame on the active timeline (n is a natural number).

The reclassifying unit 45 displays a straight line 83 at the position of the divided area as a sign indicating the divided area corresponding to the active frame of the selected channel on the timeline bar C4. That is, the straight line 83 indicates the position on the active timeline of the frame displayed in the area C3 and the frame 96 of the area C2. A target range bar 85 is displayed below the timeline bar C4. Similar to the timeline bar C4, the target range bar 85 extends in the left-right direction over a range corresponding to the section occupied by the target frame group.

The timeline bar C4 can be expanded or contracted in the left-right direction. Specifically, a scale change bar 86 is arranged on the reclassification window W4, and the scale change bar 86 is an object obtained by combining two objects of the slide groove 61 and the slider 62. The slide groove 61 has a GUI such as a groove that linearly extends in the left-right direction, and the slider 62 has a GUI that slides in the slide groove 61 in the left-right direction. The user can reciprocate the slider 62 in the left-right direction along the slide groove 61 by operating the slider 62 via the input unit 20. The reclassifying unit 45 changes the horizontal scale of the timeline bar C4 stepwise according to the horizontal position of the slider 62 in the slide groove 61. Specifically, as the slider 62 moves to the right, the horizontal scale of the timeline bar C4 becomes shorter stepwise, and as the slider 62 moves to the left, the horizontal scale of the timeline bar C4 becomes a step. Become longer. Note that when the horizontal scale of the timeline bar C4 is changed, the size in the horizontal direction of all the divided areas is also changed equally in proportion thereto. However, when the slider 62 reaches the leftmost position in the slide groove 61, the timeline bar C4 changes to the thumbnail list C7 (see FIG. 7). In addition, when the horizontal scale of the timeline bar C4 and the thumbnail list C7 is long and does not fit in the reclassification window W4, a scroll bar 87 that scrolls in the horizontal direction appears and displays the timeline bar C4. The area for this is substantially enlarged in the left-right direction.

The thumbnail list C7 is an arrangement of thumbnail images 88 of all frames belonging to the active timeline in the horizontal direction. Therefore, the user can check frames other than the target frame group included in the active timeline on the reclassification window W4. The nth thumbnail image 88 from the left on the thumbnail list C7 is a thumbnail image of the nth frame on the active timeline (n is a natural number). In the thumbnail list C <b> 7, a surrounding frame 89 is attached to the thumbnail image 88 of the active frame of the selected channel as a sign for distinguishing from the other thumbnail images 88.

The timeline bar C4 and the thumbnail list C7 accept the selection of an arbitrary frame on the active timeline from the user, similarly to the timeline bar T5. In other words, the user selects an arbitrary frame from the active frame group by selecting an arbitrary thumbnail image 88 in the divided area on the timeline bar C4 or the thumbnail list C7 by a click operation or the like. Can do. Then, each time a frame is selected on the timeline bar C4 and the thumbnail list C7, the reclassifying unit 45 determines whether or not the frame is included in the target frame group. If it is determined that the channel is included, the channel or classification to which the frame belongs is switched to the selected channel. At the same time, the reclassifying unit 45 switches the frame to the active frame of the selected channel. The change of the selected channel and the active frame here is reflected in real time on the display of the channel list area C1, the frame list area C2, the reproduction area C3, the frame rate area C5, and the channel name area C6. The positions of the straight line 83 and the surrounding frame 89 are also changed in real time.

The user can reclassify a plurality of frames included in the target frame group using the various functions described above. Then, when it is determined that the reclassification has ended, the OK button E5 is pressed. In response to this, the reclassification unit 45 closes the reclassification window W4 and newly creates one or more moving images belonging to the new timeline. The one or more newly created moving images correspond to the channels that are finally defined on the reclassification window W4 and are selected by the user (hereinafter referred to as the final channel) on a one-to-one basis. Selection of the final channel by the user is performed by putting a selection mark in a selection box 98 arranged on the channel object 91 as shown in FIG. In the present embodiment, the default value of the selection box 98 is “selected”. Therefore, the user only has to operate the selection box on the corresponding object 91 for the channel that it is determined that the creation of the moving image is not necessary, so that the state becomes “non-selected”. In the present embodiment, a selection box 99 is also arranged below the reproduction area C3. The selection box 99 is an object for selecting whether or not to create a moving image of the selected channel (however, the classification of “unclassified” is not included), and is synchronized with the selection box 98 on the channel object 91. ing. The reclassifying unit 45 determines the selection state of the selection boxes 98 and 99 and determines the final channel.

As described above, the newly created video is obtained by arranging all the frames belonging to the corresponding final channel along a new timeline. At this time, a new display window W2 corresponding to each of these new moving images is created. Further, the reclassifying unit 45 creates a still image file group corresponding to the frame group included in these new moving images on a one-to-one basis, stores it in the processed file area 52, and handles it as the 0th-order frame. That is, thereafter, these new moving images can be processed in the same manner as the still image file group captured in the original image area 51. Specifically, it can be reproduced in the same manner in the display window W2, and is similarly subject to various image processing.

Note that new videos that correspond to the “Uncategorized” category are not created. Therefore, when the OK button E5 is pressed, the reclassifying unit 45 may determine whether or not an unclassified frame remains, and if so, may notify the user to that effect. In this case, for example, the reclassification unit 45 displays a confirmation window W5 as shown in FIG. 8 on the display 10 when it is determined that an unclassified frame remains. Then, after the “Yes” button 74 is pressed, the process proceeds to the above-described new movie creation process. When the “No” button 75 is pressed, the confirmation window W5 is closed and the reclassification window W4 is restored. On the other hand, if it is determined that no unclassified frame remains, the process immediately proceeds to the above-described new movie creation process.

<1-4. Automatic classification algorithm>
Hereinafter, the details of the automatic classification process will be described with reference to FIG. As described above, the automatic classification process starts when the start button 72 on the area setting window W3 is pressed. Hereinafter, a plurality of frames constituting the target frame group are represented as F ₁ , F ₂ ,..., F _J (J is an integer of 2 or more). Note that the arrangement of the frames F ₁ , F ₂ ,..., F _J is equal to the arrangement on the timeline.

First, in step S1, the automatic classification unit 41 reduces the size of each frame F ₁ , F ₂ ,..., F _J included in the target frame group. At this time, for example, the number of pixels in the horizontal and / or vertical direction of the frame may be reduced so as to be a specified number, or may be reduced so as to be a predetermined ratio from the original size. At this time, the aspect ratio may or may not be saved. This reduction step speeds up the following processing and reduces the influence of noise. In the following, the frame F _1, F ₂ after reduction, · · ·, F _J also, F _1, F _2, · · ·, expressed as F _J.

In step S2, the setting unit 42 defines a channel CH _1, classifies the frames F ₁ to the channel CH _1. Specifically, labeling of the channel name CH ₁ is given to the frame F ₁ . The setting unit 42 sets the frame F ₁ as a reference frame G ₁ representing a frame group belonging to the channel CH ₁ . Note that the number of channels can be increased at any time by step S10 described later. For the newly created channel CH _l, the reference frame G _l representing the frame group belonging to each channel is set (l = 2, 3,...).

When the above preprocessing is completed, the setting unit 42 sequentially selects the frames F _j along the timeline from the remaining frames F ₂ , F ₃ ,..., F _J (j = 2, 3, .., J), set as a comparison frame. On the other hand, each time the comparison frame F _j is set, the calculation unit 43, the determination unit 44, and the setting unit 42 repeat the following steps S3 to S10 with respect to the comparison frame F _j , and the existing reference frames G ₁ , G _{1 2} ,..., G _L (L is the number of existing channels).

In step S3, the calculation unit 43 divides the entire comparison frame _Fj into partial regions (blocks) D ₁ , D ₂ ,..., D _K of a specified number K (K is an integer of 2 or more) (FIG. 10). Here, a comparative entire frame F _j to say, if it has been designated area on the area setting window W3 are entire area, if not specified is the entire screen of the frame is there. Note that the partial areas D ₁ , D ₂ ,..., D _K do not have to be divided as a whole into the comparison frame F _j, and may partially overlap each other. The shape does not need to be a rectangle, and may be, for example, a circle or another polygon. Further, the shape and size can be uniform or different from each other.

In step S4, calculating section 43, existing channel CH _1, CH _2, · · ·, reference frames G _1, G ₂ of CH _L, · · ·, each of G _L, subregion D _'1, Divided into D ′ ₂ ,..., D ′ _K (see FIG. 10). Subregion _{D '1, D' 2,} ···, D 'K , in the screen frame, each partial region D _1, D _2, · · ·, are defined to occupy the same position as D _K .

Next, in step S5, the calculation unit 43 calculates the comparison frame F _j and each reference frame G _l (for each combination of the partial areas D _k , D ′ _k (k = 1, 2,..., K). l = 1, 2,..., L) (hereinafter referred to as local similarity) Y _lk is calculated. That is, the local similarity Y _lk, a similarity between the partial region D _'k of partial areas D _k and the reference frame G _l comparison frame F _j. In this embodiment, a correlation coefficient called normalized cross-correlation (ZNCC) is calculated, but in other embodiments, other similarities such as a correlation coefficient called NCC and similarities called SDD and SAD are used. May be calculated.

Next, in step S6, the calculation unit 43 calculates the similarity between the comparison frame F _j and each reference frame G _l (l = 1, 2,..., L) (hereinafter referred to as total similarity). ) _B1 is calculated. Specifically, the calculation unit 43 determines partial regions D _k and D ′ _k having a high local similarity Y _lk for each l (l = 1, 2,..., L). At this time, for example, it is possible to determine that the local similarity Y _lk falls within the upper specified ratio, and it is possible to determine that the local similarity Y _lk is higher than a predetermined threshold. Then, the calculation unit 43 compares each l (l = 1, 2,..., L) using only the partial regions D _k and D ′ _{k having} a high local similarity Y _{lk in} the entire frame. The degree of similarity between the frame F _j and the reference frame G _l is calculated again to obtain the total degree of similarity B _l (see FIG. 10). In this embodiment, normalized cross-correlation (ZNCC) is calculated as the total similarity B ₁ , but other similarities may be calculated in the same manner as the local similarity Y _lk . Further, the total similarity B ₁ and the local similarity Y _lk may be calculated by different methods.

As described above, in the present embodiment, the information on the partial regions D _k and D ′ _{k having} a low local similarity Y _lk is not considered in calculating the overall similarity B ₁ , for the following reason. That is, the frames of the same channel have the same background image but are different in the moving body portion. If the overall similarity B ₁ is determined using the information of the entire frame when the proportion of the moving body portion in the entire frame is large, the overall similarity B ₁ is low even if the frames of the same channel are used. Because it becomes. Therefore, here, in order to avoid erroneous determination, the partial regions D _k and D ′ _{k having} a low local similarity Y _lk that are considered to include many mobile parts are not used for calculating the total similarity B ₁ . As a result, the influence of the moving object is reduced, and it can be accurately determined whether or not the compared frames belong to the same channel.

In the subsequent step S7, the determination unit 44 determines the maximum _{one of} the overall similarities B ₁ , B ₂ ,..., B _L calculated in step S6, and the maximum is greater than a predetermined threshold value. Determine whether it is larger. If it is determined that the value is greater than the predetermined threshold, the process proceeds to step S8. If it is determined that the value is equal to or less than the predetermined threshold, the process proceeds to step S10.

In step S8, the determination unit 44 determines that the comparison frame F _j belongs to the channel CH _MAX . MAX is a value of _l that gives the maximum total similarity B _l . More specifically, the determination unit 44, the labeling of the channel name CH _MAX assigned to the comparison frame F _j, to classify the comparison frame F _j to the channel CH _MAX. As a result, the comparison frame F _j belongs to the same channel as the frame group belonging to the channel CH _MAX .

After step S8, the process proceeds to step S9. In step S9, the setting unit 42 updates the reference frame G _MAX of the channel CH _MAX. Specifically, the setting unit 42 synthesizes the comparison frame F _j and the existing reference frame G _MAX to form a new reference frame G _MAX . In the present embodiment, a weighted average is employed as a mode of synthesis such that the weight of the comparison frame F _j is larger than that of the existing reference frame G _MAX . By doing so, the reference frame G _MAX representing the frame group belonging to the channel CH _MAX becomes a composite image in which the weight is placed on the latest frame. As a result, it is possible to cope with a time-series change in determining the channel difference between frames. In another embodiment, the comparison frame F _j may be set as a new reference frame G _MAX as it is. It is also possible to omit the step S9, so as not to update the reference frame G _MAX.

On the other hand, step S10 is a step that is executed when it is determined that the total similarity B ₁ , B ₂ ,..., B _L calculated in step S6 is less than or equal to a predetermined threshold value. In other words, step S10 is a step executed when the comparison frame F _j is not similar to the reference frame G _l of any channel CH _l . In step S10, the setting unit 42 defines a new channel CH _{L + 1} in addition to the existing channels CH ₁ , CH ₂ ,..., CH _L , and sets the comparison frame F _j to the channel CH _{L + 1} . Classify. Specifically, labeling of a new channel name CH _{L + 1} is given to the comparison frame F _j . The setting unit 42 sets the comparison frame F _j as a reference frame G _{L + 1} that represents a frame group that belongs to the new channel CH _{L + 1} .

When Steps S3 to S10 for all the frames F ₁ , F ₂ ,..., F _J are completed, the process proceeds to Step S11. In step S11, the existing channels CH ₁ , CH ₂ ,..., CH _L are corrected. Specifically, the calculation unit 43, reference frame G _1, G _2, · · ·, calculated on the total per frame overall similarity between G _L. The whole frame here is the same as step S3 when the area is specified on the area setting window W3, and when the area is not specified, the frame screen is displayed. The whole. In this embodiment, as in steps S5 and S6, normalized cross-correlation (ZNCC) is calculated as the similarity, but other similarities may be calculated. Further, the similarity may be calculated by a method different from the total similarity B ₁ and the local similarity Y _lk . Then, the determination unit 44, If there is more than a predetermined threshold value to these similarities, was what channel CH _l corresponding to the reference frame G _l providing such similarity, integrated into one channel. Specifically, labeling with the same channel name is re-assigned to all the frames F _j belonging to the channel CH _l to be integrated. Here, three or more channels CH ₁ may be integrated into one.

Furthermore, the determination unit 44, or more if the channel CH _l not only include still one frame by performing the integration of the channel is present, to extinguish the channel CH _l. Specifically, the determination unit 44 re-assigns “unclassified” labeling to the frames F _j belonging to all the channels CH ₁ to be eliminated. In the present embodiment, the channel CH _l containing only one frame F _j is a target for disappearance, but such reference values may be set to two, three, and so on. It is also possible to set an upper limit on the number of channels that can be created. In this case, it is possible to eliminate the channel F _j from a channel with a small number so that the number of channels does not exceed the upper limit.

According to the above processing, the numerical value included in the channel name CH _l of the finally created channel is highly likely not to be a serial number. Therefore, the determination unit 44 finally created channel CH _l, CH01, CH02, the channel name and ... and reapplication sequentially, the labeling which is given to each frame F _j, these Update with a new channel name. After that, the automatic classification process ends.

<2. Second Embodiment>
Hereinafter, a second embodiment of the present invention will be described. FIG. 11 shows the flow of automatic classification processing according to the second embodiment. The second embodiment and the first embodiment are different only in the automatic classification algorithm. Further, as can be seen by comparing FIG. 9 and FIG. 11, the difference between the two automatic classification processes is only that steps S24 to S26 are inserted instead of steps S4 to S6 in the second embodiment. is there. Therefore, for the sake of simplicity, only the difference will be described below.

In the second embodiment, step S24 is executed after execution of steps S1 to S3 similar to those in the first embodiment. In step S24, the calculation unit 43 detects feature points in the partial areas D ₁ , D ₂ ,..., D _{K of} the comparison frame F _j . At this time, the number of the partial regions D _1, D _2, · · ·, feature points detected in the D _K, it is preferable that the same number. Hereinafter, the feature points in the comparison frame F _j are represented as P ₁ , P ₂ ,..., P _U (U is an integer of 2 or more).

The calculation unit 43 sets a partial region V _u in the vicinity of each feature point P _u (u = 1, 2,..., U) in the comparison frame F _j (see FIG. 12). The partial region V _u is a region having a predetermined size with the feature point P _u as the center. Then, calculating unit 43, existing channel CH _1, CH _2, · · ·, reference frames G _1, G ₂ of CH _L, · · ·, for each of the G _L, partial regions V _'1, V' ₂ ,..., V ′ _U are set (see FIG. 12). Partial region _{V '1, V' 2,} ···, V 'U , in the screen frame, each partial area V _1, V _2, · · ·, are defined to occupy the same position as V _U .

As described above, in the present embodiment, from the comparison frame F _j, subregion D _1, D _2, · · ·, in units of D _K, the feature point P _1, P _2, · · ·, is P _U Detected. As a result, the feature point _{P 1, P 2, ···,} P U is, without thereby biased in a partial area of the entire screen of the comparison frame F _j, approximately evenly from the entire comparison frame F _j Detected.

Next, in step S25, the calculation unit 43 calculates the comparison frame F _j and each reference frame G _l (for each combination of the partial regions V _u , V ′ _u (u = 1, 2,..., U). l = 1,2, ···, calculates the local similarity Y _lu and L). That is, the local similarity Y _lu, a similarity between the partial region V _'u comparison frame F _j of the partial region V _u and the reference frame G _l. In addition, since the local similarity in 1st and 2nd embodiment is common at the point that both represent the similarity in a partial region, it represents using the same symbol Y. In this embodiment, a correlation coefficient called normalized cross-correlation (ZNCC) is calculated, but other similarities may be calculated as in the first embodiment.

Next, in step S26, the calculation unit 43 calculates the overall similarity B _l for the entire frame between the comparison frame F _j and each reference frame G _l (l = 1, 2,..., L). . Note that the overall similarity in the first and second embodiments is common in that both represent the similarity in the entire frame, and therefore the same symbol B is used. Specifically, the calculation unit 43 determines partial regions V _u and V ′ _u having a high local similarity Y _lu for each l (l = 1, 2,..., L). At this time, for example, it is possible to determine that the local similarity Y _lu falls within the higher specified ratio, and it is possible to determine that the local similarity Y _lu is higher than a predetermined threshold. For each l (l = 1, 2,..., L), the comparison frame F _j and the reference frame G are used by using only V _u and V ′ _u having a high local similarity Y _{lu in} the entire frame. _The degree of similarity with _l is calculated again to obtain the total degree of similarity B _l . Also in the present embodiment, the normalized cross correlation (ZNCC) is calculated as the total similarity B ₁ , but other similarities may be calculated as in the first embodiment. Also in this embodiment, the total similarity B ₁ and the local similarity Y _lu may be calculated by different methods.

The overall similarity B ₁ , B ₂ ,..., B _L is calculated by the above step S26. The subsequent processing flow is the same as in the first embodiment.

As described above, in the present embodiment, in calculating the overall similarity, the feature point P _1, P _2, · · ·, partial regions V ₁ of the vicinity of the P _U, V _2, · · ·, other than V _U Area information is not considered. This is because, even if frames of different channels are taken from similar scenes, if the overall similarity B _l is determined using the information of the entire frame, the overall similarity B _l increases. Because. For example, in two images taken at different locations in the same store, the same uniform color wallpaper or floor may occupy most of the frame. In such a case, the overall similarity B _l is high. turn into. Thus, here, to avoid an erroneous determination, regions with high possibility that copies of the same uniform color wallpaper, background such as a floor, that is, the feature point P _1, P _2, · · ·, other than the vicinity of the P _{U Are} not used for calculating the total similarity B ₁ . As a result, it is possible to reduce the influence of the similar background and accurately determine whether or not the compared frames belong to the same channel.

Furthermore, in this embodiment, in calculating the total similarity B ₁ , information on the partial regions V _u and V ′ _{u having} a low local similarity Y _lu is not taken into consideration. This is because, when the ratio of the moving body portion in the entire frame is large, when it is determined overall similarity B _l using the information of the entire frame, even if the frame to each other in the same channel, the overall similarity B _l This is because it becomes lower. Therefore, here, in order to avoid erroneous determination, the partial regions V _u and V ′ _u having a low local similarity Y _lu that are considered to include many mobile parts are not used for the calculation of the total similarity B _l . As a result, it is possible to reduce the influence of the moving body and more accurately determine whether the compared frames belong to the same channel.

<3. Third Embodiment>
Hereinafter, a third embodiment of the present invention will be described. FIG. 13 shows a flow of automatic classification processing according to the third embodiment. The third embodiment is different from the first and second embodiments only in the automatic classification algorithm. As can be seen from a comparison between FIG. 13 and FIG. 9, the difference between the automatic classification processes according to the first and third embodiments is that, in the third embodiment, a subroutine S30 is used instead of steps S5 and S6. Only the points that are inserted. Therefore, for the sake of simplicity, only the difference will be described below.

As described above, in the first and second embodiments, the correlation coefficient is calculated as the similarity between frames. In the third embodiment, the similarity between frames is evaluated based on the color tendency of the frames. . Further, as the color tendency of the frame, an overall color tendency in the frame screen and a local color tendency in the frame screen are comprehensively considered. This will be specifically described below.

First, in the third embodiment, the same steps S1 to S4 as in the first embodiment are executed, and then the subroutine S30 is executed. Subroutine S30 is a process for evaluating various color indexes indicating local or overall color trends in the frame screen, and details thereof are shown in FIG. In particular, density is taken into consideration in steps S31 to S33 included in the subroutine S30, saturation is taken into consideration in steps S34 to S38, and hue is taken into consideration in steps S39 to S43. In steps S31 to S33, the density is taken into consideration, but brightness or luminance may be used instead of the density. In steps S44 to S46 included in the subroutine S30, the overall impression level of the frame is considered.

First, in step S31, calculation unit 43 calculates an average density Hd overall comparison frame F _j. Here, a comparative entire frame F _j to say, if it has been designated area on the area setting window W3 are entire area, if not specified is the entire screen of the frame is there. Further, the calculation unit 43 calculates an average density Ed _k in each partial region D _k (k = 1, 2,..., K) in the comparison frame F _j . The average density is calculated as an average value of the density of all the pixels in the target area, and the density of each pixel is calculated as the average value of the RGB values of the pixel. Subsequently, the calculation unit 43 assigns a value of “1” to the partial areas D _k (k = 1, 2,..., K) when the density Ed _k > the density Hd. When the density Ed _k ≦ density Hd, a value of “0” is given (see FIG. 14). Note that “1” and “0” are indicators that indicate whether the density of the local region is relatively high or low with respect to the entire frame, and are color indicators that represent local color trends related to density (brightness) ( Hereinafter, the concentration index).

In step S32, calculation unit 43, the same processing as step S31, each of the existing channel CH _1, CH _2, · · ·, reference frames G ₁ of _{CH L, G 2, ···,} G L Run against. That is, the calculation unit 43 assigns each l (l = 1, 2,..., L) to each partial region D ′ _k (k = 1, 2,..., K) in the reference frame G _l . On the other hand, a concentration index is calculated (see FIG. 14). Incidentally, as shown in FIGS. 13 and 17, since that would step S32 is repeatedly executed, for reference frames G _l concentration index is already calculated, it is referred the value, the processing of a new calculation omitted Is done.

In the subsequent step S33, the calculation unit 43 calculates the comparison frame F _j and each reference frame G _l (l = 1, 2,..., L) based on the density index calculated in steps S31 and S32. the similarity is calculated Bd _l between. Specifically, the calculation unit 43 calculates the similarity between the density index distribution pattern in the comparison frame F _j and the density index distribution pattern in each reference frame G _l as the similarity Bd _l. . In the present embodiment, the number of combinations of partial areas D _k and D ′ _k where the density indices “1” and “0” do not match is counted, and a square value of a value obtained by dividing the count value by the number K of partial areas is obtained. , the similarity Bd _l (see FIG. 14). Here, the division of the number K of the partial area is performed, the similarity Bd _l and maintained at a value between 0 and 1, in order to normalize. Further, the similarity calculated in step S33 and steps S38 and S43, which will be described later, becomes a smaller value as they are similar. In this sense, these similarities can also be regarded as dissimilarities.

Incidentally, the composition of an image is determined by the arrangement of a plurality of elements in the screen. Many images have a composition that is roughly divided into an area representing the object of interest and an area representing the background. That is, it can be considered that one of the region having the density index “1” and the region “0” represents the object of interest and the other represents the background thereof. From this concept, the distribution pattern of concentration indicators in comparison frame F _j, represents the composition of the comparison frame F _j, the distribution pattern of concentration indicators in the reference frame G _l is the reference frame G _l It can be said that it represents the composition of Therefore, it can be said that the similarity Bd _l represents the similarity of composition between the comparison frame F _j and the reference frame G _l .

In step S34, calculation unit 43, based on the distribution pattern of concentration indicators in comparison frame F _j, dividing the comparison frame F _j 2 two partial regions R _1, to R ₂ (see FIG. 15). The partial region R ₁ is a region _obtained by combining all the partial regions D _k having the density index “1” in the comparison frame F _j , and the partial region R ₂ has the density index “0” in the comparison frame F _j . This is a region where all the partial regions D _k are combined. That is, in step S34, based on the composition of the comparison frame F _j, is divided into an area representing the background and a region representing the object of interest the comparison frame F _j.

In step S35, similarly to step S34, calculation unit 43, the concentration in each of the existing channel CH _1, CH _2, · · ·, reference frames G ₁ of _{CH L, G 2, ···,} G L Based on the index distribution pattern, each of the reference frames G ₁ , G ₂ ,..., G _L is divided into partial areas R ′ ₁ and R ′ ₂ (see FIG. 15). The partial region R ′ ₁ is a region _obtained by combining all the partial regions D ′ _k whose density index is “1” in the reference frame G ₁ , and the partial region R ′ ₂ has a density index in the reference frame G ₁ . This is a region where all the partial regions D ′ _k that become “0” are combined. That is, in step S35, based on the composition of the reference frame G _l, divides the reference frame G _l in the region that represents the object of interest and a region representing the background.

In subsequent step S36, the calculation unit 43 calculates three average values for each of R, G, and B for each of the partial regions R ₁ and R ₂ in the comparison frame F _j . Subsequently, the calculation unit 43, the three average values of RGB by the partial region R _1, to calculate the saturation partial region R _1, the RGB separate three average values in the partial region R _2, the partial region to calculate the saturation of R _2. Next, the calculation unit 43 calculates the relative saturation (hereinafter referred to as relative saturation) between the partial regions R ₁ and R ₂ . The relative saturation is calculated as an absolute value of the difference between the saturation of the partial region R _{1 and} the saturation of the partial region R ₂ . Relative saturation is a color index (hereinafter referred to as saturation index) representing a color tendency related to saturation.

In subsequent step S37, calculation unit 43, the same processing as step S36, each of the existing channel CH _1, CH _2, · · ·, reference frames G ₁ of _{CH L, G 2, ···,} G L Run against. That is, the calculation unit 43 calculates the saturation of each of the partial regions R ′ ₁ and R ′ ₂ in the reference frame G ₁ for each l (l = 1, 2,..., L), Relative saturation, which is the absolute value of the difference, is calculated. Incidentally, as shown in FIGS. 13 and 17, since the step S37 will be repeatedly executed, for reference frames G _l relative saturation has already been calculated, the value is referred to, the processing of a new calculation Omitted. Step S35 can be omitted as well.

In the subsequent step S38, the calculation unit 43 calculates the relative saturation of the comparison frame F _j calculated in steps S36 and S37 and the relative saturation of each reference frame G _l (l = 1, 2,..., L). The similarity Bs _l is calculated. In the present embodiment, the similarity Bs _l is calculated as a square value of a value obtained by dividing the difference between the two relative chromas by 255. Here, the division by 255 is performed in order to maintain the similarity Bs _l at a value between 0 and 1 and normalize it.

In subsequent step S39, the calculation unit 43 divides the comparison frame F _j into _two regions, a main region O ₁ and a sub region O ₂ . Specifically, among the partial regions R ₁ and R ₂ calculated in step S34, the region with the larger area is set as the main region O ₁ and the region with the narrower region is set as the sub region O ₂ (see FIG. 16). That is, step S39, based on the composition of the comparison frame F _j, is divided into an area representing the background and a region representing the object of interest the comparison frame F _j.

In subsequent step S40, as in step S39, the calculation unit 43 converts each of the reference frames G ₁ , G ₂ ,..., G _{L into two} regions, a main region O ′ ₁ and a sub region O ′ _2. To divide. Specifically, of the partial regions R ′ ₁ and R ′ ₂ calculated in step S35, the region with the larger area is set as the main region O ′ ₁ and the region with the narrower region is set as the sub-region O ′ _2. Set (see FIG. 16). That is, in step S40, based on the composition of the reference frame G _l, divides the reference frame G _l in the region that represents the object of interest and a region representing the background.

In subsequent step S41, the calculation unit 43 calculates the average hue of the main area O ₁ and calculates the average hue of the sub area O ₂ . The average hue is a color index (hereinafter referred to as a hue index) representing a local color tendency related to the hue, and is calculated as an average value of the hues of all the pixels in the target area.

In step S42, calculation unit 43, the same processing as step S41, each of the existing channel CH _1, CH _2, · · ·, reference frames G ₁ of _{CH L, G 2, ···,} G L Run against. That is, the calculation unit 43 calculates a hue index for each of the regions O ′ ₁ and O ′ ₂ in the reference frame G _l for each l (l = 1, 2,..., L). Incidentally, as shown in FIGS. 13 and 17, to become a thing step S42 is repeatedly executed, for reference frames G _l hue index has already been calculated, is referred the value, the processing of a new calculation omitted Is done. Step S40 can be omitted as well.

In subsequent step S43, calculation unit 43 compares the frame F and the hue index of the main region O ₁ in _j, the reference frame _{G l (l = 1,2, ···} , L) main region O _'1 in The similarity Bh _1l with the hue index at is calculated. In the present embodiment, the similarity Bh _1l is calculated as a square value of a value obtained by dividing the difference between the hue indexes of the main regions O ₁ and O ′ ₁ by 180. The calculation unit 43 also calculates the hue index in the sub-region O ₂ in the comparison frame F _j and the hue index in the sub-region O ′ ₂ in each reference frame G _l (l = 1, 2,..., L). The similarity Bh _2l is calculated. Specifically, the square value of the value obtained by dividing the difference between the hue indexes of the sub-regions O ₂ and O ′ ₂ by 180 is set as the similarity Bh _2l . Here, the reason why the division by 180 is performed is to maintain the similarity Bh _1l and Bh _2l between 0 and 1 and normalize them.

Subsequent steps S44 to S46 are steps for evaluating a color index indicating an overall color tendency in the frame screen. In this embodiment, various impressions Z ₁ , Z ₂ ,..., Z _I (I is an integer of 2 or more) are defined in order to evaluate the overall color tendency in the frame screen. Yes. Various kinds of information for defining these impression degrees Z ₁ , Z ₂ ,..., Z _I are stored in an impression degree definition area 53 in the software management area 50.

Hereinafter, the impression degree Z _i (i = 1, 2,..., I) will be described. Each impression degree Z _i is associated with one or a plurality of color conditions, and weights are defined for these color conditions. Note that the weight of one or more color conditions associated with the same impression degree Z _i is 1 in total. FIG. 18 shows an example of the impression degree Z ₁ “natural”, and the impression degree Z ₁ “natural” is associated with three color conditions of “green”, “brown”, and “beige”. Also, weights of 0.5, 0.25, and 0.25 are assigned to the three color conditions of “green”, “brown”, and “beige”, respectively. As shown in FIG. 18, evaluation values for density (lightness), saturation, and hue values are defined for each color condition. Here, when a certain pixel is given, the value of each color condition is calculated by deriving evaluation values corresponding to the density, saturation, and hue values of the pixel and then multiplying these evaluation values. . Then, the value of the impression degree Z _i of the pixel is calculated as a value obtained by adding the above weighting values to the values of the respective color conditions.

In step S44, the calculation unit 43 calculates the value of each impression degree Z _i (i = 1, 2,..., I) in the entire comparison frame F _j . The whole frame here is the entire area when the area is designated on the area setting window W3, and the whole frame screen when the area is not designated. Specifically, for each i (i = 1, 2,..., I), the calculation unit 43 calculates the value of the impression degree Z _i for each pixel included in the entire frame of the comparison frame F _j . An average value of these values is calculated, and the average value is set as the value of the impression degree Z _{i in} the entire frame. The value of the impression Z _i of the entire frame, the color index representing a color tendency of the entire frame about the impression Z _i (hereinafter, the impression index) is.

In step S45, calculation unit 43, the same processing as step S44, each of the existing channel CH _1, CH _2, · · ·, reference frames G ₁ of _{CH L, G 2, ···,} G L Run against. That is, the calculating section 43, each of l (l = 1,2, ···, L) for, impression Z _1, Z _2, · · ·, for each of Z _I, impression index for the reference frame G _l Is calculated. Incidentally, as shown in FIGS. 13 and 17, to become a thing step S45 is repeatedly executed, for reference frames G _l impression index has already been calculated, is referred the value, the processing of a new calculation omitted Is done.

In step S46, calculation unit 43, impression Z _1, Z _2, · · ·, based on the Z _I, Comparative frame F _j and each reference frame _{G l (l = 1,2, ···} , the similarity is calculated Bi _l of the entire frame with L). Specifically, the calculation unit 43 calculates the entire frame of the comparison frame F _j for each l (l = 1, 2,..., L) and each i (i = 1, 2,..., I). It calculates the square value of the difference between the value of the impression Z _i of the entire frame value and the reference frame G _l of impression Z _i in. For each l (l = 1, 2,..., L), the square root of the sum of these I square values (the comparison frame F _j and the reference frame G _{l in the} I-dimensional impression degree space) distance) to calculate the value obtained by subtracting from 1, to this similarity Bi _l.

Next, in step S47, the calculation unit 43 performs comparison based on the already calculated similarities Bd _l , Bs _l , Bh _1l , Bh _2l , Bi _l (l = 1, 2,..., L). A similarity (hereinafter referred to as a total similarity) B _l between the frame F _j and each reference frame G _l (l = 1, 2,..., L) is calculated. Note that the overall similarity in the first and third embodiments is common in that both represent the similarity in the entire frame, and therefore, the same symbol B is used. In the present embodiment, the calculation unit 43 calculates, for each l (l = 1, 2,..., L), the square root of the total value of Bd _l , Bs _l , Bh _1l , Bh _2l (four-dimensional for four indices). After calculating a value (composition comparison result) obtained by subtracting the distance between the comparison frame F _j and the reference frame G _l in the space from 1 (composition comparison result), the value obtained by multiplying the value and Bi _l (impression comparison result) is totally similar. Degree _B1 . In other embodiments, different methods can be adopted. For example, both the composition comparison result and / or the impression comparison result are multiplied by an appropriate coefficient, and an appropriate value is added or subtracted. You may make it multiply a value. Alternatively, a sum of the square root of 1-Bd ₁ , the square root of 1-Bs ₁ , the square root of 1-Bh _1l , the square root of 1-Bh _2l , and Bi _l while appropriately adding coefficients is used as the total similarity B _1. Also good.

Through the above step S47, the total similarity B ₁ , B ₂ ,..., _BL is calculated. The subsequent processing flow is the same as in the first embodiment.

<4. Application>
The image processing program 2 can handle image processing for a wide variety of moving images. For example, the image processing program 2 can be used in a scene where an organization such as the police analyzes a surveillance video of a security camera for investigation of an incident. For example, many security cameras may be installed in the same store, but these surveillance videos are often recorded in the form of a mixed time-lapse movie. And in such a case, if you use a dedicated device in the store where the security camera is installed, you can separate each channel included in the mixed time-lapse video, but in the police etc., each model of each manufacturer Since it does not have a corresponding dedicated device, playback becomes difficult. Therefore, the timeline separation process that can separate the mixed time-lapse moving image into the moving image for each channel can be utilized in such a scene regardless of the specification of the moving image.

<5. Modification>
As mentioned above, although one Embodiment of this invention was described, this invention is not limited to the said embodiment, A various change is possible unless it deviates from the meaning. For example, the following changes can be made.

<5-1>
The reclassification window W4 in the above embodiment may be composed of a plurality of windows.

<5-2>
In the above embodiment, by switching the selected channel, the frame group is switched and displayed in units of channel or classification in the frame list area C2. However, a frame list area for displaying a list of frames may be provided for each of a plurality of channels and a classification of “unclassified”. In this case, in order to use the screen space efficiently, it is preferable to display the frame group as a thumbnail in each frame list area, and it is preferable to arrange the thumbnail image group along the timeline. Further, in this case, the channel object 91 and the unclassified object 92 in the form of icons are omitted, and the frame list area for each channel is used as a channel object for the association operation, and the “unclassified” classification object is used. The frame list area can be used as an unclassified object for the association operation. That is, by selecting an arbitrary frame from the “unclassified” frame list area and dragging and dropping it to an arbitrary frame list area (channel object) for a channel, the frame is associated with the area. It is also possible to configure it so that it can be classified into channels. The same applies to a case where a frame included in a channel is moved to another channel or classification.

Alternatively, a frame list area for displaying a list of “unclassified” frame groups and a frame list area for displaying a list of frame groups belonging to the channel are provided, and the “unclassified” frame groups are included in the former area. May be always displayed. In this case, in the latter area, it is possible to display only a list of frames belonging to the currently selected channel.

<5-3>
In the third embodiment, are defined various color index, but these are overall similarity B _l is defined variously combined, it is also possible to define the color index in other embodiments, other It is also possible to define the overall similarity B ₁ by combining color indexes in a manner. For example, in the third embodiment, the composition is determined based on the color index related to density, but the composition may be determined based on the color index related to hue and / or saturation, for example. Also, the similarity Bs _l is calculated based on the relative saturation between the partial areas R ₁ and R ₂ , but the absolute saturation difference between the partial areas R ₁ and R ′ ₁ , A difference in absolute saturation between R ₂ and R ′ ₂ may be used as the similarity Bs _l . Further, the difference in density between the partial areas R ₁ and R ′ ₁ and the difference in density between the partial areas R ₂ and R ′ ₂ may be calculated as the similarity and added to the total similarity B _1. .

<5-4>
In the above embodiment, the area setting window W3 is displayed immediately after the start of the channel separation process, and the area can be set to avoid erroneous determination. However, such an area setting window W3 may not be displayed so that the area cannot be set. Alternatively, although the area setting window W3 is not displayed immediately after the start of the channel separation process, a button may be provided on the reclassification window W4 for setting the area and instructing the redo of the automatic classification process. Good. In this case, the user presses the above button and displays the area setting window W3 only when the result of the automatic classification process performed without setting the area is not appropriate (there are many erroneous determinations). Then, after setting the area, the automatic classification process can be executed again. Alternatively, the area setting window W3 is displayed immediately after the start of the channel separation process, and the automatic classification process can be executed with the area set, and the above buttons are provided on the reclassification window W4. May be.

<5-5>
In step S9 of the above embodiment, a new reference frame _GMAX is obtained by combining the comparison frame _Fj and the existing reference frame _GMAX by weighted averaging. However, the method of updating the reference frame G _MAX is not limited thereto. For example, for each pixel, a histogram of pixel values of all frames (including the comparison frame F _j at that time) included in the channel CH _MAX is created, and the most frequent pixel value is determined as the pixel of the reference frame G _MAX . It is good also as a pixel value. Alternatively, the pixel value serving as the median value in the histogram may be used as the pixel value of the pixel in the reference frame _GMAX .

<5-6>
In the above embodiment, it is determined whether or not the number of channels L has reached a certain number immediately after step S10, and when it is determined that the number has reached a certain number or more, You may make it perform the correction process of a channel. This is because if the number of channels L increases too much, the processing load becomes large. However, even if there is a channel with the number of frames of 1, if the frame of 1 is close to a certain value in time from the currently selected comparison frame F _j , the channel is not deleted. It is preferable to do so. This is because the channel is a recently created channel and there is a high possibility that a frame classified as the channel will appear in the future. If the number of channels L exceeds a certain value only by deleting the channels as described above, the channels with the number of channels of 2 or more are also sequentially deleted, and the channels The number can be kept below a certain number.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Image processing program 41 Automatic classification part 42 Setting part 43 Calculation part 44 Determination part 45 Reclassification part 91 Channel object 92 Unclassified object C3 Playback area W4 Reclassification window (reclassification screen)

Claims (7)

1. An image processing apparatus for classifying the frames into the channels when frames belonging to different channels are mixed in one moving image,
   By performing image processing on a plurality of frames included in the moving image, a similarity between frames is calculated based on a local color tendency and / or an overall color tendency, and the plurality of frames are determined according to the similarity. An automatic classification unit that classifies the frames into a plurality of channels,
   Image processing device.
2. The automatic classification unit
   A specific frame included in the moving image or a frame obtained by combining two or more specific frames included in the moving image is set as a reference frame, and another specific frame included in the moving image is set as a comparison frame A setting section;
   The comparison frame and the reference frame are divided into a plurality of partial areas, a first color index is calculated for each partial area, and a composition of the comparison frame and the reference frame is derived based on the first color index. A calculation unit that calculates the similarity between the composition of the comparison frame and the composition of the reference frame;
   A determination unit that determines, based on the similarity, whether the comparison frame belongs to the same channel as the reference frame or a frame from which the reference frame is synthesized;
   The image processing apparatus according to claim 1.
3. The automatic classification unit
   A specific frame included in the moving image or a frame obtained by combining two or more specific frames included in the moving image is set as a reference frame, and another specific frame included in the moving image is set as a comparison frame A setting section;
   The comparison frame and the reference frame are divided into a plurality of partial areas, a first color index is calculated for each partial area, and a composition of the comparison frame and the reference frame is derived based on the first color index. And dividing the comparison frame and the reference frame into a plurality of new partial areas based on the composition, and for each of the comparison frame and the reference frame, a second color in at least one of the new partial areas. A calculation unit that calculates an index and calculates the degree of similarity between the second color index of the comparison frame and the second color index of the reference frame;
   A determination unit that determines, based on the similarity, whether the comparison frame belongs to the same channel as the reference frame or a frame from which the reference frame is synthesized;
   The image processing apparatus according to claim 1.
4. The automatic classification unit
   A specific frame included in the moving image or a frame obtained by combining two or more specific frames included in the moving image is set as a reference frame, and another specific frame included in the moving image is set as a comparison frame A setting section;
   The comparison frame and the reference frame are divided into a plurality of partial areas, a first color index is calculated for each partial area, and a composition of the comparison frame and the reference frame is derived based on the first color index. And dividing the comparison frame and the reference frame into a plurality of new partial areas based on the composition, and for each of the comparison frame and the reference frame, a second color in at least two of the new partial areas. An index is calculated, and for each of the comparison frame and the reference frame, a third color index that is a difference between the second color indices is calculated, and the third color index of the comparison frame and the reference frame A calculation unit for calculating the similarity with the third color index;
   A determination unit that determines, based on the similarity, whether the comparison frame belongs to the same channel as the reference frame or a frame from which the reference frame is synthesized;
   The image processing apparatus according to claim 1.
5. The automatic classification unit evaluates the color tendency based on at least one of density, brightness, luminance, saturation, and hue;
   The image processing apparatus according to claim 1.
6. An image processing program for classifying the frames for each channel when frames belonging to different channels are mixed in one moving image,
   By performing image processing on a plurality of frames included in the moving image, a similarity between frames related to a local color tendency and / or an overall color tendency is calculated, and the plurality of frames is determined according to the similarity Causing the computer to perform the step of classifying the channel into a plurality of channels,
   Image processing program.
7. An image processing method for classifying the frames into the channels when frames belonging to different channels are mixed in one moving image,
   By performing image processing on a plurality of frames included in the moving image, a similarity between frames related to a local color tendency and / or an overall color tendency is calculated, and the plurality of frames is determined according to the similarity Classifying the channel into a plurality of channels,
   Image processing method.

Images