JP2005236891A - Image processing apparatus and method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that a moving image part desired by a user is not efficiently retrieved only by integrating and listing shots for viewing with the shot corresponding to the cut of a commercial moving image as a unit, since the lengths of the shots of the moving image photographed by a video camcorder has length largely distributed from several seconds to one hour, which is long, and a cut point image is not always suitable for representing the shots. <P>SOLUTION: The moving image with the boundary of the shot defined therein is supplied with a frame as a unit (S801). One or more representative frames are detected from the shot on the basis of the correlation of the supplied frame (S803-S806). Indexes corresponding to link information to the moving image are generated, so as to generate a group which is obtained by grouping the generated indexes by prescribed criterion (S806). Information concerning the moving image, the index, and the group are recorded in a recording medium (S808). <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and method, and relates to, for example, image processing for generating and recording a moving image index and reproducing a moving image according to the generated index.

  In recent years, with the widespread use of digital cameras, mobile phones, digital video camcorders (DVC), and the like that can shoot moving images, individuals have been able to shoot large amounts of moving images. In general, the amount of data of video data is enormous. However, when the outline of the content is known or when a desired scene is searched for, the video is conventionally fast-forwarded or rewinded.

  The fast forward and rewind playback described above is not an efficient method, and in order to improve efficiency, for example, image data of cut points described in Japanese Patent Publication No. 5-74273 is detected as a representative screen, and an index image is obtained. A method of making was invented. This cut point generally refers to a point that becomes a joint between materials in an edited moving image. In the case of a movie shot with DVC, the unit of a series of movies from the start to the end of recording after pressing the shooting button is called a shot, and the beginning of the shot is the cut point.

  A video camera for analog recording on a magnetic tape produces a moving image in which shots are continuous. In this case, the cut point can be detected by utilizing the abrupt change of the frame image at the shot boundary. That is, it is only necessary to detect a point where the difference between the images of adjacent frames is extremely large and set it as a cut point.

  For movies and movies used in television broadcasting (hereinafter referred to as “commercial movies”), the cut (unit of a series of movies) is at most several seconds, and there are hundreds of cuts in one program. Since the cut is short, an image of the frame of the cut point (hereinafter referred to as “cut point image”) can be used as the representative image. However, when the cut point images are displayed as a list, the number of the cut point images becomes enormous, and shots are integrated and grouped and displayed for convenience of scene search. For example, in Japanese Patent Laid-Open No. 2002-27410, in order to reduce the number of cut point images to be listed, a thinning process is performed so that the time intervals of those frames are not dense and similar frames are not selected. A method of performing is disclosed.

  Japanese Patent Laid-Open No. 2000-36966 discloses a method of detecting cut points in the same manner and performing grouping based on the correlation of cut point images. When such a group structure is displayed on the display, other cut point images are displayed around the first cut point image in the form of a child image. Furthermore, Japanese Patent Laid-Open No. 9-293139 discloses a method of hierarchizing by clustering each frame of a moving image.

  However, shots of videos shot with video camcorders (hereinafter referred to as “personal videos”) range from a few seconds to longer in one hour, and the length of the shot is more dispersed than the length of the commercial movie. Is big. Further, since the shot is long, the cut point image may not necessarily be an image suitable for representing the shot. For this reason, if a shot corresponding to a cut of a commercial moving image is integrated and listed as a unit, a moving image portion desired by the user cannot be efficiently searched.

  In addition, since the method disclosed in Japanese Patent Laid-Open No. 9-293139 performs processing on all frames, the moving image is approximated by using a representative frame representing a certain moving image section regardless of the cut or shot boundary. The accuracy is higher than the method of hierarchizing. However, the processing cost is high, and it is impossible to mount on a device such as a video camcorder, which is mainly operated by a battery, because the performance balance as a product is greatly broken.

  In addition, the method disclosed in JP 2000-36966 A for displaying a similar image in a smaller area in the list of representative frames is too small for a small display of a digital camera, a mobile phone, or a video camcorder. It is difficult to discriminate, and if the image is enlarged, the listability is impaired.

JP 2002-27410 JP JP 2000-36966 JP JP-A-9-293139

  An object of the present invention is to solve the above-mentioned problems individually or collectively, and to search for a desired scene of a moving image at low cost and efficiently.

  Another object is to enable efficient search even when the display screen is small.

  The present invention has the following configuration as one means for achieving the above object.

  An image processing apparatus according to the present invention detects a one or more representative frames from a shot on the basis of a supply unit that supplies a moving image in which a shot boundary is defined for each frame, and a correlation between the supplied frames. An index generation unit that generates an index that associates link information with a group, a grouping unit that generates a group in which the generated index is grouped according to a predetermined criterion, and records the moving image, the index, and information about the group And recording means for recording on a medium.

  An image processing method according to the present invention supplies a moving image in which a shot boundary is defined for each frame, detects one or more representative frames from a shot based on the correlation of the supplied frames, and links to a moving image. An index in which information is associated is generated, a group in which the generated index is grouped according to a predetermined criterion is generated, and the information about the moving image, the index, and the group is recorded on a recording medium.

  According to the present invention, it is possible to efficiently search for a desired scene of a moving image at low cost.

  In addition, even when the display screen is small, efficient search can be performed.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described in detail with reference to the drawings.

[Constitution]
FIG. 1 is a block diagram illustrating a configuration example of a digital camcorder using a removable magneto-optical disk type recording medium according to the first embodiment.

  The main control unit 201 includes a CPU 202, a ROM 203, a RAM 204, and the like, and the CPU 202 executes various control processes stored in the ROM 203, thereby including various processes described below. Execute control. Various processes shown in FIG. 2 are stored in the ROM 203 as a control program. Specifically, a drive control unit 301 for controlling the drive unit 205, a moving image recording unit 302 for recording and reproducing a moving image on the magneto-optical disk 206 attached to the drive unit 205, and a moving image stored on the magneto-optical disk 206 A time code counter 303 for updating a time code for time management, a date timer 304 for acquiring a shooting date and time, an index generation unit 305 for generating an index of a moving image stored in the magneto-optical disk 206, an index group A grouping unit 306 and a navigation unit 307 are generated.

  The drive unit 205 includes a mechanical mechanism, a servo mechanism, an optical pickup, and the like, and is controlled by the main control unit 201.

[Initialization]
Next, initialization processing when the magneto-optical disk 206 is mounted in the drive unit 205 will be described.

  The drive control unit 301 that has detected the mounting of the magneto-optical disk 206 stores nonvolatile information (media management information 401, shot management information 402, index management information 403, and group management information 404) on the magneto-optical disk 206 shown in FIG. Read into memory 211.

  FIG. 4 is a diagram showing a storage state of the nonvolatile memory 211 after information 401-404 is read from the magneto-optical disk 206. The camcorder ID 501 is an identifier for identifying individual camcorders, is expressed by nine-digit numbers, and is recorded when the camcorder is manufactured. The media counter 502 is a counter that counts up the number of unused media handled by the camcorder, and stores “0” when the camcorder is manufactured. The media management information 401 read from the magneto-optical disk 206 includes information such as the media ID 503, the latest update date and time 504, the maximum time code 505, the shot number 506, and the index number 507.

  When the media control information 401 is read into the nonvolatile memory 211, the drive control unit 301 determines whether or not the mounted magneto-optical disk 206 is unused. Since the media ID 503 of the magneto-optical disk 206 is set to “0” at the time of manufacture, if the value of the media ID 503 is “0”, it is determined that the magneto-optical disk 206 is unused. In the case of an unused magneto-optical disk 206, the initial value “0” is assigned to the maximum time code 505, the shot number 506, and the index number 507, respectively, and then the media counter 702 is incremented. Furthermore, an ID for identifying the medium is newly generated, and the generated ID is written in the medium ID 503. The ID to be generated is, for example, a 12-digit code in which the camcorder ID (nine digits) recorded in the nonvolatile memory 211 is the upper nine digits and the count value (three digits) of the media counter 502 is the lower three digits. There is no limit to the number of digits such as ID. If it is determined that the magneto-optical disk 206 is not unused, the media counter 502 and the media ID 503 are not updated.

[Recording video]
Next, a moving image recording process will be described.

  When the shooting start button of the input unit 212 is pressed, the reflected light of the subject projected on the image sensor 208 through the optical system 207 is converted into an electrical signal. All pixels in one frame are read out sequentially from the image sensor 208 and input to the imaging system processor 209. After performing noise removal and automatic gain adjustment, the imaging system processing unit 209 generates YPrPb data that is A / D converted and separated into a luminance signal Y and color difference signals Pr and Pb. This YPrPb data is stored in the buffer memory 210, and then compressed by the moving picture recording unit 302 by a compression algorithm using discrete cosine transform (DCT). Finally, a synchronization code, an error correction code, and the like are added to the drive unit 205. To the moving image recording area 405 (see FIG. 3) on the magneto-optical disk 206. In this way, the recording of the moving image is continued until the shooting end button of the input unit 212 is pressed.

  On the other hand, when the shooting start button is pressed, the servo mechanism is stabilized, and the above-described recording of the moving image is started, the time code counter 303 starts counting up in units of frames, and the index generation unit 305 generates an index. The generated index is recorded as index management information 403 in the nonvolatile memory 211. The grouping unit 306 groups the indexes and records them as group management information 404 in the nonvolatile memory 211. This index generation and recording, and grouping and recording will be described in detail later.

  When the recording of the moving image is started, the drive control unit 301 increments the shot number 506 of the nonvolatile memory 211, and newly generates shot information at the end of the valid shot information in the shot management information 402. Add As shown in FIG. 5, the shot management information 402 includes a plurality of pieces of shot information, and each piece of shot information includes a shot number, a time code, and a shooting date / time. The generated shot information includes the value of the shot number 506 as the shot number, the value of the maximum time code 505 as the time code, and the value of the date / time timer 304 at the timing when the shooting start button is pressed as the shooting date / time.

  Further, when the recording of the moving image is ended by the shooting end button, the drive control unit 301 stops the count up of the time code counter 303, and the value of the time code counter 303 at that time is set as the maximum time code 505 in the nonvolatile memory 211. Store. Further, after the value of the date / time timer 304 is written in the nonvolatile memory 211 as the latest update date / time 504, the media management information 401, shot management information 402, index management information 403, and group management information 404 on the nonvolatile memory 211 are magneto-optically recorded. Write to disc 206.

[Record index]
FIG. 6 is a flowchart for explaining the index recording process of the index generation unit 305.

  When the shooting start button of the input unit 212 is pressed and the above-mentioned moving image recording is started, one frame of YPrPb data of the moving image stored in the buffer memory 210 is extracted (S801), and whether or not the frame has been extracted. Is determined (S802). If the frame can be extracted, the first feature amount is extracted from the image data of the extracted frame (S803). The feature amount to be extracted is obtained by associating the average color of each block obtained by dividing the frame into, for example, N × M blocks, with the position of each block. Next, the image data, the feature amount, and the time code are stored in the cache buffer assigned to the RAM 204 as shown in FIG. 7 (S804).

  The cache buffer is composed of a ring buffer and is initialized at the start of index generation. The size of the cache buffer is the larger one of the sizes necessary for determining a search range or a frame for which correlation is to be determined, which will be described later, in the representative frame extraction process.

  Next, by determining whether or not a predetermined amount of data has accumulated in the cache buffer (S805), the processing of steps S801 to S804 is repeated until the data amount has accumulated.

  When a predetermined amount of data is accumulated or when it is determined that the frame has not been taken out by pressing the shooting end button (S802), representative frame extraction processing is performed (S806). The outline of this process is to focus on one frame buffered in the cache buffer, evaluate whether the frame of interest can be a representative frame, and if it can be a representative frame, refer to the neighboring frames. Thus, it is determined whether or not the frame of interest is suitable for the representative frame, and if it is determined that the frame is suitable, it is added to the index.

  Next, it is determined whether the evaluation has been performed up to the last frame (S807). For example, if the recording end button is pressed and the recording ends, and there is no unprocessed frame on the buffer memory 210, it can be determined that the evaluation has been performed up to the last frame. Since there is a delay between the frame fetched by the cache buffer and the frame to be evaluated as the representative frame, the same is true even if the frame cannot be fetched as much as the representative frame has not been evaluated for a certain period of time after starting index generation in the previous S805. The time is evaluated. If not processed to the last frame, the process returns to S801, and if processed, the process proceeds to S808.

  Finally, the index information is recorded in the index management information 403 of the nonvolatile memory 211 (S808), and the process ends.

[Representative frame extraction processing]
FIG. 8 is a flowchart for explaining representative frame extraction processing (S806).

First, a frame for which a correlation is obtained is determined (S901). The frame for which the correlation is obtained is the frame of interest and the frame of the time code tc obtained by the following equation.
tc = tp + a + b (tn-tp)
Where tc <tn
tn is the time code of the frame of interest
tp is the time code of the last representative frame
a ≥ 0, 0 ≤ b ≤ 1

  The time code tn is initialized to “0” at the start of index generation, and increases by one frame each time this process is called.

  Further, a and b are parameters for adjusting the detection frequency of the representative frame. For example, when a = 3 (seconds) and b = 0, it is possible to prevent a large number of frames from being detected locally in a section where the motion is intense as compared with the case where a = 0 and b = 0. If a = 0 and b = 0.5, the center of the immediately preceding representative frame and the target frame is referred to. As a result, the amount of detection of the representative frame can be suppressed even in a place where the movement is small as compared with the case where a = 0 and b = 0. The values of both coefficients a and b may be determined at the time of index generation, or may be controlled during index generation based on the frequency of the detected representative frame. However, if b = 0, the cache buffer size need only be able to store the feature value for one frame. For example, if b = 1, the size of the feature value for the entire video must be cached. .

  If a sufficient memory size cannot be secured as a cache buffer, it can be easily solved by the following method. In the first method, a maximum value is set for the interval between representative frames, and when a representative frame is not detected for a predetermined time or longer, a frame after the elapse of a predetermined time from the last representative frame is output as a representative frame. In the second method, the resolution of the time code tc is reduced from one frame unit to, for example, five frame units, and the frame storing the feature amount is thinned out to one fifth.

  Next, it is evaluated whether or not the frame of interest is the head of a shot (S902). Since the first frame of a shot is always adopted as a representative frame, in this case, the process proceeds to step S907. If the frame of interest is not the first frame of the shot, if the frame for which correlation is to be determined cannot be specified (for example, when tc ≧ tn), the process ends (S903).

If the frame for which correlation is to be determined is specified, correlation determination processing is performed (S904). Specifically, the frame feature amounts corresponding to the time codes tn and tc are acquired from the cache buffer, and the sum of the distances d between the feature amounts of the N × M blocks is obtained by the following equation.
d = Σ _{n = 1} ^N Σ _{m = 1} ^M (w1 × | Yi (n, m)-Yq (n, m) |
+ w2 × | Pri (n, m)-Prq (n, m) |
+ w3 × | Pbi (n, m)-Pbq (n, m) |}
Where n and m are the horizontal and vertical index numbers of the block
Yi (n, m) is the brightness of the average color of the n × mth block
Pri (n, m), Pbi (n, m) is the color difference of the average color of the n × mth block
Yq (n, m) is the brightness of the average color of the (n, m) block in the extracted feature
Prq (n, m), Pbq (n, m) is the color difference of the average color of the above (n, m) block
w1, w2, and w3 are the weights for calculating the distance d.

  Note that the values of the weights w1, w2, and w3 are adjusted so as to improve accuracy. Then, it is determined that the smaller the distance d, the more similar the feature amount and the higher the correlation between frames.

  Next, the distance d is compared with a predetermined threshold value S (S905). If d ≦ S, the target frame and the immediately preceding representative frame are similar, and the process ends. The threshold value S may be determined at the time of index generation, or the threshold value S may be controlled during index generation based on the frequency of the detected representative frame. If d> S, a search process described later is performed in order to search for a frame more suitable as a representative frame near the time code tn (S906).

  Next, the detected information is set in each item of the index management information 403, a reduced image (thumbnail image) is created from the image data of the representative frame on the cache buffer, and the index management information as shown in FIG. (S907), and the process ends. Note that the reduced image may be created when the image data is registered in the cache buffer, but in that case, a reduced image is created for all frames, so that the reduction rate and the assignable memory This is determined in consideration of the reusability of data generated in the process of extracting the quantity and feature quantity, the processing cost of the reduction process, and the like.

[Search process]
The purpose of the search process is to search for a frame more suitable as a representative frame in the vicinity of the frame selected as a representative frame candidate in the preceding process. For this purpose, the search processing according to the present embodiment obtains the correlation between frames separated by a predetermined interval, and selects a place where there is little motion, that is, a place (frame) with little blur.

  FIG. 10 is a flowchart for explaining the details of the search process.

  First, variables are initialized (S1001). That is, the time code tn is set to the variable t indicating the frame position serving as a reference, and “∞” is set to the variable min_d indicating the distance of the location with the least movement.

  Next, "0" is substituted for the variable i for controlling the loop, and the value indicating the search range is substituted for the variable k (S1002), the loop end determination is performed (S1003), and the loop is ended (i = k) In this case, the search process is terminated and the search result is output. Here, k is a predetermined value indicating the upper limit of the search range.

  Next, it is determined whether or not to perform a forward search (S1004). This determination is based on the fact that the next frame for which the distance d is to be obtained is the previous representative frame, or that the search is not required if it is earlier than that, and depending on the value of the coefficient a in the process of step S901, Since the direction frame has a higher correlation with the immediately preceding representative frame, the forward search range may be narrower than the backward search range. In that case, it may be determined that the search is unnecessary when the variable i exceeds a predetermined threshold value less than the search range k. If no forward search is necessary, the process proceeds to step S1008.

  When the forward search is necessary, the feature amount of the frame of time code (t-i-n) and (t-i) is obtained from the cache buffer, and the distance d is obtained (S1005). The distance d is obtained by the same method as in step S904. However, n is a predetermined integer that is not zero. Also, when shooting in a dark place, there is an imaging system that secures brightness by increasing the interval at which images are extracted from the imaging device. In this case, frames with the same contents continue, so n is set to near zero. That is not suitable. Then, the calculated distance d is compared with the minimum value min_d of the distance so far (S1006) .If d <min_d, it is determined that the motion of the video is the smallest in the search range so far, and the search result is Update (S1007). That is, d is substituted for min_d, and the frame at the position of the time code t−i is set as a representative frame candidate.

  Next, it is determined whether or not a backward search is performed (S1008). This determination takes into consideration that the search is not required when the next frame whose distance is to be obtained is the next shot. If no backward search is necessary, the process proceeds to step S1012.

  If backward search is required, obtain the distance d from the cache buffer by obtaining the feature quantities of the frames of time code (t + in) and (t + i) (S1009). The minimum value min_d is compared (S1010). If d <min_d, it is determined that the motion of the moving image is the smallest in the search range so far, and the search result is updated (S1011). That is, d is substituted for min_d, and the frame at the position of time code t + i is set as a representative frame candidate.

  Next, the search range is expanded by one frame by incrementing the variable i (S1012), and the process returns to step S1003.

[Group]
Next, grouping in the grouping unit 306 will be described.

  FIG. 11 is a diagram showing a configuration example of the grouping management information 404. Further, during grouping, in addition to the information shown in FIG. 11, the average feature value of the frames that are members of each group is held as the representative feature value of each group.

  FIG. 12 is a flowchart for explaining details of grouping.

  First, each representative frame is initialized as a temporary group having the number of members “1”, and the feature amount of each representative frame is used as the group feature amount for obtaining the correlation (S1501).

  Next, a set of groups having a minimum inter-group distance is obtained from all groups, and the distance is set as min_d (S1502). The distance d is obtained by the same method as in step S904. In addition, in order to speed up the nearest-neighbor search of multidimensional vectors, multidimensional index techniques are actively studied, but these techniques can also be used.

  Next, min_d is compared with a predetermined threshold value S2 (S1503), and if min_d ≧ S2, further grouping is impossible and the process proceeds to step S1507. If min_d <S2, the most similar groups (with the smallest inter-group distance) are combined into one (S1504). That is, an index that is a member of each group is incorporated into one group. Next, the feature values of each member of the combined group are averaged to obtain a representative feature value (S1505), and it is determined whether or not the number of groups has been reduced to the desired number (S1506). Return to step S1502.

  When grouping is impossible or the number of groups reaches a desired number, a group having only one member, that is, a group that is isolated without being joined is deleted (S1507), and a representative index of each group is obtained (S1508). As the representative index, an index having the smallest time code among the members of the group is adopted. Then, the representative index number and the like are recorded in the group management information 404 (S1509), and the process ends.

[Example of index output]
Next, an example of index output by the navigation unit 307 will be described. When the moving image navigation button of the input unit 212 is pressed, the index management information 403 and the group management information 404 are read from the nonvolatile memory 211 and a list of indexes is displayed on the display unit 213.

  FIG. 13 is a diagram showing an example in which a list of indexes is output to the display unit 213. The index and the representative index of the group are arranged in the order of time code to the video from the upper left to the horizontal direction. However, in the case of a grouped index, the time code of the representative index of the group is adopted. Note that the display indicated by reference numeral 1601 in FIG. 13 is a reduced display of the grouped representative index images. Reference numeral 1602 denotes a modification display indicating a group, and a state in which three images overlap is designed. Reference numeral 1603 is a reduced display of an image of an isolated index that is not grouped. If the index does not fit in the screen of the display unit 213, the entire index can be displayed by scrolling or screen switching based on the instruction of the input unit 212.

  FIG. 14 is a diagram showing another form of index list display. As a display indicating the group, an icon as indicated by reference numeral 1604 is displayed instead of the overlapping of the images indicated by reference numeral 1602. It should be noted that the index layout method can be modified in many ways. In short, it is only necessary to indicate to the user whether or not the index is a group when the index is displayed as a list.

[Example of playback]
Next, an example of selection and reproduction by the navigation unit 307 will be described. A feature of this moving image reproduction is that different moving image reproduction is performed depending on the method of selecting the grouped index, and the user navigates interactively to a desired moving image.

  The index displayed as a list on the display unit 213 can be in a focused state only at one place. In the case of a group representative index, there are two attention states: an image attention state and a group attention state. FIG. 15 is a diagram showing a change in display of each part of the index list display in the attention state. The thickness of the image frame and the size of the icon are highlighted so that the user can easily recognize the image of interest and the index. Of course, since the purpose is to emphasize the image of interest or the index, an emphasis display method by animation such as changing the color of the frame or icon or rotating the icon may be used.

  The user can freely move the target image or group by operating the input unit 212. For example, FIG. 16 is a diagram showing the moving direction of the image of interest or index when the input unit 212 has a cross direction key having four buttons, upper, lower, left, and right. If the image in the left column of the center row is the image of interest, when the up, down, left direction key is pressed, the state of interest shifts to an image (or group) in that direction. If you press the right arrow key, the center image (or group) will be in the focused state. If you press the right arrow key again, the right column image in the center row will be the focused image. In the code order, the image in the left column in the lower row becomes the attention image. Although not shown, when a key in the direction toward the outside of the screen (up or down) is pressed, the screen is switched or scrolled in the vertical direction as necessary, and any index among all indexes or group representative indexes is displayed. It is possible to make the state of attention.

  When a pointing device that can specify coordinates on the screen, such as a mouse or a trackpad, is available as the input unit 212, the coordinates included in the group representative index image area are specified, and the index image area When the coordinates inside are designated, the image is noticed. When the coordinates in the area of the decoration display 1602 or the icon 1604 are selected, the group attention state is set. In addition, if a scroll area (not shown) is defined and displayed on the screen of the display unit 213, the coordinates of the scroll area are instructed, and either one of the representative indexes of all indexes or groups is focused. It is possible to

  In the image attention state, when the user operates the OK button of the input unit 212 to instruct the transition to the next process, the moving image recorded on the magneto-optical disk 206 is accessed, and the moving image is recorded based on the time code indicated by the index. Cue playback (display on display unit 212) is performed. Similarly, when a transition to the next process is instructed in the group attention state, each member of the attention group shifts to a display state in which details can be confirmed.

  FIG. 17 is a diagram illustrating an example of a display screen of the display unit 212 when the display state is shifted to a display state in which each member of the group can be confirmed.

  The members denoted by reference numerals 1902, 1905, and 1908 in FIG. 17 are obtained by arranging the time stamps of the members of the group from top to bottom in ascending order. Then, the front and rear indexes of each member are displayed from the members 1902, 1905, and 1908 in the left-right direction. That is, reference numeral 1901 is the index image of the time stamp immediately preceding member 1902, and reference numeral 1903 is the index image of the time stamp immediately following member 1902. Similarly, reference numerals 1904 and 1906 are index images before and after the member 1905, and reference numerals 1907 and 1909 are index images before and after the member 1908. Although not shown, also in this case, as described with reference to FIG. 16, the screen can be switched or scrolled to display the index images before and after all the members of the group. Depending on the size of the display screen, a plurality of index images may be displayed before and after.

  Further, the member 1902 whose frame is highlighted is in a focused state. Here, when a transition to the next process is instructed, the moving image is cued and reproduced from the time code of the image in the attention state, as described with reference to FIG.

  FIG. 18 is a diagram showing another example of the display screen of the display unit 212 when each member of the group is transferred to a state where it can be confirmed.

  The members denoted by reference numerals 2001 and 2002 in FIG. 18 are the index images of the members of the selected group, and the moving images of a predetermined time before and after those time stamps are repeatedly reproduced continuously on the multi-screen 2003 denoted by reference numeral 2003. . As another example, a moving image of a predetermined time before and after the time stamp of the group member may be continuously reproduced on one screen.

  FIG. 19 is a diagram showing another example of the display screen of the display unit 212 when each member of the group is transferred to a state where it can be confirmed.

  FIG. 19 shows a layout of images and the like on the display unit 212 so that more information can be provided. Reference numeral 2101 denotes a moving image playback area, reference numerals 2102-2104 denote frame images of time codes of members of the selected group, and a frame image 2103 with a highlighted frame is a member in a focused state. A bar 2107 indicates a timeline indicating the entire moving image, a marker 2104-2106 indicates where all members of the group are in the entire moving image, and a black marker 2105 indicates a position corresponding to the frame image 2103 of interest. A marker 2108 is a marker indicating a position being reproduced in the reproduction area 2101.

  The frame of interest can be moved to the previous and next members with the left and right keys of the cross key of the input unit 212, and the members that cannot be displayed can be displayed by screen switching or scrolling. In addition, if there is a pointing device in the input unit 212, the corresponding group member can be brought into the focused state by designating the coordinates in the area of the frame image 2102-2104 and the marker 2104-2106. At that time, the image of the member in the attention state is displayed as the center frame image 2103, and the members before and after the image are displayed as the frame images 2102 and 2104. Of course, when the frame of interest moves, the marker 2105 also moves to the corresponding position, and playback of the moving image starts from that position.

  There are many possible variations in the number, size, position, etc. of each part in this layout, but the main point is that the attention state and the index and group members before and after that, the playback screen, and the overall position of the index or group member Of these three marker displays, at least two may be displayed on the screen simultaneously. Further, this layout may be used as an index output in place of FIG. 13 or FIG. In other words, an index list is output in the area of the frame image 2102-2104, and information indicating whether or not it is a group may be represented by the presence or absence of a modification or icon indicated by reference numerals 1602 or 1604 in FIGS.

  FIG. 20 is a block diagram showing another configuration example of the digital camcorder (or digital camera). The difference from the first embodiment is that a card-type non-volatile memory (memory card) 2301 is used as a recording medium and that the drive unit 205 is omitted because there is no mechanical part. The basic operation is the same as that of the first embodiment except that there is no movable part. When handling a moving image, the same processing as in the first embodiment can be performed only by giving the memory card 2301 the same configuration as that of the magneto-optical disk 206 in the first embodiment (see FIG. 3).

  The image processing according to the second embodiment of the present invention will be described below. Note that in this embodiment, the same reference numerals as those in the first embodiment denote the same parts, and a detailed description thereof will be omitted.

  FIG. 21 is a block diagram illustrating a configuration example of a computer according to the second embodiment, in which the optical system 207, the imaging device 208, the imaging system processing unit 209, the buffer memory 210, and the nonvolatile memory 211 are omitted, and a hard disk 2502 is connected instead. And the configuration of the first embodiment is different in that it can be connected to another computer 2504 via a communication interface (I / F) 214 and a network 2503.

  The configuration (computer) of the third embodiment does not have a function of directly capturing and recording a moving image, but a moving image stored in the magneto-optical disk 206 or a moving image stored in another computer 2504 is transmitted via the network 2503. Can be stored in the hard disk 2502. In the first embodiment, the media management information, shot management information, and index management information stored in the magneto-optical disk 206 (or memory card 2301) are stored in the hard disk 2502 and read into the nonvolatile memory 211. The media management information, shot management information, index management information, and group management information are read into a predetermined area of the RAM 204.

  Then, for the video stored in the hard disk 2502, the index generation unit 305 can create and update media management information, shot management information, index management information, and group management information as in the first embodiment. is there. This processing may be performed when moving images are stored in the hard disk 2502.

  Since the second embodiment is a computer, each processing program in the moving image recording unit 302 and the index generation unit 305 is stored in the magneto-optical disk 206 in advance and is read into the RAM 204 when the computer is started up. It may be executed as a processing program.

[Modification]
In the above, an example in which a time code is used as information indicating the start position or start time on the recording medium has been described, but other information, for example, real time (clock) may be used. Alternatively, the frame number from the beginning may be used.

  In the above description, an example in which a feature amount is obtained from a supplied frame image in the index generation method and the index selection method has been described. However, certain processing such as color conversion, edge extraction, trimming, and masking is performed on the frame image. Alternatively, an image obtained secondarily may be used. Further, it may be averaged over time with neighboring frames. By averaging over time, it is possible to detect a representative frame that suppresses the influence even if it differs from the previous and subsequent frames by noise or non-volatility.

  As an example of the feature amount for obtaining the correlation, in the above description, an example of using grid-like block division has been described, but the first correlation calculation method with a low processing cost and a narrow correlation allowable range, and the processing cost is lower than that, As long as the second correlation calculation method having a wide allowable range is combined, the correlation calculation method and the feature amount obtained from the frame image may be other methods. For example, image feature descriptors such as colors, textures and shapes defined in ISO / IEC 15938-3 may be used.

  In the above description, an example in which a reduced image is recorded as the image data in the index recording has been described. However, the image data itself extracted in step S801 may be used. Further, in the second embodiment, it is possible to access a desired time code in the moving image data in a short time, so that image data can be acquired from the moving image. Therefore, the image data in the index need not be stored. In this case, the data amount of index information can be reduced.

  In addition, the index information may be recorded together with attached information regarding the representative frame. The attached information includes, for example, information indicating whether the representative frame is the head of the shot, voice information, character information obtained as a result of voice recognition, character information obtained as a result of character recognition of characters such as a telop displayed in the frame, There are camera information such as exposure status and camera work information such as pan and zoom. Such information is output together with the representative frame when it is displayed and printed, so that a person can easily find a desired representative frame. Further, the character information can be used for a search process using a keyword in the representative frame.

  In the above description, only the indexes are grouped. However, the groups may be further grouped into hierarchies. Briefly describing a method of grouping groups, a binary tree is generated by recording a process of group integration in a grouping unit. Since the nodes of the binary tree are a set of groups, the feature values of the nodes may be obtained by averaging the feature values of the groups included in the nodes. The distance can be calculated by the same method as in step S904. By repeating the connection of nodes with a short distance, it can be transformed into an n-ary tree with a low depth. The management information may be provided with a table for managing parent-child relationships between groups. For display, the frame image may be displayed with the youngest time code among the nodes as the time code of the node. Information indicating whether or not the child has a child may be indicated by the presence or absence of the modification 1602 or the icon 1604.

[Other embodiments]
Note that the present invention can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, and a printer), and a device (for example, a copying machine and a facsimile device) including a single device. You may apply to.

  Also, an object of the present invention is to supply a storage medium (or recording medium) on which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU or CPU) of the system or apparatus. Needless to say, this can also be achieved by the MPU) reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is determined based on the instruction of the program code. Needless to say, the CPU of the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above.

Block diagram showing a configuration example of a digital camcorder using a removable magneto-optical disk type recording medium of Example 1, The figure which shows the control program of various processing which is stored in ROM, The figure which shows the various information stored in the magneto-optical disk, The figure which shows the storage state of the non-volatile memory after information is read from a magneto-optical disk, Figure explaining shot management information, A flowchart for explaining an index recording process by the index generation unit; The figure explaining the storage state of the cache buffer, A flowchart for explaining representative frame extraction processing; The figure explaining index management information, A flowchart for explaining the details of the search process; A diagram explaining grouping management information, Flowchart explaining details of grouping, The figure which shows an example which output the list of indexes to the display part, Figure showing another form of index list display, The figure which shows the change of the display of each part of the index list display in the attention state, Figure showing the moving direction of the image of interest or index, The figure which shows an example of the display screen of a display part at the time of shifting to the display state which can confirm each member of a group, The figure which shows the other example of the display screen of the display part at the time of shifting to the state which can confirm each member of a group, The figure which shows the other example of the display screen of the display part at the time of shifting to the state which can confirm each member of a group, Block diagram showing another configuration example of a digital camcorder (or digital camera), FIG. 6 is a block diagram illustrating a configuration example of a computer according to a second embodiment.

Claims (9)

Supply means for supplying a moving image in which shot boundaries are defined in units of frames;
Index generating means for detecting one or more representative frames from the shot based on the correlation of the supplied frames and generating an index that associates link information to the video;
Grouping means for generating a group in which the generated index is grouped according to a predetermined criterion;
An image processing apparatus comprising: a recording unit that records information on the moving image, the index, and the group on a recording medium.   Further, a list of images corresponding to frames of moving images recorded on the recording medium is output for display according to the information on the index and group recorded on the recording medium, and the image selected from the list is supported. 2. The image processing apparatus according to claim 1, further comprising a navigation unit that reproduces the moving image based on an index.   3. The image processing apparatus according to claim 1, wherein the grouping unit performs the grouping based on similarity of the frame images of the index.   4. The image processing apparatus according to claim 1, wherein the grouping unit determines an index representing the group from the group of the indexes.   The navigating means outputs, for display, a symbol capable of determining whether or not an image in the list is a group of the index, and an image corresponding to a frame corresponding to the representative index as an image of the index group 5. The image processing device according to claim 4, wherein the image processing device is output for display.   6. The image processing apparatus according to claim 5, wherein the navigating unit reproduces the moving image by a different method depending on whether the symbol is selected or an image in the list is selected. Provide a moving image with shot boundaries defined in frame units,
Based on the correlation of the supplied frames, one or more representative frames are detected from the shot, and an index that associates link information to the video is generated.
Generate a group by grouping the generated index according to a predetermined standard,
An image processing method, wherein information relating to the moving image, the index, and the group is recorded on a recording medium.   8. A program for controlling an information processing apparatus to execute image processing according to claim 7.   9. A recording medium on which the program according to claim 8 is recorded.

Images