JP2005175532A - Video processing apparatus and representative image setting method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video processing apparatus capable of easily changing a representative image of a similar video section similar to a video section whose representative image is changed by an operator. <P>SOLUTION: In an image pickup apparatus, video data are separated into a plurality of video sections, and a key frame is automatically extracted from the separated video sections, and if the extracted key frame is changed by the operator, a key frame extracting method is obtained in a video section where the key frame is changed. A similar video section similar to the video section where this key frame is changed is extracted, and the extracted key frame is added as a key frame of the similar video section to the already extracted key frame. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a video processing apparatus and a representative image setting method for acquiring a representative image for identifying a video section from video data.

  Conventionally, as this type of video processing apparatus, video data recorded on a storage medium is automatically divided into a set (video section), and at least one still picture data is represented from each of the divided video sections. A list of images (referred to as “key frames”) is known. FIG. 58 is a diagram showing a state in which key frames are extracted from the video segments divided for each video section. By displaying a list of key frames, the operator can quickly confirm the contents of the stored video data. In addition, editing work for each divided video section can be easily performed.

  Conventionally, as a method of dividing video data, for example, a method of automatically dividing a portion where a video changes rapidly with time as a break of a scene is known. Also, as a method for extracting a representative image, a method for extracting the first frame, end frame or intermediate frame of a divided scene as a representative image, a method for automatically estimating and extracting a representative image that seems to be important by camera operation It has been known.

  However, the method of automatically dividing the video data and automatically extracting the key frame from the divided video section as described above does not necessarily result in an optimal key frame for the operator. For this reason, an operation for changing the key frame for each automatically divided video section is required. For example, as a method of changing the key frame, the operator selects a key frame to be changed, plays the video before and after the scene change of the scene to which the key frame belongs, and displays a list of each frame image of the played video. There is known a method for changing a key frame so as to set a key frame (for example, see Patent Document 1).

  Conventionally, a fingerprint recognition device is known as a fingerprint recognition technique (see, for example, Patent Document 2). In this fingerprint recognition device, a shadow generated by a finger pressed against the sensor unit is detected as a fingerprint by the sensor unit, and this fingerprint is collated with fingerprint data in a user recognition data list stored in advance.

  As a conventional technique, a camera that recognizes the face of a subject is known (see, for example, Patent Document 3). This camera not only recognizes the face of the subject, but also the direction of the face (the line of sight is facing / not facing the camera), the state of the subject's eyes (the eyes are closed / not closed), Facial expressions (joy face, sad face, angry face, surprise face, etc.), subject posture (recognizes whether a recognized person is in a specific pose), etc. can be recognized.

Similarly, an apparatus equipped with a person detection function for recognizing a subject is known (for example, see Patent Document 4). In this person detection function-equipped device, it is possible to identify the speaker from the recognition of the subject and the voice information.
JP 09-093527 A JP 2002-123821 A JP 2001-51338 A JP 2001-67098 A

  However, the above-described conventional video processing apparatus has the following problems, and improvements have been demanded. That is, when changing the key frame, one key frame changing operation is only for one video section. Therefore, if the automatically set frame cannot be satisfied as a key frame, the operator has to perform a change operation for each of the key frames in all video sections that cannot be satisfied. When the number of video sections to be changed is large, the number of times of change increases, and the change operation is troublesome for the operator. In particular, even if there are many similar video sections, the operator has to repeat the same changing operation.

  Therefore, an object of the present invention is to provide a video processing apparatus and a representative image setting method capable of easily changing a representative image in a similar video section similar to the video section in which the operator has changed the representative image. To do.

  In order to achieve the above object, a video processing apparatus of the present invention acquires a representative image changing means for changing a representative image extracted from video data, and a representative image extracting method in a video section in which the representative image is changed. Representative image extraction method acquisition means, similar video section extraction means for extracting a similar video section that is a video section similar to the video section in which the representative image is changed, and representative image extraction in the video section in which the representative image is changed Using the method, a representative image of the extracted similar video section is extracted, and the extracted representative image is used as a representative image of the similar video section as a representative image of the similar video section extracted by the representative image extracting means. And a similar video section representative image adding means for adding to the representative image.

  The representative image setting method of the present invention includes a representative image changing step of changing a representative image extracted from video data, and a representative image extracting method acquiring step of acquiring a representative image extracting method in a video section in which the representative image is changed. The extraction using a similar video section extraction step of extracting a similar video section that is a video section similar to the video section in which the representative image is changed, and a representative image extraction method in the video section in which the representative image is changed. A similar image that extracts a representative image of the similar video section and adds the extracted representative image as a representative image of the similar video section to the representative image of the similar video section extracted in the representative image extraction step A section representative image adding step.

  ADVANTAGE OF THE INVENTION According to this invention, the representative image of the similar video area similar to the video area which the operator changed the representative image can be changed easily.

  Embodiments of a video processing apparatus and a representative image setting method of the present invention will be described with reference to the drawings. The video processing apparatus according to the present embodiment is applied to an imaging apparatus. In this imaging apparatus, the captured video data is divided based on the start / end of the recording operation. This divided video section is called a “shot”. Furthermore, video data in a shot is divided based on attached information such as operation information (operation information such as zoom) and shooting environment information (information such as pan and tilt). Each divided video section is referred to as a “sub-shot”. A key frame (representative image) is automatically extracted from the video section divided for each sub-shot based on the attached information such as the operation information and the shooting environment information described above.

[First Embodiment]
FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus according to the first embodiment. This image pickup apparatus includes a zoom lens 11 that forms an image of an object, a diaphragm 12 that adjusts the amount of light, an image pickup device 13 that includes a CCD that converts input light into an electrical signal, and a sample hold / AGC circuit 14 that performs sample hold and gain adjustment. An A / D conversion unit 15 that performs analog-digital conversion (A / D conversion), and a video signal processing unit 16 that processes the A / D converted signal to generate a video signal.

  The imaging apparatus also divides the video data and manages a video management unit 17 that manages information such as attached information and key frames in the divided video segments, and a display unit that displays captured contents and key frames in each video segment. 18, a lens position detection unit 19 that detects the position of the lens, a lens drive unit 20 that drives the lens, an X-direction rotation detection unit 21 that detects the horizontal rotation of the imaging device, and a vertical rotation of the imaging device. A Y-direction rotation detection unit 22 and a system control unit 23 that controls the entire imaging apparatus are included. Here, the X-direction rotation detection unit 21 performs signal processing on a signal detected by a gyro sensor or an acceleration sensor using a band-pass filter or the like, and then performs A / D conversion to generate an output signal. The Y direction rotation detection unit 22 has the same configuration as the X direction rotation detection unit 21. If a rotation detection unit is mounted for the purpose of image stabilization, it may be used as well.

  The imaging apparatus also includes an operation input unit 24 on which keys for operating zooming and recording start / end, keys for changing key frames, and the like, MPEG_CODEC 28 for encoding video data in MPEG format, and a recording medium. A recording unit 29 configured by a drive for recording information by driving, an optical disk, a magnetic disk, a magneto-optical disk, a magnetic tape, a hard disk, a non-volatile memory, a non-volatile memory card, a recording medium 30, and a fingerprint are detected. A fingerprint detection unit 31 including a fingerprint sensor.

  On the operation input unit 24, a wide key 25 for moving the zoom in the wide (wide-angle, ie, image reduction) direction, a tele key (26) for moving in the tele (telephoto, ie, image enlargement) direction, and the start / end of recording are instructed. There are provided a REC key 27 for selecting, a menu key 32 for instructing key frame change and key frame list display, a cursor key 33 used for selecting a video section for changing the key frame from the key frame list, and the like. . Here, the wide key 25 and the tele key 26 are, for example, seesaw-type interlocking keys, and output to the system control unit 23 an output signal indicating which key is being pressed by what level of pressing force. . The REC key 27 is a toggle-type key. When pressed during recording, the REC key 27 instructs to end recording, and when pressed during non-recording, instructs to start recording.

  FIG. 2 is a diagram illustrating a configuration of the system control unit 23. The system control unit 23 includes a well-known CPU 41, ROM 42, RAM 43, I / O interface 44, bus 45, and interrupt signal line 46. The ROM 42 stores programs to be described later, table values, and the like that are executed by the CPU 41.

  FIG. 3 is a block diagram showing the configuration of the video management unit 17. The video management unit 17 performs a video division unit 56 that divides the captured video data, a key frame extraction unit 57 that extracts a key frame that symbolizes the video section in the divided video section, and a key frame change. A similar video section extraction unit 58 that extracts a video section similar to the video section, and an auxiliary information management unit 59 that acquires and stores the auxiliary information used in the imaging apparatus. Each of these units is realized by executing a program stored in a ROM (not shown) by a known CPU (not shown) in the video management unit 17. Signals exchanged between the operation input unit 24, the display unit 18, the fingerprint detection unit 31, the video signal processing unit 16, the detection units 19, 21, and 22 and the video management unit 17 are controlled by the system control. The system control unit 23 is omitted in FIG. 3 for easy understanding.

  The operation of each part of the imaging apparatus having the above configuration will be described. The light from the subject received through the zoom lens 11 is adjusted in light quantity by the diaphragm 12 and imaged on the surface of the image sensor 13. When converted into an electrical signal by the image sensor 13, the electrical signal is A / D converted by the sample hold / AGC unit 14 and the A / D converter 15 and then input to the video signal processing unit 16. The video signal processing unit 16 performs processing such as aperture correction, gamma correction, white balance correction, and electronic zoom on the input signal for each luminance and color component, and when the video signal is generated, the video signal is The data is output to the video management unit 17 via the control unit 23.

  On the other hand, based on the rotation signal of the imaging device obtained from the X-direction rotation detection unit 21 and the Y-direction rotation detection unit 22, the system control unit 23 performs panning (camera horizontal rotation operation) and tilt (camera vertical direction). Rotation operation) is detected. Shooting environment information such as pan and tilt and operation information from the operation input unit 24 are output to the video management unit 17 via the system control unit 23. In addition, the video management unit 17 stores and holds various types of input information in the attached information management unit 59, and the video division unit 56 uses the video segmentation unit 56 to provide one or more video sections (shots) from the recording start operation to the end operation. Is divided into video segments (sub-shots). The attached information management unit 59 adds sub-shot section information to the input information, and the key frame extraction unit 57 performs key frame extraction processing.

  In this key frame extraction process, a frame image (key frame) suitable for representing each section is determined for each section of the sub-shot based on various pieces of input information. Information on the determined key frame is added to the attached information management unit 59. These pieces of attached information are output to MPEG_CODEC 28. In MPEG_CODEC 28, an input video signal is encoded. At this time, the MPEG_CODEC 28 stores the attached information such as shooting environment information, operation information, and sub-shot information output from the video management unit 17 together with the encoded video signal, and outputs it to the recording unit 29 connected to the MPEG_CODEC 28. To do. Note that a method for storing auxiliary information such as shooting environment information, operation information, and sub-shot information will be described later.

  The recording unit 29 creates one file on the recording medium 30 for each shot. Note that one shot may be saved as a plurality of files, or a plurality of shots may be saved as one file. Further, as long as the recorded digital information can be read, it may be stored by a method that does not depend on the file system.

  The system control unit 23 starts / stops recording in accordance with the operation of the REC key 27. Further, the system control unit 23 controls the lens driving unit 20 according to the operation of the wide key 25 and the tele key 26, and controls the electronic zoom for the video signal processing unit 16. At this time, the zoom magnification can be changed faster as the pressing force is larger. The function of enlarging an image captured by the lens optical system is called optical zoom, and the function of enlarging the image in the vertical and horizontal directions by electrical signal processing regardless of the lens optical system is called electronic zoom.

  Further, the system control unit 23 determines the zoom magnification at the time of optical zoom based on the lens position detection signal from the lens position detection unit 19. Further, the system control unit 23 controls processing in the video management unit 17. In addition, the system control unit 23 removes signal components due to camera shake and the like based on the rotation information output from the X direction rotation detection unit 21 and the Y direction rotation detection unit 22, and displays pan or tilt information by a user operation. Detect and output to the video management unit 17. In addition, the system control unit 23 controls each unit of the imaging apparatus as necessary.

  The fingerprint detection unit 31 detects a shadow formed by a finger pressed against the sensor unit as a fingerprint by the sensor unit, and stores a fingerprint data list (a user ID and fingerprint data registered in pairs). The fingerprint data registered in the user recognition data list). If the fingerprint data matches, a user ID that identifies the operator is acquired. On the other hand, if the fingerprint data does not match, the fingerprint data and the user ID are registered in the user recognition data list as a new user.

  Next, a method for recording attached information such as operation information and shooting environment information together with a moving image will be described. As this recording method, a method of storing attached information in an unused area in image data management information in units of frames is known. Specifically, for example, when the video data is an MPEG-2 stream, there is a method of storing it in the user area of the picture header. In addition, there is a method of adding information generated as a result of calculation after the end of shooting as a footer at the end of a moving image file. For example, in the case of an MPEG-2 transport stream, a TS packet storing attached information in the private area may be added to the end of the file. There is also a method to separate the movie and attached information. In this case, in order to associate the movie with the attached information, the reference information is stored, such as providing a file name generation rule so that the file name can be uniquely obtained between the movie file name and the attached information file name. A hyperlink to the reference information may be pasted.

  In this embodiment, the result of detecting pan is recorded, but instead of recording this detection result, information obtained by quantizing raw information such as an output signal of an acceleration sensor before pan detection, and pan detection Information obtained by quantizing the result of signal processing performed at each stage may be stored. If the quantized information is difficult to process in real time in terms of the processing load of the imaging device (camcorder) body, panning and shooting environment information can be used as operation information and shooting environment information during post-processing by the camcorder body or processing by an external device. Used to generate and record zoom unit information. Further, by using such quantized information, if pan detection can be performed by a pan detection means with improved accuracy in the future, the expandability can be achieved.

  Next, details of the attached information are shown. The data structures shown in FIGS. 4 to 8, 15 to 17, 19, and 21 described later each represent one data element. When there are a plurality of data elements, the attached information is configured as a list accessible to each data element. In the present embodiment, the data structure is represented by a bit string. However, as long as the system control unit 23 can interpret the data structure, a text format or a unique DTD (Document Type Definition) is defined, and XML (Extensible Markup Language) is defined. It may be expressed in a format or the like. Further, in the case of storing in a format suitable for MPEG7, it may be stored in Segment DS, which is a tool for describing the data structure.

  FIG. 4 is a diagram showing the structure of the shooting environment information data. FIG. 5 shows the structure of the shooting environment information data list. The shooting environment information data includes shooting environment information such as pan and tilt acquired from the lens position detection unit 19, the X direction rotation detection unit 21, the Y direction rotation detection unit 22, and the like. As shown in FIG. 5, shooting environment information data is created every time shooting environment information is acquired, and the start address (pointer) of the shooting environment information data is changed every time the frame number from which the shooting environment information is acquired changes. Stored in the information data list. For example, when referring to the shooting environment information data of frame number 1, the first stored frame 1 pointer is acquired from the shooting environment information data list, and the shooting environment information data 1 is referred to.

  FIG. 4 shows each element data of the shooting environment information data. In the element data Time, a frame number based on the start (start) of the playback time of the moving image file when the shooting environment changes is described. The element data DataType describes data representing the event type of the shooting environment information. In the event type of the shooting environment information, pan start, pan continuation, pan end, tilt start, tilt continuation, and tilt end are set. Pan / tilt continuation is used when the speed or direction changes. Here, the element data DataType is “0” to start panning, “1” to continue panning, “2” to end panning, “4” to start tilting, “5” to continue tilting, “6” Represents the end of tilt. The element data Speed exists when the element data DataType is pan start, pan continuation, tilt start, and tilt continuation, and represents the rotation speed of pan and tilt. In this embodiment, the rotational speed is described in seven stages from “1” at the lowest speed to “7” at the highest speed. Element data Direction is present when element data DataType is pan start, pan continuation, tilt start, and tilt continuation, and represents the direction of pan and tilt. In the present embodiment, in the case of pan, “1” represents right and “0” represents left. In the case of tilt, “1” represents the top and “0” represents the bottom.

  FIG. 6 shows the structure of the operation information data. FIG. 7 is a diagram showing the structure of the operation information data list. The operation information data includes operation information such as zoom acquired from the operation input unit 24, the lens position detection unit 19, the lens driving unit 20, and the like. As shown in FIG. 7, operation information data is created every time operation information is acquired, and the start address (pointer) of the operation information data is stored in the operation information data list every time the frame number from which the operation information is acquired changes. Is done. For example, when referring to the operation information data of frame number 1, the first stored frame 1 pointer is acquired from the operation information data list, and the operation information data 1 is referred to.

  FIG. 6 shows each element data of the operation information data. In the element data Time, a frame number based on the start (start) of the reproduction time of the moving image file when the operation occurs is described. In the element data DataType, data representing the event type of the operation information is described. As the event type of the operation information, zoom start, zoom continuation, zoom end, recording start, recording end, and the like are set. Zoom continuation is used when the speed changes. Here, when the element data DataType is “0”, the zoom starts, “1” indicates the continuation of zooming, “2” indicates the end of zooming, “3” indicates the start of recording, and “4” indicates the end of recording. Element data Speed exists when the element data DataType is the start of zooming and the continuation of zooming, and represents the speed of zoom change. In this embodiment, the speed is described in seven stages from “1” at the lowest speed to “7” at the highest speed. Element data Direction is present when element data DataType is the start of zooming or the continuation of zooming, and represents the direction of zooming. Element data Direction “0” indicates the tele side, and “1” indicates the wide side. The element data “Magnification” is present when the element data “DataType” is the start of zooming and the continuation of zooming, and indicates the zoom magnification. The element data “Magnification” is described in 32 levels, “0” for the telephoto side and “31” for the widest side.

  FIG. 8 shows the structure of sub-shot information data. FIG. 9 shows the structure of the sub-shot information data list. The sub-shot information data is information related to the video section divided by the video dividing unit 56. As shown in FIG. 9A, every time a video is divided into sub-shot 1, sub-shot 2,..., Sub-shot M, sub-shot information data is sub-shot as shown in FIG. Information 1, sub-shot information 2,..., Sub-shot information M are created. For example, the sub-shot information 1 includes frame number 1 to frame number 6. The sub-shot information 2 is composed of frame number 7 to frame number 11. The same applies to the subsequent sub-shot information.

  The top address (pointer) of the created sub-shot information data is stored in the sub-shot information data list shown in FIG. When referring to the sub-shot information data 1, the pointer stored in the sub-shot information data 1 is acquired from the sub-shot information data list, and the sub-shot information data 1 is referred to.

  FIG. 8 shows each element data of the sub-shot information data. Element data StartTime represents the start time of the sub-shot, and is described by a frame number with reference to the start (start) of the playback time of the moving image file. Element data EndTime represents the end time of the sub-shot, and is described by a frame number based on the start of the reproduction time of the moving image file. In the element data ContentsDataType, classification information classified based on the attached information in the video section of the sub-shot information is described. Details of this classification information will be described later.

  In the element data NumberOfKeyFrame, the number of key frames set in the video section indicated by the element data StartTime and the element data EndTime is described. In the element data KeyFrame, a frame number based on the start of the reproduction time of the moving image file of the frame serving as the key frame is stored. The element data NumberOfTimes stores the number of times the key frame described by the element data KeyFrame is selected when editing or video reproduction is performed. At the time of key frame extraction, the value 1 is stored in the element data NumberOfTimes. The element data KeyFrame and NumberOfTimes are described by the number indicated by the element data NumberOfKeyFrame.

  FIG. 9C shows the sub-shot information when one key frame is set. The sub-shot section indicated by this sub-shot information corresponds to sub-shot 1 in FIG. When sub-shot 1 is set from frame number 1 to frame number 6, “1” is stored in element data StartTime, and “6” is stored in element data EndTime. In addition, since “3” is stored in the classification information, the classification type of the video section of this sub-shot is “3”. In addition, “1” is stored in the element data NumberOfKeyframe. Thereby, it can be seen that one key frame is set in the video section of the sub-shot. Since element data KeyFrame stores “6”, frame number 6 is set as a key frame. Since “1” is stored in the element data NumberOfTimes, it can be seen that the key frame of frame number 6 has not yet been selected during video playback or editing.

  FIG. 9D shows sub-shot information when two key frames are set. The sub-shot section indicated by this sub-shot information corresponds to sub-shot 2 in FIG. When sub-shot 2 is set from frame number 7 to frame number 11, “7” is stored in element data StartTime, and “11” is stored in element data EndTime. Further, since “7” is stored in the classification information, the classification type of the video section of this sub-shot is “7”. In addition, “2” is stored in the element data NumberOfKeyframe. Thereby, it can be seen that two key frames are set in the video section of the sub-shot. Since element data Keyframe1 stores “8”, frame number 8 is set as a key frame. In addition, since “11” is stored in the element data Keyframe2, the frame number 11 is set as a key frame. Since “2” and “1” are stored in each element data NumberOfTimes of the element data Keyframes 1 and 2, the key frame of frame number 8 has been selected once during video playback or editing. Understand. It can also be seen that the key frame of frame number 11 has not yet been selected during video playback or editing.

  FIG. 10 is a diagram showing the structure of video segment classification information. The video segment classification information is data relating to the classification of the attached information in the video segment, and there are as many types as the type to be classified. This video section information stores the key frame extraction method in the classified video section, and is referred to in the key frame extraction process.

  FIG. 11 is a diagram showing a video section classification information list. In this video segment classification information list, the start pointer of the video segment classification information is stored for the number of video segment classifications (L). Therefore, when the video segment classification information 1 is referred to as the video segment classification information, a pointer indicating the video segment classification information 1 in the video segment classification information list is acquired, and the pointer destination is referred to, so that the video segment classification information 1 is obtained. Can be obtained.

  The classification information, which is element data of the video section classification information, represents the classification type of the video classified based on the attached information regarding the video section. The classification condition information that is element data includes condition information for determining whether or not the classification type is stored in the classification information based on at least one or a combination of the attached information. Yes.

  FIG. 12 shows the structure of the classification condition information. The number of stored classification determination process type information, which is element data of the classification condition information, represents the number of stored classification determination process type information. The classification determination processing type information is identification information (type of processing) for comparing the acquired auxiliary information with the value described in the comparison value when determining whether or not the video section is described in the classification information. ).

  The comparison value that is element data indicates a value to be compared with the attached information acquired by the classification determination processing type information. The determination type information stores the type of determination method used when comparing the attached information with the comparison value. For example, the value 1 indicates “=”, and indicates that the value acquired in the classification determination process is compared with whether the comparison value is “=”, that is, is equal. The value 2 indicates “! =”, And indicates that the value acquired in the classification determination process and the comparison value are “! =”, That is, whether they are not equal. The value 3 indicates “>” and indicates whether the value acquired in the classification determination process is “>”, that is, whether the acquired value is greater than the comparison value. The value 4 indicates “<”, and indicates whether the value acquired in the classification determination process and the comparison value are “<”, that is, whether the acquired value is smaller than the comparison value. The value 5 indicates “≦”, and indicates whether the value acquired in the classification determination process and the comparison value are “≦”, that is, whether the acquired value is equal to or less than the comparison value. The value 6 indicates “≧” and indicates whether the value acquired in the classification determination process is “≧”, that is, whether the acquired value is equal to or greater than the comparison value.

  Further, the number of key frame extraction type information stored in FIG. 10 describes the number of key frame extraction methods in the video section classified by the classification information. The key frame extraction type information describes a key frame extraction method in the video section classified by the classification information. Specifically, “0” indicates that a default key frame extraction method designated in advance by the operator or preset by the imaging apparatus is performed. “1” indicates that a key frame is extracted from the first frame of the sub-shot. “2” indicates that a key frame is extracted at the end frame of the sub-shot. “3” indicates that a key frame is extracted in an intermediate frame of the sub-shot. “4” indicates that a key frame is extracted at the zoom start (reduction → enlargement) frame of the sub-shot. “5” indicates that a key frame is extracted at the zoom end (reduction → enlargement) frame of the sub-shot. “6” indicates that a key frame is extracted at a zoom middle (reduction → enlargement) frame of the sub-shot. “7” indicates that a key frame is extracted at the zoom start (enlargement → reduction) frame of the sub-shot. “8” indicates that the key frame is extracted at the zoom end (enlargement → reduction) frame of the sub-shot. “9” indicates that a key frame is extracted in the middle zoom (enlargement → reduction) frame of the sub-shot. “10” indicates that a key frame is extracted at the tilt start frame of the sub-shot. “11” indicates that a key frame is extracted at the tilt end frame of the sub-shot. “12” indicates that a key frame is extracted at an intermediate frame of the sub-shot tilt. “13” indicates that a key frame is extracted at the pan start frame of the sub-shot. “14” indicates that a key frame is extracted at the pan end frame of the sub-shot. “15” indicates that a key frame is extracted in the pan intermediate frame of the sub-shot. The key frame extraction type information is stored in the number specified by the number of stored key frame extraction type information.

  FIG. 13 is a diagram showing video segment classification information. The video segment classification information “0” indicates a default key frame extraction method designated in advance by the operator or preset by the imaging apparatus. If none of the video segment classification information (except the default video segment classification information) exists in the video segment classification information list, the key frame extraction method described in the default video segment classification information is used. The Here, “1” is set in the key frame extraction type information, and the start frame of the video section becomes the key frame.

  The video section classification information “1” stores classification condition information using attached information in the pan operation. Specifically, referring to the shooting environment information and the operation information, whether the start frame is the start of panning, zooming, the number of times of panning during a shot, or the length of panning during a shot Is determined to be “large” or the like.

  FIG. 14 is a diagram showing the classification condition information in the video section classification information “1”. Since there are five classification determination processing type information of the video section classification information “1”, “5” is stored in the number of classification determination processing type information storages. In each of the five classification determination processing type information, classification determination processing type, comparison value, and determination type information are set. In the first classification determination processing type information, a classification determination processing type 1 that acquires the event type of the shooting environment information of the start frame and is compared with the comparison value (pan start) is stored.

  The second and third classification determination processing type information stores a classification determination processing type 2 for acquiring and comparing the event type of the operation information of the start frame. The fourth classification determination processing type information stores a classification determination processing type 3 for obtaining and comparing the number of pans in a shot. The fifth classification determination processing type information stores a classification determination processing type 4 for acquiring and comparing the length of a pan in a shot.

  As a result of these classification determination processes, the classification information is “1” when all the values indicated by the comparison values match. The key frame is set to the pan center frame corresponding to “3” of the key frame extraction method stored in the key frame extraction type information. The video data classified into the video section of the classification information 1 is presumed to have been taken with the intention of what kind of space the photographer is in by panning. For example, when shooting a landscape or the entire venue, such a shooting method is performed. Accordingly, in the video section classified as the classification information 1, it is considered that the video being panned is more meaningful than the end of panning, so the center frame of the panning operation section is set as a key frame.

  Next, video section classification information when the classification information is “2” is shown. In the classification information 2, video section classification condition information using auxiliary information in the zoom operation is set. As shown in FIG. 13, referring to the shooting environment information and the operation information, whether the start frame is the start of zooming, the zoom direction is the tele side, panning, or zooming during a shot Judgment is made based on whether it is once.

  As a result of these determination processes, the classification information is “2” when all the values indicated by the comparison values match. The key frame is set as the end frame which is the key frame extraction method “2” stored in the key frame extraction type information. The video data classified into the video section of this classification information 2 was taken with the intention of displaying the location of the subject to be noticed from the screen on which the zoom is reduced and enlarged and the entire subject is displayed. It is guessed. Therefore, it is considered that the end of zoom is more important than the start of zoom, and the zoom end frame of the video section of the zoom operation (reduction → enlargement) is set as a key frame. In this manner, the video section is classified by acquiring and determining the state of the attached information.

  FIG. 15 is a diagram showing user key frame extraction history information. FIG. 16 is a diagram showing the structure of the user key frame extraction history list. The user key frame extraction history information is created when the operator sets / changes the key frame with respect to the key frame by the key frame extraction method preset in the imaging apparatus, and the key frame extraction method by the operator Is registered information. When the classification information of the video section from which the key frame is extracted matches the classification information of the video section in which the key frame has been changed, the key frame extraction type information is obtained from the key frame extraction history information. A key frame is extracted by a key frame extraction method corresponding to the above.

  As shown in FIG. 16, the user key frame extraction history information list is a list for acquiring user key frame extraction history information. The user key frame extraction history information list includes key frame extraction history information for each user. The head pointer is stored. That is, when acquiring the user 2 key frame extraction history information, the user 2 key frame extraction history information pointer can be referred to by acquiring it from the user key frame extraction history information list.

  Each element data of the user key frame extraction log | history information of FIG. 15 is shown. The element data “user ID” identifies a user who holds key frame extraction history information. As the element data “user ID”, the user ID acquired by the fingerprint detection unit 31 is used. The element data “number of key frame extraction history information stored” indicates the number of key frame extraction history information held by the user. In the element data “link information of key frame extraction history information”, a head address to the key frame extraction history information is described.

  In the user 1 key frame extraction history information in FIG. 16, the identification information of the user 1 is stored in the user ID. There are N pieces of key frame extraction history information for the user 1, and the head pointer of the N pieces of key frame extraction history information indicates the storage location of each key frame extraction history information.

  FIG. 17 shows the structure of key frame extraction history information. FIG. 18 is a diagram showing the structure of the key frame extraction history information list. The key frame extraction history information is created when the operator changes the key frame with respect to the key frame extracted by the key frame extraction method of the video section classification information preset in the imaging device. The information for registering the key frame extraction method of the video section where the change is made.

  As shown in FIG. 16, the key frame extraction history information is linked to the user key frame extraction history information, and is obtained by referring to a pointer stored in the user key frame extraction history information. The acquired key frame extraction history information is linked to key frame extraction detailed information as necessary, as shown in FIG.

  Each element data of key frame extraction history information data is shown. The element data “classification information” describes the classification type (classification information) of the video section acquired from the sub-shot information in the video section whose key frame has been changed by the operator. In the element data “number of key frame extraction type information stored”, the number of key frame extraction methods in the video section classified in the classification information is described. The element data “key frame extraction type information” describes a method for extracting a new key frame obtained by the operator setting / changing the key frame. Specifically, when a negative serial number is stored, it means that a link to key frame extraction detailed information is performed, and a key frame extraction is performed using a value stored in the key frame extraction detailed information. . This key frame extraction type information is the same as the key frame extraction type information of the video section classification information, and there are the number specified by the number of stored key frame extraction type information.

  FIG. 19 is a diagram showing a structure of key frame extraction detailed information. FIG. 20 shows the structure of the key frame extraction detailed information list. The key frame extraction detailed information indicates a key frame extraction method of the video section classified by the classification information of the key frame extraction history information, and stores a value used for extracting the key frame. This key frame extraction detailed information is an extraction method that requires a value different from the key frame extraction method in the classification information (excluding the extraction method that refers to the key frame extraction detailed information), and the key frame is changed by the operator. Created when

  As shown in FIG. 20, the key frame extraction detailed information list stores the head pointer of the key frame extraction detailed information. The key frame extraction detailed information is referred to from key frame extraction history information (key frame extraction type information) as shown in FIG. For example, when “−2” is designated in the key frame extraction history information (key frame extraction type information), it indicates that the key frame extraction detailed information of the key frame extraction detailed information type information 2 is set.

  Each element data of key frame extraction detailed information data is shown. The element data “key frame extraction detailed information type information” describes the type of key frame extraction method (a method of extracting using a value stored in the key frame extraction detailed information). A serial number is stored in the key frame extraction detailed information type information. This serial number is a value obtained by multiplying the key frame extraction type information by “−1”. The element data “number of key frame extraction related values stored” indicates the number of stored values used for extracting key frames. The element data “key frame extraction related value” is a value used to extract a key frame.

  FIG. 21 is a diagram showing the structure of key frame extraction method determination information. FIG. 22 is a diagram showing the structure of the key frame extraction method determination information index list. The key frame extraction method determination information is information for specifying a method for extracting a newly set key frame when the operator changes the set key frame. The key frame extraction method determination information includes key frame extraction for specifying a key frame extraction method based on at least one of or a combination of frame additional information set as a new key frame and frame position information of the new key frame. The method condition information and key frame extraction type information for setting the key frame in the video section that matches all the key frame extraction condition information are held.

  As shown in FIG. 22, the key frame extraction method determination information exists for each classification information. The index information in the classification information is stored in the key frame extraction method determination information index list. The element data of the index information includes classification information (video section classification information), index storage count (number of key pointer extraction method determination information for the video section classified in the classification information), and index (key frame). There is a leading pointer of extraction method decision information).

  For example, when acquiring the key frame extraction method determination information 1 from the classification information, the index information 1 of the classification information is acquired by referring to the index information 1 of the key frame extraction method determination information index list. From this index information 1, the head pointer of the key frame extraction method determination information is acquired for the number of stored indexes (Q), and key frame extraction method determination information is acquired. That is, when the operator performs a key frame change operation, the classification information of the video section in which the key frame is changed is acquired from the sub-shot information, and the classification information acquired from the key frame extraction method determination information index list is included. Index information is obtained, and key frame extraction method determination information is obtained. Based on the acquired key frame extraction method determination information, a newly set key frame extraction method is specified. The key frame extraction method determination information includes default key frame extraction method determination information so that the key frame is always extracted. For example, the key frame extraction method determination information is specified by obtaining the ratio of the key frame position in the video section. The method to do is described.

  Each element data of key frame extraction method determination information is shown. In the element data “key frame extraction method condition information”, key frame extraction method condition information for specifying the position of a frame that newly becomes a key frame in the key frame change operation is described.

  FIG. 23 shows the structure of key frame extraction method condition information. The element data “number of key frame extraction method determination process type information storage” stores the number of determination processes indicated by the key frame extraction method determination process type information. The element data “key frame extraction method determination processing type information” indicates a processing type when comparing the frame number of a newly set key frame with the frame number acquired in the key frame extraction method determination processing.

  For example, when the key frame extraction method determination processing type information is “1”, this indicates “a process of comparing the start frame number of the sub-shot with the key frame number”. “2” indicates “a process of comparing the end frame number of the sub-shot and the key frame number”. “3” indicates “a process of comparing the zoom (reduction → enlargement) start frame number with the key frame number”. “4” indicates “a process of comparing the zoom (reduction → enlargement) end frame number with the key frame number”. “5” indicates “a process of comparing the zoom (reduction → enlargement) intermediate frame number with the key frame number”. As the intermediate frame number, the ratio of the intermediate value is acquired.

  Further, “6” indicates “a process of comparing the zoom (enlargement → reduction) start frame number with the key frame number”. “7” indicates “a process of comparing the zoom (enlargement → reduction) end frame number with the key frame number”. “8” indicates “a process of comparing the zoom (enlargement → reduction) intermediate frame number with the key frame number”. As the intermediate frame number, the ratio of the intermediate value is acquired.

  Further, “9” indicates “a process of comparing the tilt start frame number with the key frame number”. When it is “10”, it indicates that “this is a process of comparing the tilt end frame number with the key frame number”. “11” indicates that “this is a process of comparing the tilt intermediate frame number with the key frame number”. As the intermediate frame number, the ratio of the intermediate value is acquired.

  Further, “12” indicates “a process of comparing the pan start frame number and the key frame number”. “13” indicates “a process of comparing the pan end frame number with the key frame number”. “14” indicates “a process of comparing the pan intermediate frame number with the key frame number”. As the intermediate frame number, the ratio of the intermediate value is acquired.

  The element data “key frame extraction type information” describes a method for extracting a key frame in a video section that matches the key frame extraction method condition information. Since this key frame extraction type information has the same value as the key frame extraction type information of the key frame extraction history information, description thereof is omitted.

  The element data “determination type information” stores a new key frame number and a method for determining the frame number. For example, when the determination type information is “1”, it indicates that “=”, that is, whether or not the value acquired in the classification determination process is equal to the comparison value. “2” indicates that “! =”, That is, whether the value acquired in the classification determination process is not equal to the comparison value. When “3”, it indicates “>”, that is, whether or not the value acquired in the classification determination process is larger than the comparison value. “4” indicates that “<”, that is, whether or not the value acquired in the classification determination process is smaller than the comparison value. “5” indicates that “≦”, that is, whether or not the value acquired in the classification determination process is equal to or less than the comparison value. “6” indicates that “≧”, that is, whether or not the value acquired in the classification determination process is greater than or equal to the comparison value.

  FIG. 24 is a diagram showing a specific example of key frame extraction method determination information. 14 shows key frame extraction method determination information when the video section classification information shown in FIG. 13 is assumed. FIG. 24 shows key frame extraction method determination information of the video section classification information (video section classification information = 1) in FIG.

  In the key frame extraction method determination information of the classification information 1, there are S index information, and S head pointers of the key frame extraction method determination information are stored. With reference to the key frame extraction method determination information in the order in which the pointers of the key frame extraction method determination information are acquired, the decision processing type information described in the key frame extraction condition information of the key frame extraction method determination information Get the frame number. Then, the determination type specified by the determination type information is compared with the frame number of the newly set key frame.

  When all the judgment processes stored in the key frame extraction condition information match, the key frame extraction type information described in the key frame extraction method determination information is acquired. For example, the classification information 1 of the moving image section shown in FIG. 24 indicates that the panning is performed once and the panning section is a long moving image section, and key frame extraction condition information using attached information related to the classification information 1 Is stored in the key frame extraction method determination information.

  For example, the index (pointer) 1 indicates determination processing type information as to whether or not a new key frame is a pan start frame, acquires a frame number set as a key frame, and indicates the frame number by that frame number. It is compared whether or not the event type of the shooting environment information is pan start. When the event type matches the pan start, it is considered that the key frame is set at the pan start position. As the key frame extraction method described in the key frame extraction method determination information, the key frame extraction type information “13” is set. Determine the pan start frame.

  For example, the index (pointer) 3 describes determination processing type information for determining whether or not the new key frame is an intermediate frame of pan. The event type of the shooting environment information of the frame number set as the key frame is referred to, and it is compared whether or not the event type is equal to the comparison value (pan continuation). If panning is continued, the position information of which frame in the pan has become a key frame is calculated according to Equation (1).

Position information = (frame number to be a key frame-pan start frame number) / (pan end frame number-pan start frame number) (1)
This calculated value is stored in the key frame extraction detailed information (see FIG. 19). Type information of key frame extraction detailed information in which a key frame extraction method for setting a key frame in the middle of a pan frame is stored is acquired from the key frame extraction type information. In FIG. 24, the type information of the key frame extraction detailed information is “3” = (− 3) × (−1). The value calculated by the formula (1) is stored in the key frame extraction detailed information whose type information of the key frame extraction detailed information is “3”. For example, when the acquired (calculated) value is “0.1”, “0.1” is stored in the related value 1 of the type information 3 of the key frame extraction detailed information as shown in FIG.

  The key frame setting processing operation of the imaging apparatus having the above configuration will be described. 25 and 26 are flowcharts showing a key frame setting process procedure in the imaging apparatus. This processing program is stored in a storage medium in the imaging apparatus, and is executed after the power is turned on by a CPU in the imaging apparatus (specifically, a CPU in the system control unit 23 and a CPU in the video management unit 17). Is done.

  First, initialization processing is performed (step S1), and input from the operation input unit 24 is awaited (step S2). When there is any input from the operation input unit 24, the content of the input operation is determined. That is, it is determined whether or not the input operation is to start shooting by pressing the REC button 27 (step S3). If the input operation is pressing of the REC button 27, a photographing process is performed (step S6). At this time, the above-described operation information and shooting environment information are output to the recording unit 29 and are temporarily stored in the shooting environment information list in the RAM 43 in time series order.

  It is determined whether or not the shooting has been completed (step S7). If the shooting has not been completed, the process of step S6 is repeated until the shooting is completed. Here, the end of shooting is when the REC button 27 is pressed again, when the recording capacity of the recording medium 30 is full, or when it is likely that power cannot be supplied, or when any of the conditions is satisfied. Done.

  When shooting is completed in step S7, video division processing is performed to divide the shot video data into sub-shot video sections (step S8). A key frame setting process for setting a frame image representing the divided video section is performed (step S9). Details of the key frame setting process will be described later. Thereafter, the process returns to step S2 to wait for the next input operation.

  On the other hand, if the input operation is not pressing the REC button 27 in step S3, it is determined whether or not an instruction to display a key frame list has been issued (step S4). The key frame list display is instructed when a button or the like of the operation input unit (operation panel) 24 for which the key frame list display function is set in advance is pressed, or when the key frame list display is specified from the menu button 32. And so on. When an instruction to display a key frame list is given, key frame list display processing is performed (step S10). Thereafter, the process returns to step S2 to wait for the next input operation. In this key frame list display process, sub shot information is acquired from all pointers registered in the sub shot information list (see FIG. 9), and the key frame and key frame are obtained from the element data KeyFrame of the sub shot information (see FIG. 8). And the corresponding image data is acquired and displayed on the display unit 18.

  FIG. 49 is a diagram showing an operation screen for displaying a list of key frames for each moving image section. In this operation screen, a list of key frames for each moving image (video) section obtained by dividing the captured video is displayed. Sub-shot information is also created for moving image sections A to P. As will be described later, when the key frame changing operation in the moving image section B is performed, the key frame changing process in the moving image section B is started.

  On the other hand, if the key frame list display instruction has not been issued in step S4, it is determined whether or not a key frame change instruction has been issued (step S5). Whether or not there is an instruction to change the key frame can be determined by selecting the key frame change by the menu button 32 of the operation input unit (operation panel) 24 in which the function of changing the key frame is set in advance. When a key frame change instruction is issued, a key frame change process is performed (step S11). Details of the key frame changing process will be described later. Thereafter, the process returns to step S2 to wait for the next input operation.

  On the other hand, when the key frame change instruction is not performed in step S5, it is determined whether or not the user recognition instruction is performed (step S12). The presence / absence of this user recognition instruction can be determined by pressing a finger against the sensor unit of the fingerprint detection unit 31. When a user recognition instruction is given, user identification processing is performed (step S13). Thereafter, the process returns to step S2 to wait for the next input operation. In this user identification process, a fingerprint recognition technique disclosed in Patent Document 2 is used, and a shadow generated by a finger pressed against the fingerprint detection unit 31 is detected as a fingerprint by the sensor unit, and this fingerprint is stored in advance. It is compared with the fingerprint data from the user recognition data list (a list in which the user ID and the fingerprint data are a pair). If the fingerprint data matches, a user ID that identifies the operator is acquired. On the other hand, if the fingerprint data does not match, the fingerprint data and the user ID are registered in the user recognition data list as a new user. The user ID of the operator currently used is held in the RAM 43.

  On the other hand, if there is no user recognition instruction in step S12, other processes are performed (step S14), and the process returns to step S2. When an interrupt is generated by the interrupt signal 46, this process is interrupted and necessary interrupt processing is performed.

  FIG. 27 is a flowchart showing the video division processing procedure in step S8. In the present embodiment, the video data captured by the imaging apparatus is divided into shot units (units divided based on the start / end of the recording operation). Furthermore, video data in a shot is divided into sub-shot units based on information such as operation information (operation information such as zoom) and shooting environment information (information such as pan and tilt).

  First, a pointer is set at the head of the stored shooting environment information list (step S21). It is determined whether or not the event type information of the shooting environment information indicated by the pointer is the start of panning (step S22). If panning has started, it is determined whether or not the zoom section is in progress (step S23). This determination is performed in order to divide into sub-shots by paying attention to the operation that occurs first when a plurality of operations are performed within the same time, such as starting zooming during panning. Further, it is performed by referring to the element data before and after the shooting environment information data list from the currently processed pointer in the shooting environment information list.

  If the zoom is not performed in step S23, the movement amount on the screen is converted (step S24). Even when the rotation angle is the same, when the image pickup apparatus is zooming, the movement on the screen is intense. Therefore, the conversion is performed by multiplying the rotation angle by a coefficient predetermined for each zoom position. Then, the total movement amount of the entire pan from the pointer currently being processed in the shooting environment information list to the end of panning is calculated.

  It is determined whether or not the amount of movement on the screen exceeds a threshold value (step S25). If it exceeds the threshold, it is determined whether or not the time difference from the end of the previous pan operation exceeds the threshold (step S26). This is because operations performed frequently are excluded from sub-shot targets. If the time difference from the end of the previous pan operation exceeds the threshold, the pan start point is set as a division point (step S27).

  When the division point of the video is determined, the end point, the number of key frames, and the sub-shot information in the sub-shot information of the video located before the division point are newly created and added to the sub-shot information. On the other hand, the start point is stored in the new sub-shot information. Like the operation information and the shooting environment information, these sub-shot information is output to the recording unit 29 and temporarily stored in the sub-shot information list in the RAM 43 in time series order.

  Thereafter, the process proceeds to step S32. Further, if the zoom section is in step S23, the amount of movement on the screen does not exceed the threshold value in step S25, or if the time difference from the end of the previous pan operation does not exceed the threshold value in step S26, division is performed. The process proceeds to step S32 without determining a point.

  On the other hand, as a result of the determination in step S22, if the event type information in the shooting environment information is not pan start, it is determined whether or not the event type information in the shooting environment information is zoom start (step S28). If the zoom has been started, it is determined whether or not it is during the pan period (step S29). The determination as to whether or not it is during the pan period during zooming is performed for the same reason as the determination as to whether or not it is during the zoom period during panning.

  If it is not in the pan section in step S29, it is determined whether or not the zoom start is intended by the photographer (step S30). In this process, it is estimated that the photographer intended one zoom operation even when a plurality of zoom operations were performed. For example, it is an operation characteristic for beginners, and the operation of the wide button 25 or the tele button 26 cannot be stopped at the intended zoom magnification, and it is presumed that an operation of returning to the reverse direction was performed in order to correct it. . This can be estimated from the fact that the time from the end of the zoom operation to the start of the return operation is within a predetermined range, the zoom speed is high, and the zoom direction is reversed. . In addition, the beginner's operation has a feature that the zooming is performed intermittently rather than smoothly. Such an intermittent operation can be estimated from the time difference and the direction of zooming, and is considered to have been performed in continuous zoom intervals.

  As a result of the determination in step S30, when it is estimated that the zoom start is intended by the photographer, the zoom start point is set as a division point (step S31). As described above, when the video division point is determined, the end point, the number of key frames, and the sub-shot information in the sub-shot information of the video located before the division point are newly created and added to the sub-shot information list To do. Also, the start point is registered in the new sub-shot information. Like the operation information and the shooting environment information, these sub-shot information is output to the recording unit 29 and temporarily stored in the sub-shot information list in the RAM 43 in time series order. When the process at the current pointer in the shooting environment information list is completed, the process proceeds to step S32. If the zoom is not started in step S28, the pan is in step S29, or if the zoom start is not intended by the photographer in step S30, the process proceeds to step S32 without determining a division point. . Then, it is determined whether or not there is next input operation information (step S32). If there is next input operation information, the process returns to step S22. If there is no next input operation information, this process ends.

  FIG. 28 is a flowchart showing the key frame setting processing procedure in step S9. First, the user ID of the operator who sets the key frame is acquired from the RAM 43 (step S41). When the user ID is not set, the user ID of the common user specified in advance in the imaging apparatus is acquired.

  An initialization process for performing the key frame setting process is performed (step S42). For example, the pointer is moved to the head of the sub-shot information list for extracting the key frame. Then, it is determined whether or not the key frame setting for each video section shown in the sub-shot information list is finished (step S43). If the key frame setting is finished, this process is finished. On the other hand, if the key frame setting is not completed, the process proceeds to step S44.

  By referring to the sub-shot information list and the sub-shot information, the classification information of the video section for setting the key frame is acquired (step S44). This classification information acquisition process is performed by referring to the attached information (such as imaging environment information and operation information) related to the video section of the sub-shot and based on the classification condition information stored in the video section classification information. Details of the classification information acquisition process will be described later. The key frame extraction history information of the operator is acquired from the user ID acquired in step S41, and the key frame extraction history information of the classification information acquired in step S44 is acquired from the user ID (step S45). Based on the classification information acquired in step S44, key frame extraction type information is searched and acquired (step S46). The key frame is extracted using the key frame extraction type information acquired in step S46 (step S47). The frame number of the key frame extracted in step S47 is stored in the element data Keyframe of the sub-shot information (step S48). At this time, information of element data NumberOfKeyframe is added according to the extracted key frame. When there are two key frame extraction type information in the video section classification information, two key frames are acquired, and when the acquired key frame number is not the same, the information of the element data NumberOfKeyframe of the sub-shot information is “+2”. Is added. The acquired two frame numbers are registered in the element data Keyframe.

  FIG. 29 and FIG. 30 are flowcharts showing the classification information acquisition processing procedure in step S44. First, the pointer I is set at the head of the video section classification information list (step S51). For example, the pointer I is moved to the head pointer of the video section classification information 1 in the video section classification information list shown in FIG. Then, it is determined whether or not the pointer I has moved to the end of the video section classification information list (step S52). That is, it is determined whether or not all the video segment classification information has been referred to. If all the video section classification information is referred to in step S52, it means that there is no applicable classification information, so default classification information is set (step S67), and this process is terminated.

  On the other hand, if all the video segment classification information is not referenced in step S52, the video segment classification information indicated by the pointer I is acquired (step S53). The video section classification information 1 head pointer shown in FIG. 11 is acquired, and the video section classification information 1 is acquired. The number of classification determination processing type information stored in the classification condition information of the video section classification information is acquired (step S54). In the classification condition information (see FIG. 14) in the classification information 1 (see FIG. 13), “5” is stored in the number of stored classification determination processing type information, so “5” is stored as the number of stored classification determination processing type information. To be acquired.

  Thereafter, in steps S55 to S57, the classification determination process stored in the classification condition information is performed, and it is determined whether or not all the classification determination processes match. If all the classification determination processes match, the classification information having the classification condition information is classified. First, it is determined whether or not all the classification determination processes of the classification determination process type information stored in the classification condition information have been completed (step S55). When all the classification determination processes are completed, the process proceeds to step S65. On the other hand, if all the classification determination processes are not completed in step S55, the classification determination process for the next classification determination process type information is performed.

  It is determined whether or not the value of the classification determination processing type information is “1” to “L” (L is the number of definitions of the classification determination processing type information, and the value is 5 in the classification information 1) (step S56). If it is not “1” to “L”, the pointer I of the video segment classification information list is added (step S56A), the process returns to step S52, and the process proceeds to determination of the next video segment classification information.

  On the other hand, if the value of the classification determination processing type information is “1” to “L” in step S56, the classification determination processing for each classification determination processing type information is performed in steps S59 to S64. That is, it is determined whether or not the classification determination processing type information is “1” (step S59). When the classification determination process type information is “1”, the determination process of the classification determination process type information 1 is performed (step S60), and the process proceeds to step S57. On the other hand, if the classification determination processing type information is not “1”, it is determined whether or not the classification determination processing type information is “2” (step S61). When the classification determination process type information is “2”, the determination process of the classification determination process type information 2 is performed (step S62), and the process proceeds to step S57. On the other hand, when the classification determination processing type information is not “2”, the same processing as the classification determination processing type information 1 and 2 is performed until the classification determination processing type information is “L−1”. If the classification determination process type information is not “L−1”, it is determined whether or not the classification determination process type information is “L” (step S63). If the classification determination process type information is “L”, the determination process of the classification determination process type information L is performed (step S64), and the process proceeds to step S57. Details of this classification determination process will be described later.

  As described above, after performing the classification determination processing for each classification determination processing type information in step S59 to step S64, it is determined whether or not the result of each classification determination processing is true (considerable to match) (step S57). ). If the return value of each classification determination process is not true, the pointer I is added (step S58), the process returns to step S52, and the process proceeds to the next video segment classification information. On the other hand, if it is determined in step S57 that the two match, the process returns to step S55 to move to the next classification determination processing type information.

  When all the classification determination processes of the classification determination process type information are completed in step S55, it is determined whether or not all the classification determination processes of the classification determination process type information match and become true (step S65). If there is video segment classification information for which the result of classification determination processing for classification condition information is all true, classification information is acquired (step S66), and this process is terminated. On the other hand, if all the classification determination processes of the classification condition information are not true, the process returns to step S52. If all the video segment classification information has been referred to in step S52, the default classification information is set in step S67 as described above, and the process is terminated.

  For example, in the first classification determination process shown in FIG. 14, the classification determination process type information 1 is set, and the event type information of the shooting environment information of the start frame is acquired. Then, it is shown that the comparison between the acquired event type information and the comparison value (0: start of panning) is performed by the determination method type 1 (=). That is, a comparison is made as to whether they are equal.

  FIG. 31 is a flowchart showing the determination processing procedure of the classification determination processing type information 1 in step S60. First, a comparison value (0: pan start) and determination method type information (1: =) are acquired (step S71). The start frame number is acquired from the element data StartTime of the sub-shot information, the shooting environment information that matches the frame number is searched, and the event type information is obtained (step S72).

  The determination method type acquired in step S71 is determined (step S73). That is, when the determination type is “1: =”, a comparison is made as to whether or not they are equal (step S75), and the process proceeds to step S74. If the determination type is “2:! =”, A comparison is made as to whether they are not equal (step S76), and the process proceeds to step S74. When the determination type is “3:>”, it is compared whether or not the event type information is larger than the comparison value (step S77), and the process proceeds to step S74. When the determination type is “4: <”, it is compared whether or not the event type information is smaller than the comparison value (step S78), and the process proceeds to step S74. After such processing is performed until the determination type is “T−1” (where “T” is a value of 5 or more), when the determination type is “T”, the determination type corresponding to “T” Then, the event type information is compared with the comparison value (step S78), and the process proceeds to step S74. On the other hand, if it is determined in step S73 that the determination type is not one of the above values and other (etc), the process proceeds to step S82.

  Here, since the determination method type is “1”, that is, “=”, in step S75, the event type information acquired in step S72 is compared with the comparison value (0: pan start) acquired in step S71. It is determined whether this comparison is true and true (step S74). As a result of the comparison, if it is established, “1” is set as a return value (step S81), and this process is terminated. On the other hand, if the result of comparison is not established, “0” is set as a return value (step S82), and this process is terminated.

  Next, the second classification determination process in FIG. 14 is shown. FIG. 32 is a flowchart showing the determination processing procedure of the classification determination processing type information 2 in step S60. The same step numbers are assigned to the same step processes as in FIG. First, classification determination processing type information 2 is set, and event type information of operation information of the start frame is acquired. Then, the comparison between the acquired event type information and the comparison value (1: zoom continuation) is performed with the determination method type 2, that is, “! =”.

  Specifically, in step S71 described above, a comparison value (1: zoom continuation) and determination method type information (2:! =) Are acquired. The start frame number is acquired from the element data StartTime of the sub-shot information, the operation information that matches the frame number is searched, and the event type information is obtained (step S72A). Since the determination method type acquired in step S71 is “2”, that is, “! =”, The process branches to step S76 in step S73. In step S76, it is compared whether or not the event type information acquired in step S72 and the comparison value (1: zoom continuation) acquired in step S71 are not equal (“! =”). Then, the value obtained as a result of the comparison in steps S74, S81, and S82 is set as a return value, and the determination process of this classification determination process type 2 is terminated. Similarly, the third classification determination process in FIG. 14 is shown. This classification determination processing procedure is the same as the second classification determination processing except that the comparison values are different, and thus the description thereof is omitted.

  In the fourth classification determination process of FIG. 14, the classification determination process type information 3 is set, and the number of times of panning in the sub-shot is acquired. In this classification determination process, the acquired event type is compared with the comparison value (1: pan continuation) with determination method type 1, that is, “=”. FIG. 33 is a flowchart showing the determination processing procedure of the classification determination processing type information 3 in step S60. The same step numbers are assigned to the same step processes as in FIG. In step S71, a comparison value (1: pan continuation) and determination method type information (1: =) are acquired. Then, the start frame number is acquired from the element data StartTime of the sub-shot information, the end frame number is acquired from the element data EndTime, the shooting environment information that matches the frame number in the frame number is acquired, and the event type is “ The number of “bread start” is obtained (step S72B). In step S73, since the determination method type acquired in step S71 is “1”, that is, “=”, in step S75, the number of pan starts acquired in step S72B and the comparison value acquired in step S71 (1: pan). Compare (Continuation) is equal ("="). The value of the result of comparison in steps S74, S81, and S82 is set as a return value, and the determination processing of this classification determination processing type 4 ends.

  The fifth classification determination process in FIG. 14 indicates that the classification determination process type information 4 is set and the bread length is acquired. In this classification determination process, a comparison between the acquired event type and a comparison value (a preset threshold value indicating that the bread length is large) is performed using determination method type 3, that is, “>”. FIG. 34 is a flowchart showing the determination processing procedure of the classification determination processing type information 4 in step S60. The same step numbers are assigned to the same step processes as in FIG. In step S71, a comparison value (threshold value) and determination method type information (3:>) are acquired. The start frame number is acquired from the element data StartTime of the sub-shot information, the end frame number is acquired from the element data EndTime, and the time from the start of panning to the end of panning is acquired (step S72C).

  In step S73, since the determination method type acquired in step S71 is “3”, that is, “>”, the process branches to step S77. In step S77, the time from the start of pan to the end of pan acquired in step S72C is stepped. It is compared whether it is longer (">") than the comparison value (threshold value) acquired in S71. Then, the value obtained as a result of the comparison in steps S74, S81, and S82 is set as a return value, and the determination process of this classification determination process type 4 ends.

  FIG. 35 is a flowchart showing the key frame extraction processing procedure in step S47. First, it is determined whether or not the key frame extraction type information is a negative value (step S91). If it is a negative value, key frame extraction based on the key frame extraction detailed information is performed (step S92), and this process ends. Details of the key frame extraction processing based on the key frame extraction detailed information will be described later.

  On the other hand, if the key frame extraction type information is not a negative value in step S91, the key frame extraction processing corresponding to the key frame extraction type information is performed by the processing from step S93 to step S98. Specifically, it is determined whether or not the key frame extraction type information is “1” (step S93). When the key frame extraction type information is “1”, the first frame of the sub-shot video section is acquired with the key frame extraction type 1 (step S94). Then, this process is complete | finished.

  On the other hand, if the key frame extraction type information is not “1” in step S93, it is determined whether or not the key frame extraction type information is “2” (step S95). When the key frame extraction type information is “2”, the end frame of the sub-shot video section is acquired with the key frame extraction type 2 (step S96). Then, this process is complete | finished. Similarly, when the key frame extraction type information is not “2” in step S95, the same processing is performed until the key frame extraction type information is “V−1” (“V” is a value of 3 or more in the key frame extraction type information). When the key frame extraction type information is not “V−1”, it is determined whether or not the key frame extraction type information is “V” (step S97). When the key frame extraction type information is “V”, a key frame extraction process corresponding to the key frame extraction type information “V” is performed (step S98). Then, this process is complete | finished. On the other hand, if the key frame extraction type information is not “V”, the present process is terminated.

  Next, a key frame extraction process for each key frame extraction type information is shown. FIG. 36 is a flowchart showing a key frame extraction processing procedure of key frame extraction type information 1 for setting a start frame as a key frame. First, the start frame number is acquired from the element data StartTime of the sub-shot information (step S101). The value 1 is added to the element data NumberOfKeyframe of the sub-shot information (step S102). The frame number acquired in step S101 is stored in the element data Keyframe of the sub-shot information (step S103). Then, this process is complete | finished. Thereby, the start frame is set as a key frame.

  FIG. 37 is a flowchart showing the key frame extraction processing procedure of the key frame extraction type information 2 for setting the end frame as a key frame. First, the end frame number is acquired from the element data EndTime of the sub-shot information (step S111). The element data NumberOfKeyframe of the sub-shot information is incremented by 1 (step S112). The frame number acquired in step S111 is stored in the element data Keyframe of the sub-shot information (step S113). Then, this process is complete | finished. Thereby, the end frame is set as a key frame.

  FIG. 38 is a flowchart showing a key frame extraction processing procedure of key frame extraction type information 3 for setting an intermediate frame in panning as a key frame. First, the start frame number is acquired from the element data StartTime of the sub-shot information (step S121). The end frame number is acquired from the element data EndTime of the sub-shot information (step S122). By referring to the shooting environment information within the frame number range acquired in step S121 and step S122, the pan start frame number and pan end frame number are acquired, and the pan intermediate frame number is obtained according to equation (2) (step S123). ).

(Pan end frame number−pan start frame number) / 2 + pan start frame number (2)
The element data NumberOfKeyframe of the sub-shot information is incremented by 1 (step S124). The frame number acquired in step S123 is stored in the element data Keyframe of the sub-shot information (step S125). Then, this process is complete | finished. Thereby, the intermediate frame during panning is set as a key frame.

  FIG. 39 is a flowchart showing the key frame extraction processing procedure using the key frame extraction detailed information in step S92. First, the key frame extraction detailed information is searched using the key frame extraction type information acquired in step S46 (step S131). When referring to the key frame extraction detailed information type information (see FIG. 18), since the key frame extraction information type information is multiplied by “−1”, “−1” is added to the value of the key frame extraction type information. As a result, key frame extraction detailed information type information is acquired. By performing a search using the acquired key frame extraction detailed information type information, the key frame extraction detailed information is acquired. All the key frame extraction related values stored in the acquired key frame extraction detailed information are sequentially acquired (step S132).

  Then, by the processing from step S133 to step S138, the related value acquired for each key frame extraction detailed information type information is used to perform the key frame extraction processing. Specifically, it is determined whether or not the key frame extraction detailed information type information is “1” (step S133). In the case of “1”, the key frame extraction process of the key frame extraction detailed information type information 1 is performed (step S134), and this process ends. On the other hand, if it is not “1”, it is determined whether or not the key frame extraction detailed information type information is “2” (step S135). In the case of “2”, the key frame extraction processing of the key frame extraction detailed information type information 2 is performed (step S136), and this processing is terminated. On the other hand, if it is not “2”, the same processing is performed until the key frame extraction detailed information type information becomes “N−1” (where N is “3” or more of the key frame extraction detailed information type information). When the key frame extraction detailed information type information is not “N−1”, it is determined whether or not the key frame extraction detailed information type information is “N” (step S137). In the case of “N”, the key frame extraction process of the key frame extraction detailed information type information N is performed (step S138), and this process ends. On the other hand, if it is not “N”, this processing is terminated as it is.

  For example, when the key frame extraction detailed information type information is “3”, that is, when the key frame exists during panning, the position of the key frame extraction related value during panning acquired in step S132 as key frame setting processing Perform processing to set keyframes in

  FIG. 40 is a flowchart showing the key frame extraction processing procedure of the key frame extraction detailed information type information 3. In the key frame extraction process of the key frame extraction detailed information type information 3, the key frame is extracted according to the key frame extraction related value of the key frame extraction detailed information 3 (value indicating the ratio of the sub-shot information in the pan start frame). The

  First, the start frame number and the end frame number of the sub-shot information are acquired, and the shooting environment information is searched from the frame numbers within the range, and the event type is pan start, and the pan start frame number is searched. Is acquired (step S141). Similarly, referring to the shooting environment information from the frame numbers within the range of the start frame number and end frame number of the sub-shot information, search for the event type of pan end, and acquire the pan end frame number (Step S142). After that, based on the key frame extraction related value (0.1) acquired in step S132, the key frame number is calculated and acquired according to equation (3) (step S143).

(Pan end frame number−pan start frame number) × 0.1 + pan start frame number (3)
The element data NumberOfKeyframe of the sub-shot information is incremented by 1 (step S144), and the key frame number acquired in step S143 is set in the element data Keyframe of the sub-shot information (step S145). Then, this process is complete | finished.

  FIG. 41 is a flowchart showing the key frame change processing procedure in step S11. First, a moving image (video) section whose key frame is to be changed is acquired (step S151). This moving image section is changed, for example, when the operator selects a key frame from the list of key frames displayed on the display unit 18 by the key frame list display process (see step S10).

  FIG. 50 is a diagram illustrating an operation screen when performing key frame change processing in a moving image section. In the upper part of this operation screen, the point display of the key frame currently being changed is performed. Here, the point display is performed by displaying the thumbnail image of the moving image section B with a thick frame. On the other hand, in the lower part of the operation screen, the frame image of the moving image section B is displayed. Further, the frame image which is the current key frame and the currently selected frame image are displayed as points. Here, the frame image B1 is a frame image that is currently a key frame, and is displayed in a dotted frame. The frame image B3 is a currently selected frame image. When setting is made on this operation screen, the key frame is set to the frame image B3.

  In this way, the newly set key frame is selected (step S152). The key frame selection process is not particularly limited. For example, when the moving image is reproduced using the element data StartTime and the element data EndTime acquired in step S151 and an image to be used as a key frame is displayed, the operator presses the Stop button of the operation input unit 24. The key frame may be selected by pressing the key. Further, all the frames between the element data StartTime and the element data EndTime may be displayed simultaneously or hierarchically, and the operator may select a key frame from a list of the frames. The frame number that becomes the selected key frame is stored in the element data KeyFrame of the sub-shot information (step S153).

  It is determined whether or not the key frame extraction method is registered in the key frame extraction history information (step S154). In this determination process, when changing the key frame, information on whether or not to generate key frame extraction history information is previously described in an environment file or the like, and the information may be acquired and determined. When changing the frame, the operator may determine whether or not to create key frame extraction history information, and is not particularly limited.

  When registering in the key frame extraction history information in step S154, the key frame extraction history information is registered (step S155), and the process proceeds to step S156. On the other hand, if not registered in the key frame extraction history information in step S154, the process proceeds to step S156 as it is. It is determined whether or not to change the key frame in the similar moving image section (step S156). In this determination process, as in step S154, information on whether or not to perform a key frame change process in a similar moving image section at the time of key frame change may be obtained from a stored environment file or the like. Alternatively, when the key frame is changed, the determination may be made by confirming whether or not the operator changes, and there is no particular limitation.

  In step S156, when changing the key frame of the similar moving picture section, the key frame changing process of the similar moving picture section is performed (step S157), and the key frame changing process is ended. On the other hand, when the key frame change of the similar moving image section is not performed in step S156, the key frame change process is terminated as it is.

  FIG. 42 is a flowchart showing the key frame extraction history information registration processing procedure in step S155. First, the classification information in the video section in which the key frame has been changed is acquired (step S161). In this classification information acquisition process, the classification information of the sub-shot information in which the key frame is changed or the classification information in which the key frame stored in the RAM 43 is changed is acquired. A key frame extraction method for a newly set key frame is acquired (step S162). When acquiring the newly set key frame extraction method, if the related value of the extraction method is stored in the key frame extraction detailed information, the related value is also acquired. When the related value is acquired in step S162, the related value is stored in the key frame extraction detailed information (step S163).

  The user ID of the operator who sets the key frame extraction history information is acquired from the RAM 43 (step S164). From the user key frame extraction history information list, the head pointer of the user key frame extraction history information indicated by the user ID is acquired, and the key frame extraction history indicated by the key frame extraction history information link information from the user key frame extraction history information The key frame extraction history information of the classification information acquired in step S161 is searched from the information (step S165).

  It is determined whether there is key frame extraction history information that matches the classification information (step S166). If there is key frame extraction history information that matches the classification information, the key frame extraction type information of the key frame extraction history information acquired in step S165 is changed to the key frame extraction type information acquired in step S162 (step S167). . Then, this process is complete | finished. On the other hand, if there is no key frame extraction history information that matches the classification information in step S166, the classification information acquired in step S161 is registered in the classification information of the new key frame extraction history information, and the key acquired in step S162. The frame extraction type information is registered in the key frame extraction type information, and “1” is added to the number of stored key frame extraction type information to create key frame extraction history information (step S168). At this time, the created head address is registered in the user's key frame extraction history information list acquired in step S164. Then, this process is complete | finished.

  FIG. 43 is a flowchart showing a newly set key frame extraction type information acquisition processing procedure in step S162. First, the frame number set as a new key frame is acquired (step S171). In this frame number acquisition process, the value acquired in the newly set key frame acquisition process in step S152 is acquired. The classification information of the video section in which the key frame has been changed is acquired (step S172). The key frame extraction method determination information of the classification information acquired in step S171 is searched (step S173). Specifically, index information in which the classification information acquired in step S171 matches the classification information of the index information is acquired from the key frame extraction method determination information index list. A pointer of key frame extraction method determination information stored in the acquired index information is acquired.

  It is determined whether or not there is key frame extraction method determination information that matches the classification information acquired in step S171 (step S174). If there is no key frame extraction method determination information that matches the classification information, default key frame extraction method determination information is acquired (step S175), and the process proceeds to step S179.

  On the other hand, if there is key frame extraction method determination information that matches the classification information in step S174, the pointer of the key frame extraction method determination information is acquired in the order stored in the index information, and the key frame extraction method determination is determined from the acquired pointer. Information is acquired (step S176). Based on the judgment processing stored in the key frame extraction method condition information stored in the key frame extraction method determination information, a new key frame is extracted by the key frame extraction method described in the key frame extraction method determination information. A condition determination process is performed to determine whether or not the image has been updated (step S177). In this determination processing, the key frame stored in the key frame extraction method determination information is obtained by acquiring the attached information described in the key frame extraction method condition information and comparing it with the frame number set in the key frame. It is determined whether or not the extraction type information is extracted. Also, a value (related value) used at the time of extraction is acquired from the key frame extraction method determination processing type information.

  It is determined whether or not all conditions are met in the condition determination process existing in the key frame extraction method condition information (step S178). If all the conditions are not met, the process returns to step S176, the next key frame extraction method determination information stored in the index information acquired in step S173 is acquired, and the same process is repeated. On the other hand, if all the conditions are matched in step S178, key frame extraction type information is acquired (step S179).

  It is determined whether or not the key frame extraction type information of the acquired key frame extraction method determination information refers to the key frame extraction detailed information (step S180). That is, when referring to the key frame extraction detailed information, the value becomes a negative value, so it is determined whether or not it is a negative value. When the key frame extraction detailed information is referred to, the head pointer of the key frame extraction detailed information and the related value acquired in step S177 are stored in the temporary buffer (step S181). Then, this process is complete | finished.

  FIG. 44 is a flowchart showing the condition information determination processing procedure in step S177. First, an initialization process in the condition information determination process is performed (step S191). In this initialization process, for example, the number of stored key frame extraction method determination process type information stored in the key frame extraction method determination information acquired in step S176 is acquired, and the determination process counter is set to “0”.

  In steps S192 to S201, the determination process stored in the key frame extraction method determination information is performed. If it is determined as “true” as a result of all the determination processes, it is determined that all the determination processes match in step S203, and the present process ends. On the other hand, if it is determined as “false” in any determination process, it is determined that there is no match in step S202, and this process is terminated.

  Specifically, it is determined whether or not the value of the determination process counter is the same as the number of stored key frame extraction method determination process type information acquired in step S191 (step S192). If they are the same, as described above, it is determined that they match in all the determination processes, “true” is stored as the determination result (step S203), and this process ends.

  On the other hand, in step S192, if the value of the determination process counter is not the same as the number of stored key frame extraction method determination process type information acquired in step S191, and the determination process has not ended yet, the key frame indicated by the determination process counter Extraction method determination processing type information is acquired (step S193). It is determined whether or not the acquired key frame extraction method determination processing type information is within a range of “1” to “R” (step S194). Here, “R” is the number of key frame extraction method determination processing type information set in advance. If the key frame extraction method determination processing type information is not within the range of “1” to “R”, but other than that, it is considered that the determination processing does not match, and the determination result is “false” (step S202). This process ends.

  On the other hand, if the key frame extraction method determination processing type information is in the range of “1” to “R” in step S194, the determination processing for each key frame extraction method determination processing type information is performed in steps S195 to S200. I do. Specifically, it is determined whether or not the key frame extraction method determination processing type information is “1” (step S195). If the key frame extraction method determination process type information is “1”, a determination process is performed on the key frame extraction method determination process type information 1 (step S196). It is determined whether or not “true” is determined in the determination process of step S196 (step S201). If it is determined as “true”, the determination process counter is incremented by +1 (step S201A), and the process returns to step S192. On the other hand, if “false” is determined in step S201, the determination result is “false” in step S202, and the process is terminated.

  If the key frame extraction method determination processing type information is not “1” in step S195, it is determined whether or not the key frame extraction method determination processing type information is “2” (step S197). If the key frame extraction method determination process type information is “2”, a determination process is performed on the key frame extraction method determination process type information 2 (step S198). Thereafter, the process proceeds to step S201. If the key frame extraction determination processing type information is not “2” in step S197, the same processing is performed until the key frame extraction method determination processing type information is “R−1”, and further the key frame extraction method determination processing type information. Is not “R−1”, it is determined whether or not the key frame extraction determination processing type information is “R” (step S199). If the key frame extraction method determination process type information is “R”, a determination process for the key frame extraction method determination process type information R is performed (step S200). Thereafter, the process proceeds to step S201. On the other hand, if the key frame extraction method determination process type information is not “R”, the process proceeds to step S202.

  Here, the determination process for the key frame extraction method determination process type information 1 in step S196 is shown. FIG. 45 is a flowchart showing the determination processing procedure for the key frame extraction method determination processing type information 1 in step S196. First, the start frame number is acquired from the sub-shot information in the video section where the key frame is changed (step S211). Based on the determination type information, it is compared whether the new key frame number and the start frame number are “=”, that is, the same (step S212). If the result of this comparison is “same”, the determination result is “true (1)” (step S213), the present process is terminated, and the process returns to the original process. On the other hand, if the result of comparison in step S212 is “not the same”, the determination result is false (0) (step S214), the process is terminated, and the process returns to the original process.

  Similarly, a determination process for the key frame extraction method determination process type information 14 is shown. FIG. 46 is a flowchart showing a determination processing procedure for the key frame extraction method determination processing type information 14. First, the event type of the shooting environment information for the frame number of the key frame is acquired (step S221). It is determined whether or not the acquired event type is “pan continuation” (step S222). If it is not “pan continuation”, the determination result is set to “false (0)” (step S223), and this process ends. On the other hand, if “pan continuation” is determined in step S222, the determination result is “true (1)” (step S224). A pan start frame number is acquired from the sub-shot information (step S225), and a pan end frame number is acquired (step S226). According to Equation (4), the ratio of key frame positions during panning is calculated and acquired (step S227).

Ratio = (frame number of key frame−pan start frame number) / (pan end frame number−pan start frame number) (4)
Next, the outline of the change processing of the similar moving image section is shown. FIG. 47 is a flowchart showing the procedure for changing the similar moving image section in step S157. First, an initialization process for changing a similar moving image section is performed (step S232). In this initialization process, for example, the pointer P indicating the sub-shot information currently being processed is moved to the top of the sub-shot information list. In addition, the classification information of the video section whose key frame has been changed is acquired. Then, the processing from step S233 to step S236 is performed, and sub-shot information that matches the classification information of the video section in which the key frame has been changed is acquired from the sub-shot information.

  Specifically, it is determined whether or not the pointer P has reached the end of the sub-shot information list (step S233). When it is determined that the comparison with the classification information in all the sub-shot information has not been completed, the classification information of the sub-shot information is acquired from the pointer P of the sub-shot information list (step S234), and the video with the key frame changed It is determined whether or not it matches the section classification information (step S235). If they match, the sub-shot information address is held in the temporary buffer as a similar moving image section (step S236). Then, the process returns to the process of step S233, and the next sub-shot information comparison process is performed. On the other hand, if they do not match in step S235, the process proceeds to the next sub-shot information comparison process.

  On the other hand, if it is determined in step S233 that the comparison with the classification information in all the sub-shot information has been completed, the sub-shot information whose classification information matches the video section whose key frame has been changed is included in the sub-shot information list. It is determined whether or not there is (step S237). If there is no sub-shot information that matches the classification information, the key frame changing process for the similar moving image section is terminated.

  On the other hand, if there is sub-shot information whose classification information matches the video section whose key frame has been changed in step S237, new key frame extraction type information and key frame extraction related values in the video section whose key frame has been changed are acquired. (Step S238A). Since this process is the same as the process of step S162 in the key frame extraction history information registration process (see FIG. 42), the new key frame extraction type information and the key frame extraction related value are already included in the key frame extraction history information registration process. Is acquired, the value acquired in the key frame extraction history information registration process is used.

  Then, based on the acquired key frame extraction method, key frames are extracted from all the video sections held as the similar moving image sections in step S236 (step S239A). The extracted key frame is additionally registered in the element data Keyframe of the sub-shot information, the value 1 is stored in the element data NumberOfTimes, and the value 1 is added to the element NumberOfKeyframe (step S240A). Then, this process is complete | finished.

  FIG. 48 is a flowchart showing the key frame list display processing procedure in step S10. First, initialization processing is performed, and the pointer J is moved to the top of the sub-shot information list (step S251). Then, it is determined whether or not the pointer J has moved to the end position of the sub-shot information list, that is, whether or not the key frame list display process of the divided video section has ended (step S252). If completed, the process proceeds to step S256.

  On the other hand, if the key frame list display process for the divided video section has not been completed, the element data NumberOfTimes is referred to from the pointer J, that is, the subshot information indicated by the subshot information pointer, and the maximum number of times of use in this divided video section is determined. A key frame is acquired (step S253). Then, it is determined whether or not the display of the acquired key frame has been finished (step S254). If not, the key frame is displayed (step S255), and the process returns to step S254. On the other hand, when the display of the key frame is finished, the process returns to step S252. In this way, a list of key frames shown in FIG. 49 is displayed.

  When the key frame list display processing for all the divided video sections is completed in step S252, an instruction input from the operator is awaited (step S256). If there is an instruction input from the operator, it is determined whether or not the key frame displayed in the list is selected (step S257). In the present embodiment, it is possible to activate the playback and editing functions for the video section by selecting an arbitrary key frame displayed in a list. When the key frame is selected, the value 1 is added to the element data NumberOfTimes of this key frame (step S258). At this time, the element data Keyframe in which the element data NumberOfTimes is updated is temporarily stored in the local memory. Then, processing such as playback and editing of the activated video section is performed (step S259), and the process returns to step S256 to wait for an instruction input from the operator.

  On the other hand, if no key frame is selected in step S257, it is determined whether or not it is an end instruction (step S260). If an end instruction is given, the sub-shot information is updated based on the element data NumberOfTimes and the element data Keyframe temporarily stored in step S258 (step S261), and this process ends. On the other hand, if the end instruction is not issued in step S260, other processes are performed (step S262), and the process returns to step S256.

  FIG. 51 is a diagram showing an operation screen for displaying a list of key frames to which new key frames are added. New key frames E-3 and I-3 are added to the similar video sections E and I, respectively. Since the operator changes the key frame of the video section B, the key frame of the video section B is changed from B-1 to B-3. On the other hand, in the video section E, a key frame E-1 from which key frames are automatically extracted and a key frame E-3 created when the key frame in the video section B is changed are displayed as key frames. .

  When the key frame I-3 of the video section I is selected on the operation screen of FIG. 51 and the editing function of the video section I is activated, the value of the element data NumberOfTimes of the key frame I-3 is “2”. Then, when this key frame list display is finished and the next key frame list display is performed, in the video section I, the value of the element data NumberOfTimes of the key frame I-3 is larger than the key frame I-1. Only key frame I-3 is displayed as a key frame. FIG. 52 is a diagram showing an operation screen on which the next key frame list is displayed after the key frame I-3 of the video section I is selected on the operation screen of FIG.

  As described above, according to the imaging apparatus of the first embodiment, video data is divided into a plurality of video sections, and keyframes are automatically extracted from the divided video sections, and then extracted by an operator. When the changed key frame is changed, the key frame extraction method in the video section where the key frame is changed is acquired. A similar video section similar to the video section in which the key frame has been changed is extracted, and the extracted key frame is added as a key frame of the similar video section to the already extracted key frame. Therefore, it is possible to easily change a key frame in a similar video section similar to the video section in which the operator has changed the key frame. Further, only key frames that are frequently used by the operator can be displayed.

  For example, when shooting a video of a specific subject such as a flower or a person, the shooting environment such as zooming and panning and tilting are often similar, and if the photographer is the same, the shooting method for the subject Are likely to be similar. Therefore, if the automatically extracted key frame is not the optimum key frame for the operator, similar video sections are automatically extracted and these key frames are changed by one change operation. Even if there are many sections, the number of change operations can be reduced, and operability can be improved.

  On the operation screen, only one key frame of each video section is selectably displayed on the upper part of the operation screen, and is registered as a key frame of the selected similar video section in the lower part. A plurality of key frames may be displayed, and only a specific key frame may be displayed in the upper part among them.

[Second Embodiment]
In the first embodiment, the key frame extraction method is specified mainly using the attached information such as the shooting environment information, the operation information, and the frame position information. The key frame extraction method may be specified with reference to information or audio information. As a conventional technique, the camera described in Patent Document 3 not only recognizes the face of the subject, but also determines the direction of the face (the line of sight is facing / not facing the camera) and the state of the subject's eyes (eyes). Closed / not closed), facial expressions (joy face, sad face, angry face, surprise face, etc.), subject posture (recognizes whether a recognized person is in a specific pose) Etc. are recognizable. In the person detection function-equipped device described in Patent Document 4, it is possible to identify a speaker from subject recognition and voice information. In the second embodiment, subject information (see FIG. 54) and speaker information (see FIG. 56) are acquired and recorded for each frame using these techniques, as with shooting environment information and operation information. To.

  FIG. 53 shows the structure of the subject information data list. FIG. 54 shows the structure of subject information data. In the subject information data, information on the subject acquired by the camera described in Patent Document 3 is stored. Each time the frame number of the created subject information changes, the head pointer of the subject information is stored in the subject information data list. Accordingly, it is possible to obtain subject information by obtaining the head pointer of the frame number from the subject information data list.

  Each element data of the subject information data of FIG. 54 is shown. In the element data “frame number”, a frame number based on the head of the moving image file is described. In the element data “subject ID”, the identification number of the subject shown in the frame number image is described. The element data “face orientation information” stores the face orientation of the subject. Here, “0” indicates that recognition is impossible, “1” indicates that the camera is facing, and “2” indicates that the camera is not facing. The element data “eye state information” stores the eye state of the subject. Here, “0” indicates that recognition is impossible, “1” indicates that the eyes are closed, and “2” indicates that the eyes are open. The element data “facial expression information” stores the facial expression of the subject. Here, “0” represents unrecognizable, “1” represents no expression, “2” represents joy, “3” represents sadness, “4” represents anger, “5” represents Represents surprise. The element data “posture information” stores the posture state of the subject. Here, “0” represents unrecognizable, “1” represents sitting, “2” represents standing, and “3” represents running.

  FIG. 55 shows the structure of the audio information data list. FIG. 56 shows the structure of audio information data. In the audio information data, information related to audio acquired by the person detection function-equipped device described in Patent Document 4 is stored. Each time the frame number of the created audio information changes, the head pointer of the audio information is stored in the audio information data list. Therefore, it is possible to acquire audio information by acquiring the head pointer of the frame number from the audio information data list.

  Each element data of the audio | voice information data of FIG. 56 is shown. In the element data “frame number”, a frame number based on the head of the moving image file is described. The element data “sound information” stores a value indicating whether or not sound is included. Here, “1” represents “entered” and “0” represents “not entered”. The element data “number of stored subjects” describes the number of stored subject IDs for the image of the frame number. The element data “subject ID” stores the subject ID of the person who is speaking when the speech is recognized in the image indicated by the frame number.

  The key frame of the key frame selected by the operator is added to the above-described video section classification information data, key frame extraction history information data, and key frame extraction method determination information data, respectively, by adding the subject information and audio information additional information. It becomes possible to improve the accuracy of the extraction method.

  For example, as key frame extraction methods using subject information and audio information, the following methods “13” to “15” can be cited.

  13. Extract the frame with the largest number of subjects. That is, the subject information from the start frame number to the end frame number of the sub-shot information is referred to, the number of subject IDs of each frame number is acquired, and the largest number of frames is extracted as a key frame.

  14 Extract the frame in which the most reflected subject appears for the first time. That is, reference is made to the subject information from the start frame number to the end frame number of the sub-shot information, the number of stored frames for each subject ID is acquired, and the search starts from the start frame number for the largest number of subject IDs. The frames to be extracted are extracted as key frames.

  15. As an example of the key frame extraction method acquired by changing the key frame, a frame in which the subject ID (A) is shown in the pan intermediate frame is extracted. That is, referring to the shooting environment information, a pan start frame and a pan end frame are obtained, a frame in which the subject ID (A) appears in the frame number is obtained from the subject information, and what is selected as a key frame. Get the second.

  As described above, according to the imaging apparatus of the second embodiment, it is possible to specify a key frame extraction method in detail by storing subject information and the like.

  The above is the description of the embodiments of the present invention. However, the present invention is not limited to the configurations of these embodiments, and the functions shown in the claims or the functions of the configurations of the embodiments are included. Any configuration that can be achieved is applicable.

  For example, in the user identification process of the above embodiment, the user is recognized using the fingerprint sensor. However, any processing method may be used as long as the process recognizes the user. FIG. 57 is a diagram showing a character table displayed on the display unit 18. The operator moves the cursor to the character table displayed as a list on the display unit 18 to specify a character string, and inputs identification information such as the name of the operator. And a user identification process may be performed by registering the serial number (user ID) automatically given by the input of this identification information in a user identification data list.

  In the above embodiment, the frame number based on the playback time of the moving image file is used as the unit of time for shooting environment information, operation information, etc., but in any unit as long as it can be synchronized with video and audio. There may be. In this embodiment, the key frame is indicated by the frame number. However, the key frame may be indicated by any unit as long as the frame can be specified. Alternatively, key frame image data itself or image data that has been subjected to image processing such as reduction may be created in another area, and pointer information or hyperlinks to the image data may be described.

  In the above embodiment, the video processing apparatus is realized by a single imaging apparatus, but is connected to a network by a communication means such as a serial interface such as USB 2.0, IEEE 1394, Ethernet (registered trademark), or wireless LAN. Alternatively, the same function may be realized.

  Further, sub-shot division and key frame extraction processing are not limited to being performed at the end of recording. For example, information on a recording medium is read by an external device such as a personal computer (PC), The same process may be performed to process the attached information.

  Further, in the present embodiment, the information from the rotation detection unit (the X direction rotation detection unit 21 and the Y direction rotation detection unit 22) that detects the rotation of the imaging device is exemplified as the shooting environment information. If the key frame selection criteria changes depending on the case, it is not limited to this. For example, the output information of a measuring device such as a GPS, a position measuring system, a thermometer, an altimeter, a compass, a barometer, a hygrometer, etc. Alternatively, biological information such as a photographer's body position, line of sight, pupil size, skin resistance, pulse, brain waves, and the like may be used.

  Moreover, although the case where one key frame was selected from the sub-shot was shown in the above embodiment, the number is not limited to one and may be plural. Further, when there is no frame image suitable as a key frame, the selection may not be performed.

  In addition, an object of the present invention is to supply a storage medium in which a program code of software for realizing the functions of the embodiment is recorded to a system or apparatus, and a computer (or CPU, MPU, etc.) of the system or apparatus as the storage medium. This can also be achieved by reading and executing the stored program code. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  Examples of storage media for supplying the program code include ROM, floppy (registered trademark) disk, memory card such as PCMCIA card and compact flash (registered trademark), hard disk, micro DAT, magneto-optical disk, CD- It may be composed of an optical disk such as R or CD-RW, a phase change optical disk such as DVD, or the like.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) or the like running on the computer is actually executed based on the instruction of the program code. This includes a case where part or all of the processing is performed and the functions of the above-described embodiments are realized by the processing.

  Further, after the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. This includes the case where the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

It is a block diagram which shows the structure of the imaging device in 1st Embodiment. 3 is a diagram illustrating a configuration of a system control unit 23. FIG. 3 is a block diagram illustrating a configuration of a video management unit 17. FIG. It is a figure which shows the structure of imaging | photography environment information data. It is a figure which shows the structure of an imaging | photography environment information data list. It is a figure which shows the structure of operation information data. It is a figure which shows the structure of an operation information data list. It is a figure which shows the structure of subshot information data. It is a figure which shows the structure of a subshot information data list. It is a figure which shows the structure of video area classification | category information. It is a figure which shows a video area classification | category information list. It is a figure which shows the structure of classification condition information. It is a figure which shows video area classification | category information. It is a figure which shows the classification condition information in video section classification information "1". It is a figure which shows user key frame extraction log | history information. It is a figure which shows the structure of a user key frame extraction log | history list. It is a figure which shows the structure of key frame extraction log | history information. It is a figure which shows the structure of a key frame extraction log | history information list. It is a figure which shows the structure of key frame extraction detailed information. It is a figure which shows the structure of a key frame extraction detailed information list. It is a figure which shows the structure of the key frame extraction method determination information. It is a figure which shows the structure of a key frame extraction method determination information index list. It is a figure which shows the structure of key frame extraction method condition information. It is a figure which shows the specific example of key frame extraction method determination information. It is a flowchart which shows the key frame setting process sequence in an imaging device. It is a flowchart which shows the key frame setting process sequence in the imaging device following FIG. It is a flowchart which shows the image | video division | segmentation process procedure in step S8. It is a flowchart which shows the key frame setting process sequence in step S9. It is a flowchart which shows the acquisition processing procedure of the classification information in step S44. It is a flowchart which shows the acquisition processing procedure of the classification information in step S44 following FIG. It is a flowchart which shows the judgment processing procedure of the classification judgment processing type information 1 in step S60. It is a flowchart which shows the judgment processing procedure of the classification judgment processing type information 2 in step S60. It is a flowchart which shows the judgment processing procedure of the classification judgment processing type information 3 in step S60. It is a flowchart which shows the judgment processing procedure of the classification judgment processing type information 4 in step S60. It is a flowchart which shows the key frame extraction process procedure in step S47. It is a flowchart which shows the key frame extraction process procedure of the key frame extraction type information 1 which sets a start frame as a key frame. It is a flowchart which shows the key frame extraction process procedure of the key frame extraction type information 2 which sets an end frame as a key frame. It is a flowchart which shows the key frame extraction process procedure of the key frame extraction type information 3 which sets the intermediate | middle frame in pan as a key frame. It is a flowchart which shows the key frame extraction process procedure using the key frame extraction detailed information in step S92. It is a flowchart which shows the key frame extraction process procedure of the key frame extraction detailed information type information 3. FIG. It is a flowchart which shows the key frame change process sequence in step S11. It is a flowchart which shows the key frame extraction log | history information registration process procedure in step S155. It is a flowchart which shows the newly set key frame extraction type information acquisition process procedure in step S162. It is a flowchart which shows the condition information determination processing procedure in step S177. It is a flowchart which shows the judgment processing procedure with respect to the key frame extraction method judgment processing type information 1 in step S196. 7 is a flowchart showing a determination processing procedure for key frame extraction method determination processing type information 14; It is a flowchart which shows the change process sequence of the similar animation area in step S157. It is a flowchart which shows the key frame list display process sequence in step S10. It is a figure which shows the operation screen which displays the key frame for every moving image area as a list. It is a figure which shows the operation screen at the time of performing the change process of the key frame in a moving image area. It is a figure which shows the operation screen which displays the key frame to which the new key frame was added as a list. FIG. 52 is a diagram showing an operation screen on which a list of next key frames is displayed after key frame I-3 of video section I is selected on the operation screen of FIG. 51. It is a figure which shows the structure of a to-be-photographed information data list. It is a figure which shows the structure of object information data. It is a figure which shows the structure of an audio | voice information data list. It is a figure which shows the structure of audio | voice information data. It is a figure which shows the character table displayed on the display part. It is a figure which shows a mode that a key frame is extracted from the inside divided | segmented for every video area.

Explanation of symbols

DESCRIPTION OF SYMBOLS 17 Image management part 18 Display part 19 Lens position detection part 21 X direction rotation detection part 22 Y direction rotation detection part 23 System control part 24 Operation input part 31 Fingerprint detection part 41 CPU
42 ROM
56 Video segmentation unit 57 Key frame extraction unit 58 Similar video segment extraction unit 59 Attached information management unit

Claims (11)

Representative image changing means for changing the representative image extracted from the video data;
Representative image extraction method acquisition means for acquiring a representative image extraction method in a video section in which the representative image is changed;
A similar video section extraction means for extracting a similar video section that is a video section similar to the video section in which the representative image is changed;
Using the representative image extraction method in the video section in which the representative image is changed, the representative image of the extracted similar video section is extracted, and the extracted representative image is used as the representative image of the similar video section. A video processing apparatus comprising: similar video section representative image adding means for adding to the representative image of the similar video section extracted by the representative image extracting means. Display means for displaying a representative image extracted from the video section;
Video section selection means for selecting a video section including the displayed representative image,
The video processing apparatus according to claim 1, wherein the representative image changing unit changes a representative image included in the selected video section. An activation unit that activates playback or editing of a video section including the representative image by selecting the displayed representative image;
Use number information updating means for updating the use number information indicating the use frequency of the representative image selected for starting the reproduction or editing,
The video processing apparatus according to claim 2, wherein the display unit displays only a representative image that is most frequently used among a plurality of representative images in the video section.   4. The video processing apparatus according to claim 2, further comprising representative image switching means for switching a representative image of the similar video section displayed by the display means from a plurality of images.   The video processing apparatus according to claim 1, wherein the representative image extraction method acquisition unit determines the representative image extraction method with reference to attached information given to a video section in which the representative image is changed. .   The representative image extraction method acquisition means includes representative image extraction method condition information using the attached information and frame position information, and a representative image setting for setting a representative image of a video section that satisfies the representative image extraction method condition information. 6. The video processing apparatus according to claim 5, wherein representative image extraction method determination information including information is held, and the representative image extraction method is determined using the representative image extraction method determination information. Classification information acquisition means for performing classification based on the attached information given to the video section and acquiring the classification information of the video section,
The similar video section extraction means holds video section classification information including classification information of the acquired video section and classification condition information using the attached information and frame position information of the representative image, and the classification condition The video processing apparatus according to claim 1, wherein a video section having classification information that matches classification information satisfying information is extracted as the similar video section.   The video processing apparatus according to claim 5, wherein the attached information includes shooting environment information and operation information. Subject information detection means for detecting subject information is provided,
The video processing apparatus according to claim 5, wherein the attached information includes the subject information. Comprising speaker information detecting means for detecting speaker information;
The video processing apparatus according to claim 5, wherein the attached information includes the speaker information. A representative image changing step for changing the representative image extracted from the video data;
A representative image extraction method acquisition step of acquiring a representative image extraction method in a video section in which the representative image is changed;
A similar video segment extraction step for extracting a similar video segment which is a video segment similar to the video segment in which the representative image is changed;
Using the representative image extraction method in the video section in which the representative image is changed, the representative image of the extracted similar video section is extracted, and the extracted representative image is used as the representative image of the similar video section. A representative image setting method comprising: a similar video section representative image adding step of adding to a representative image of the similar video section extracted in the representative image extraction step.

Images