JP2003110995A - Image processing apparatus, image processing system, image processing method, storage medium, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus by which a user can easily perform desirable sophisticated moving picture edit without notifying expert intelligence about moving picture edit (video edit). SOLUTION: An acquisition means 131 acquires image data (video data obtained by operating a video camera and photographing an object) obtained by operation of an optional device or system and information (operation information) with respect to the operation. A processing means 133 applies prescribed image processing (special effect processing at image edit) to the image data obtained by the acquisition means 131 on the basis of the operation information obtained by the acquisition means 131.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, an image processing system, and an image processing used for an apparatus or a system for editing a moving image (video) obtained by shooting with a video camera or the like. The present invention relates to a method, a computer-readable storage medium storing a program for implementing the method, and the program.

[0002]

2. Description of the Related Art Conventionally, in an image editing apparatus for editing an image (moving image) obtained by shooting with a video camera or the like, when a user performs an image editing operation, for example, the user is an object to be edited. Determine the necessary section of the video (in the video material) and record the scenes in this section in order.
This result is the result of video editing.

By the way, when a user who edits an image is, for example, a professional user who is involved in image production, such a user generally has pan, tilt, zoom, and camera angle in an image (moving image material). By combining expressions (scenes) operated by the camera such as, for example, there is a technology for obtaining a sharp and powerful image. On the other hand, non-professional users, especially amateur users, do not have enough knowledge about what kind of image effects can be obtained by the expression by the camera operation and the combination thereof, so that it seems to be a sharp image. It is difficult to obtain a good video as a result of editing.

Therefore, in order to solve the above problems, an image editing apparatus having a structure as disclosed in, for example, Japanese Patent Laid-Open No. 9-37150, has a natural "stimulating" or the like. It has a dictionary that supports the expression of words (natural language) and the video effects by operating the camera, and it is possible to easily obtain the results of advanced video editing by specifying the video effects using natural language from the user. There is.

[0005]

However, the conventional image editing apparatus described in Japanese Patent Laid-Open No. 9-37150, etc., as described above, has a configuration in which the video effect is specified by the expression in natural language. It was not possible to cope with the difference due to the subjectivity of. For example, even if the same natural language expression "stimulatingly" is used, there is often a difference in the degree of "stimulatingly" feeling between users. Very difficult. In addition, the user had to decide in advance what kind of video effect, etc., before performing the actual video editing work, so that the user can freely expand the idea while proceeding with the editing work. It was very difficult to do such video editing work.

Therefore, the present invention has been made in order to eliminate the above-mentioned drawbacks, and facilitates highly desired moving image editing without being aware of specialized knowledge about moving image editing (video editing). It is an object of the present invention to provide an image processing device, an image processing system, an image processing method, a computer-readable storage medium storing a program for implementing the same, and the program.

[0007]

[Means for Solving the Problems] Under such a purpose,
According to a first aspect of the present invention, there is provided an image data obtained by an operation on an arbitrary device or a system, an acquisition means for acquiring information regarding the operation, and an acquisition means for acquiring the operation information based on the operation information obtained by the acquisition means. The image processing apparatus is provided with a processing unit that performs predetermined image processing on the obtained image data.

A second invention is the same as the first invention,
The arbitrary device or system has a function of acquiring moving image data by a shooting operation according to an operation performed by a user, and the operation information includes information about an operation at the time of shooting.

A third invention is the same as the first invention,
The arbitrary device or system includes a camera, and the operation information includes movement of the camera when acquiring the image data, a state of the camera, a type of operation of the camera, a direction of the camera, and an operation of the camera. It is characterized in that it includes at least information of at least one of the sections.

A fourth invention is the same as the first invention,
The acquisition unit is characterized by reading and acquiring the image data and the operation information from an arbitrary recording medium.

A fifth invention is based on the first invention,
The predetermined image processing includes a special effect processing for the image data.

A sixth invention is the same as the first invention,
The special effect process includes a process of adding a transition effect to the image data.

A seventh invention is based on the first invention,
The specifying means includes a specifying means for specifying a plurality of arbitrary sections among a plurality of sections forming the moving image data obtained as the image data by the obtaining means, and the processing means is the specifying means obtained by the obtaining means. The above-mentioned moving image data is subjected to predetermined image processing based on operation information corresponding to moving image data of a plurality of designated sections.

An eighth invention is based on the first invention,
The processing means is characterized by performing predetermined image processing on the image data based on a rule regarding a combination of operations on the arbitrary device or system.

In a ninth aspect based on the eighth aspect,
The processing means may perform predetermined image processing on the image data based on a comparison result between a rule regarding a combination of the operations and information about an operation actually performed on the arbitrary device or system. Characterize.

A tenth aspect of the present invention is an image processing apparatus for moving image data obtained by an operation on an arbitrary device or system, wherein the obtaining means obtains the moving image data and information on the operation, and the obtaining means. By a plurality of arbitrary sections among the plurality of sections constituting the moving image data obtained by the means, a specifying means for specifying the order of the plurality of sections, and the operation information acquired by the acquisition means, Detecting means for detecting operation information corresponding to the section designated by the designating means, storage means for storing rules relating to operation on the arbitrary device or system, operation information detected by the detecting means, and storage means And a processing unit that adds a transition effect to the moving image data obtained by the acquisition unit based on the result of matching the rules stored in To.

An eleventh aspect of the invention is the same as the tenth aspect of the invention, wherein the storage means is at least the existence section of the operation,
And a rule regarding a sequence on the time series is stored.

In a twelfth aspect based on the tenth aspect, when the processing information detected by the detecting means is information relating to an operation in a section shorter than a predetermined length, the processing means is a target of the collation. It is characterized in that it does not.

A thirteenth invention is characterized in that, in the tenth invention, the processing means executes the processing based on the collation result on the basis of the timing of the section designation by the designating means.

In a fourteenth aspect based on the tenth aspect, the processing means obtains a transition effect prepared as moving image data different from the moving image data obtained by the obtaining means by the obtaining means. It is characterized in that it is added to the obtained moving image data.

In a fifteenth aspect based on the tenth aspect, the processing means applies a transition effect to the moving image data when the operation based on the operation information violates the rule based on the collation result. It is characterized by being given.

In a sixteenth aspect based on the tenth aspect, the processing means determines that the operation based on the operation information does not violate the rule based on the collation result, and the transition has already been made to the moving image data. When an effect is added, the transition effect is removed from the moving image data.

The seventeenth invention is characterized in that, in the tenth invention, the transition effect has a fixed length or a length according to the operation information.

An eighteenth invention is an image processing system in which a plurality of devices are communicably connected to each other, and at least one device among the plurality of devices is defined by any one of claims 1 to 1.
17. The image processing device according to any one of 17).

A nineteenth aspect of the present invention is an image processing method for moving image data obtained by a shooting operation, wherein a predetermined amount is set for the moving image data based on information about the shooting operation when the moving image data is acquired. It is characterized by including a processing step for performing image processing.

A twentieth invention is characterized in that, in the nineteenth invention, the predetermined image processing includes a special effect processing for the moving image data.

A twenty-first aspect of the invention is directed to the nineteenth aspect of the invention, which includes a designating step of designating a plurality of arbitrary sections of the plurality of sections forming the moving image data, and the processing step is designated by the designating step. It is characterized by including a step of subjecting the moving image data to predetermined image processing based on information regarding a shooting operation at the time of acquiring moving image data of a plurality of sections.

In a twenty-second aspect of the invention based on the nineteenth aspect, the processing step is based on a comparison result between a predetermined rule relating to the photographing operation and the photographing operation at the time of acquiring the moving image data. And a step of performing a predetermined image processing on

A twenty-third invention is an image processing method for moving image data obtained by a video camera, wherein the moving image data obtained by the video camera and the operation of the video camera at the time of obtaining the moving image data. A reproduction step of reproducing information on an arbitrary recording medium, and a processing step of performing special effect processing on the moving image data reproduced by the reproduction step, based on the operation information reproduced by the reproduction step. It is characterized by including.

A twenty-fourth invention is an image processing method for moving image data obtained by a video camera, wherein the moving image data obtained by the video camera and the operation of the video camera at the time of obtaining the moving image data. Information about the information from any recording medium, a designation step of designating a plurality of arbitrary sections of the plurality of sections constituting the moving image data reproduced by the reproduction step, and a designation step of the specification step. And a processing step of performing special effect processing on the moving image data reproduced by the reproducing step based on the operation information corresponding to the moving image data of the plurality of sections.

A twenty-fifth aspect of the present invention is an image processing method for moving image data obtained by a camera operation, comprising a storing step of storing the moving image data and information about the camera operation, and storing information by the storing step. A reading step of reading moving image data to be processed and camera operation information corresponding thereto; a section designating step of designating a plurality of arbitrary sections for the target moving image data read by the reading step; A detection step of detecting camera operation information corresponding to the section designated by the designation step from the camera operation information read by the reading step, a rule storage step of storing a rule relating to the camera operation, and a detection step of the detection step. Stored by the rule storing step Based on the comparison result of the rule, characterized in that it comprises a processing step of applying a transition effect to the moving image data read out by said reading step.

A twenty-sixth aspect of the present invention is an image processing method for moving image data obtained by operating a camera, comprising a storing step of storing the moving image data, and the moving image from the moving image data stored by the storing step. An extraction step of extracting camera operation information when acquiring image data,
The camera operation extracted by the extracting step includes a specifying step of specifying a plurality of arbitrary sections for the moving image data stored in the storing step, and camera operation information corresponding to the section specified by the section specifying step. A detection step of detecting from information, a rule storage step of storing a rule relating to camera operation, camera operation information detected by the detection step, and a comparison result of the rule stored by the rule storage step, And a processing step of applying a transition effect to the moving image data stored by the storing step.

A twenty-seventh aspect of the present invention is an image processing method for moving image data obtained by shooting by operating a camera, and a shooting step of shooting a subject by operating the camera and generating moving image data of the subject. An acquisition step of acquiring the setting information and the camera state information for the camera in the photographing step as camera operation information, the moving image data obtained in the photographing step, and the camera operation information obtained in the acquisition step are arbitrary recording media. A recording step of recording the moving image data to be processed and the camera operation information corresponding thereto from the recording medium, and an arbitrary section for the target moving image data read by the reading step. Section specified step multiple times, and specified by the section specification step above Corresponding camera operation information between,
A detection step of detecting from the camera operation information read by the reading step, a rule storage step of storing a rule regarding camera operation, a camera operation information detected by the detection step, and a rule storage step stored by the rule storage step. And a processing step of applying a transition effect to the moving image data read by the reading step, based on the result of matching with the rule.

A twenty-eighth invention is characterized in that, in any one of the twenty-fifth to twenty-seventh inventions, it includes an order designating step for designating the order of the plurality of sections designated by the section designating step.

A thirty-ninth aspect of the invention is the camera according to any one of the twenty-fifth to twenty-seventh aspects, wherein the camera operation information indicates a camera movement, a camera state, a type of camera operation, and a camera operation at the time of acquiring moving image data. And at least one piece of information on the section and the direction of the camera.

The thirtieth invention is characterized in that, in any one of the twenty-fifth to twenty-seventh inventions, the rule includes at least an existing section of the camera operation information and a time-series arrangement thereof.

In a thirty-first invention based on the twenty-fifth to twenty-seventh inventions, in the processing step, when the camera operation indicated by the camera operation information is a camera operation in a section shorter than a predetermined length, It is characterized in that it includes a step in which the camera operation information is not a target of the collation.

A thirty-second invention is characterized in that, in any one of the twenty-fifth to twenty-seventh inventions, the processing step includes a step of executing processing each time a section is specified by the user in the section specifying step. And

A thirty-third invention is characterized in that, in any of the twenty-fifth to twenty-seventh inventions, the transition effect is
It is characterized by including a holding step of generating and holding moving image data different from the moving image data to be processed.

A thirty-fourth invention is based on any one of the twenty-fifth to twenty-seventh inventions, and in the processing step, when the camera operation indicated by the camera operation information is against the rule based on the collation result, the processing step is performed. If the camera operation indicated by the camera operation information does not violate the rules based on the step of executing the transition effect applying process and the collation result, whether or not the transition effect has already been applied to the moving image data. And whether the transition effect has already been added as a result of the determination, the step of removing the transition effect from the moving image data is included.

A thirty-fifth aspect of the invention is the invention of any one of the twenty-fifth to twenty-seventh aspects, wherein the transition effect is
It is characterized by having a fixed length or a length according to the camera operation information.

A thirty-sixth invention is characterized in that, in any one of the twenty-fifth to twenty-seventh inventions, the detecting step includes a step of previously managing camera operation information for each section.

A thirty-seventh invention is a computer-readable program for causing a computer to realize the function of the image processing apparatus according to any one of claims 1 to 17 or the function of the image processing system according to claim 18. It is characterized by being recorded in a storage medium.

A thirty-eighth invention is characterized in that a program for causing a computer to execute the processing steps of the image processing method according to any one of the nineteenth to thirty-sixth aspects is recorded in a computer-readable storage medium.

A thirty-ninth invention is a program for causing a computer to realize the function of the image processing apparatus according to any one of claims 1 to 17 or the function of the image processing system according to claim 18. And

A fortieth aspect of the invention is a program for causing a computer to execute the processing steps of the image processing method according to any one of the nineteenth to thirty-sixth aspects.

[0047]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

[First Embodiment] The present invention is applied to, for example, a moving image editing apparatus 100 as shown in FIG. In the moving image editing apparatus 100 according to the present embodiment, particularly when the user edits a moving image (video), even if the user is a user who has little knowledge about video effects,
It is configured so that desired video editing work can be freely performed and a high-level video editing result can be easily obtained. Hereinafter, the configuration and operation of the moving image editing apparatus 100 according to this embodiment will be specifically described.

<Overall Structure of Moving Image Editing Apparatus 100> As shown in FIG. 1, the moving image editing apparatus 100 includes a photographing section 110 for photographing a subject and acquiring a video (moving image) of the subject, and a photographing section. A recording medium 120 in which the video data obtained by the unit 110 is recorded, and a moving image editing unit 1 for editing the video data recorded in the storage medium 120 based on a user's instruction.
30 and 30.

<Structure of Photographing Unit 110>
For example, as shown in FIG. 2, the image pickup unit 111, the image compression unit 112, the camera operation unit 113, the sensor unit 114, the control unit 115, and the recording unit 116 are provided.

The image pickup section 111 obtains moving image data (video data) of the subject by taking an image of the subject with a configuration including a lens, an image pickup element, an A / D converter, etc. (not shown). Output. The image compression unit 112 subjects the video data obtained by the imaging unit 111 to compression encoding processing based on an algorithm of a predetermined compression method. As a compression method here, for example, DCT (Discrete Cosine Transform) and VLC (Va
Variable Length Code) and MPEG (Moving Picture Experts Grou) using DCT and motion compensation prediction.
Examples thereof include p) -1 and MPEG-2. The camera operation unit 113 has a shooting start button, a zoom button, and the like, and inputs the operation information based on the operation by the user. The sensor unit 114 includes various sensors for detecting the state of the image capturing unit 11 (state of the image capturing unit 111, hereinafter also referred to as “camera state”), and for example, the image capturing unit 11
The detection result information (hereinafter, also referred to as “sensor information”) such as the aperture of the diaphragm and the focal length in 1 is acquired. Control unit 11
Reference numeral 5 controls the operation of the entire imaging unit 110,
For example, the imaging operation in the imaging unit 111 is controlled based on the operation information of the user input from the camera operation unit 113 and the information obtained by the sensor unit 114. Recording unit 116
Is the video data after the compression encoding process obtained by the image compression unit 112, and camera operation information from the control unit 115 (information including operation information by the user of the camera operation unit 113 and sensor information of the sensor unit 114). And are recorded on the recording medium 120 by performing multiplexing while taking timing.

On the recording medium 120, multiplexed data of video data and camera operation information is recorded by the recording unit 116 of the photographing unit 110. The recording medium 120 is, for example, a magnetic tape, a magneto-optical disk, or the like. Alternatively, a hard disk or the like can be applied.

FIG. 23 is a block diagram showing the structure of the control unit 115. As shown in FIG. 23, the control unit 115 has a well-known CPU 115 (0), ROM 115 (1),
RAM 115 (2), I / O interface 115
(3) and the bus 115 (4). ROM115
In (1), a program to be described later executed by the CPU 115 (0), a table value, and the like are stored.

<Structure of Moving Image Editing Unit 130> FIG. 3 specifically shows the control structure of the moving image editing unit 130.
The moving image editing unit 130, as shown in FIG.
30 (1), ROM130 (2), RAM130
(3), keyboard 130 (4), mouse 130
(5), external storage device 130 (6), display device 130
(7), and the NIC 130 (8), the bus 130 (1
It is configured so that they can communicate with each other via 1).

The CPU 130 (1) controls the operation of the entire moving picture editing unit 130 by executing a predetermined processing program or the like. The ROM 130 (2) is the image editing apparatus 100.
Boot program that is executed at startup of the
The processing program executed in U130 (1) and various data are stored. The RAM 130 (3) is the CPU 13
An area from which a processing program executed at 0 (1) is read out, a work area of the CPU 130 (1), and the like are provided. Keyboard 130 (4) and mouse 130 (5)
Provides various input operation environments by the user. A hard disk, a flexible disk, a CD-ROM, or the like can be applied as the external storage device 130 (6). The display unit 130 (7) includes a display and the like, and displays various processing results and the like to the user. NI
C130 (8) is a network interface and enables communication with a device or system on the network.

FIG. 4 is a diagram showing the functional configuration of the moving image editing section 130. The moving image editing unit 130 includes, for example, as shown in FIG. 4, a reading unit 131, a display unit 132, a scene sticker collating unit 133, a user operation unit 134, and a scene sticker storage unit 135.

The reading unit 131 uses the (external) storage medium 120.
In the data recorded in (data in which video data and camera operation information are multiplexed), data is read from an arbitrary position (address), and video data and camera operation are performed from this read data (multiplexed data). Get information.
As the reading unit 131, for example, a configuration such as a CD-ROM drive that directly reads data from the recording medium 120, or a configuration that reads data from a drive device built in the imaging unit 110 side via an interface such as IEEE1394. Etc. are applicable. Display 1
32 displays to the user information such as the state of video editing and the result of editing. The scene rule matching unit 133 matches a scene rule described later with camera operation information corresponding to the video data being edited, and based on the result, adds a transition effect to the scene joint. The user operation unit 134 receives an operation from the user necessary for video editing, such as designation of a scene section in the video being edited. The scene sticker storage unit 135 stores a scene sticker, which describes rules necessary for enhancing a video effect such as visibility and sharpness for a video.

<Operation of Image Editing Apparatus 100> (1) Operation of Photographing Section 110 The operation of the photographing section 110 is shown, for example, by the flowchart of FIG.

Step S201: The control section 115 determines whether or not the user has made a camera setting for the image pickup section 111 in the camera operation section 113. The camera settings here are, for example, focus mode (automatic / manual),
Exposure mode (automatic, gain priority, shutter speed priority,
Settings such as aperture priority, manual), white balance mode (auto, hold, one push, preset) are shown. As a result of this determination, if there is a camera setting, the process proceeds to the next step S202, and if there is no camera setting, the next step S202 is skipped and a step S to be described later.
Proceed to 203.

Step S202: If the result of the determination in step S201 is that there is a camera setting, the control unit 115 changes the current camera setting of the photographing unit 110 based on the camera setting, and the camera setting information after the change. (Camera setting information made by the user) as camera operation information,
It is supplied to the recording unit 116.

Step S203: After the processing of step S202 or when it is determined in step S201 that there is no camera setting, the control unit 115 determines whether the recording button (not shown) provided on the camera operation unit 113 is turned on by the user. Determine whether or not. As a result of this determination, the recording button O
If it is N, the process proceeds to the next step S204, and if it is not the recording button ON, the standby state is set and the step S204 is performed again.
Return to 201.

Step S204: When the record button is turned on as a result of the determination in step S203, the control unit 115
Sets the photographing unit 110 in a recording state (recording mode) and supplies the recording start time information at this time to the recording unit 116 as camera operation information. According to the recording mode by the control unit 115, the image pickup unit 111 starts image pickup of the subject and outputs the video data obtained by this. The image compression unit 112 compresses and encodes the video data from the imaging unit 111 and supplies the video data to the recording unit 116.

Step S205: The control unit 115 determines whether or not the user has performed a camera operation on the image pickup unit 111 in the camera operation unit 113. The camera operation here means, for example, operations such as ON / OFF of the zoom button, panning, and tilting. As a result of this determination, if there is a camera operation, the process proceeds to the next step S206, and if there is no camera operation, the next step S206 is skipped and the process proceeds to step S207 described later.

Step S206: If the result of determination in step S205 is that there is a camera operation, the control section 115 determines the image pickup section 111 based on the sensor information from the sensor section 114.
Drive control (camera control). The information of the sensor here indicates, for example, focal length and focusing distance in the image pickup unit 111, and result information such as camera shake detection. Specifically, for example, when the control unit 115 determines that a zoom operation has been performed by the user, the lens (non-operation) provided in the imaging unit 111 only in a section in which the zoom operation is performed (a section in which the zoom button is pressed). Zooming is performed by driving and controlling (illustrated), and operation control for focusing is performed in the image pickup unit 111 based on sensor information such as a focal length and a focusing distance obtained by the sensor unit 113. At this time, the control unit 115 supplies the sensor information (information such as the focal length and the focus distance) of the sensor unit 113 to the recording unit 116 as camera operation information.

The recording unit 116 stores the camera operation information (step S20) supplied from the control unit 115 as described above.
2, S204, and S206), and the image compression unit 11
The video data after compression and encoding supplied from No. 2 (see step S204) are multiplexed and recorded in the storage medium 120. As the multiplexing method here, for example, MPEG
-2 method can be applied, and video data and camera operation information can be obtained by PES.
(Packetized Elementary Stream) and packetize these into TS (Transport Stream) and PS (Program Str
eam) multiple stream.

Step S207: After the processing of step S206 or when it is determined in step S205 that there is no camera operation, the control section 115 turns off the recording button (not shown) provided on the camera operation section 113 by the user.
It is determined whether it has been done. As a result of this determination, when the recording button is OFF, the standby state is recognized and the process returns to step S201. When the recording button is not OFF, the shooting state is recognized and the process returns to step S204. Return.

Here, in the present embodiment, as described above, the information obtained by the camera operating section 113 and the information obtained by the sensor section 114 are used as the camera operating information. As the format of the camera operation information, for example, the DV format as shown in FIG. 6 or FIG.
Format formats such as those shown in can be applied.
It should be noted that FIG. 6 and FIG.
Only part of the format is shown.

(2) Operation of the moving picture editing section 130 The operation of the moving picture editing section 130 is shown, for example, by the flowchart of FIG.

Step S301: The reading unit 131 reads basic data of the multiplexed data (multiplexed data of video data and camera operation information) to be processed from the recording medium 120. Here, the recording medium 120 includes the image capturing unit 110.
The video data obtained by
11, the recording button (not shown) is recorded as file data for each period from when the user turns it on until it is turned off, or as the same file data for a plurality of clips. Good. Here, for simplicity of explanation, the recording medium 120 is
It is assumed that the video data obtained by the photographing unit 110 is recorded as file data for each clip. Also,
The basic data is I indicating the file of the target video data.
The data includes D, information indicating the length of the file (for example, the number of frames), thumbnail image data at the beginning of the target video data, and the like, and is used for display on the user interface described later.

Step S302, Step S303: The display unit 132 displays the operation screen 40 as shown in FIG. 9, for example.
Display 0. The operation screen 400 is a UI (user interface) implemented by the display unit 132 and the user operation unit 134, the details of which will be described later. The user
A desired clip and scene are designated by a UI (User Interface) 400. The scene here refers to a section desired to be edited by the user in the video to be edited, and is a minimum unit at the time of editing. The information on the scene designated by the user is generated as the scene information as shown in FIG. 10 by the user operation unit 134, for example.
The scene information is table data including a frame ID indicating a start point and an end point of a scene (section) specified by the user, which corresponds to an ID (clip ID) indicating a clip specified (selected) by the user. . In addition, since the scene information is managed in order of scenes, the order of scenes to be played back as an edited result is from the beginning of the scene information.

Step S304: The reading unit 131 acquires camera operation information corresponding to the scene designated by the user in steps S302 and S303 from the multiplexed data read in step S301.

Step S305: Scene rule matching unit 1
As will be described later in detail, the reference numeral 33 collates the camera operation information corresponding to the scene information with the scene rule information stored in the scene rule storage unit 135. The scene rule here is a rule regarding connection of scenes for obtaining a video effect in video editing. That is, the scene rule is a rule for obtaining a good edited image so as to prevent a restless image or a monotonous image without sharpness. For example, "A scene that ends with a pan and a scene that starts with a pan
Both breads must have the same orientation. The rule (1) "," the rule "2" and "the scene ending with tilt and the scene starting with tilt must be aligned in the same direction." If this scene rule is violated, the image will be restless.

Step S306: Scene rule matching unit 1
33 determines whether or not the camera operation information corresponding to the scene information violates the scene rule of the scene rule storage unit 135 based on the result of the collation in step S305.
If the result of this determination is that the scene rules have been violated, the processing proceeds to step S307, and if the scene rules have not been violated, the processing proceeds to step S311.

Step S307: If the result of the determination in step S306 is that the scene information violates the scene rule, the scene rule matching unit 133 causes the scene to be edited indicated by the scene information (the user in the video to be edited For the joint of the specified edit desired section),
By adding a transition effect (wipe, fade-in / out, dissolve, etc.), processing to hide the unsightliness of the joints in the scene is performed. Then, it progresses to step S308 mentioned later.

Step S311: As a result of the determination in step S306, when the scene information does not violate the scene rule, the scene rule collating unit 133 has already added the transition effect to the scene to be edited indicated by the scene information. If the result of this determination is that a transition effect has already been added, the process of the next step S312 is executed, and if not, the process proceeds directly to step S308 described later.

Step S312: If the result of determination in step S311 is that a transition effect has already been added to the scene to be edited, which is indicated by the scene information, the scene rule matching unit 133 performs processing for removing the transition effect. Run. This corresponds to a case where a scene part that was previously against the scene rule is not against the scene rule as the user proceeds with the editing work. After that, the next step S
Proceed to 308.

Steps S308 and S309: The user uses the UI (user interface) 400 shown in FIG. 9 to specify the next scene or another clip for continuing the editing work. The user operation unit 134 determines whether a scene has been designated (step S308) or another clip has been designated by the user's operation through the UI (user interface) 400 (step S30).
9). As a result of these determinations, if a scene is designated, the process returns to step S303 again, while if another clip is designated, the process returns to step S302.

As described above, in the moving image editing section 130, a series of operations from step S301 to step S309 is executed until all the necessary scenes designated by the user have been designated and the final scene information is completed. It And
In the video as the editing result, scenes are sequentially reproduced based on the final scene information.

When a plurality of scene rules are applied in the moving image editing section 130, appropriate scene rules may be used based on the priorities given to these scene rules in advance.

<Movie Editing Unit 130: UI (User Interface) 400> In the movie editing unit 130, various operations by the user (operations such as scene designation) are performed by the display unit 132 and the user operation unit 134. UI (user interface) as shown in FIG. 9 above
400 is used. Here, the UI (user
The interface 400 will be specifically described.

UI (User Interface) 400
Is composed of an edit work window as an area for performing edit work. Also, in the UI (user interface) 400, in the playback window 410, a clip selected by the user (a scene section from when a recording button (not shown) in the imaging unit 111 is turned on to when it is turned off) and an image of an edited result are displayed. It is possible to check by playing. In the clip window 420, the recording medium 12
A list of the thumbnail images at the beginning of each clip constituting the file data (video data) stored in 0 is displayed. The operation unit 430 includes operation buttons such as “play”, “pause”, “stop”, “rewind”, and “fast forward” from the left side thereof, and the user operates the operation unit 430 on the reproduction window 410. It is possible to operate the image to be reproduced. In the scene designation window 440, the scenes of the clip selected by the user are displayed side by side, and the scene section to be adopted can be designated. By operating the scene designation bar 450, it is possible to designate the start point and end point of the clip selected by the user (see the hatched portion in FIG. 9 above).

The operation of the clip start point and end point by the scene designation bar 450 will be specifically described. For example, when the user sequentially selects thumbnail images of desired clips from the clip window 420, these selected thumbnails are selected. The images are dragged and dropped to the scene designation window 440 in the order in which they are to be reproduced, and the start and end points are modified using the scene designation bar 450. At this time, the user can confirm the editing at any time by playing back the video of the correction result in the playing window 410 as needed.

It should be noted that thumbnail images are superposed on each clip so that the user can recognize them.

<Movie Editing Section 130: Collation Processing of Camera Operation Information Corresponding to Scene Information and Scene Rule> Here, step S305 and step S shown in FIG.
In 306, the collation process of the camera operation information corresponding to the scene information as shown in FIG. 10 and the scene rule, which is executed by the scene collating unit 133, will be specifically described. FIG. 11 shows steps S305 and S306.
6 is a flowchart showing in more detail the processing of FIG. Here, for the sake of simplicity of explanation, as a scene rule, the rule that “both the pan direction ends in the pan and the pan start scene in the pan must be aligned” as described above. 1) is used, but the present invention is not limited to this.

The following steps S501 to S51
The process No. 1 is sequentially executed for all the scenes indicated by the clip IDs set in the scene information as shown in FIG.

Step S501: The target scene in the scene information and the camera operation information corresponding thereto are acquired.
For example, in the scene information shown in FIG. 10 above, when the scene with the first clip ID = 1 is the target scene, information regarding this scene is extracted, and from the frame ID = 1, which is the clip start point, to the last point of the clip. The camera operation information existing within a certain frame ID = 300 is read. The camera operation information here is pan information. Then, in which section of the scene of frame ID = 1 to 300, the pan information as the camera operation information is stored in an internal memory or the like (not shown). The retained information at this time is retained until a series of processing is completed.

Step S502: For the target scene,
It is determined whether or not there is the previous scene. The existence of the previous scene here means "is another clip ID set immediately before the target scene currently read?" In the scene information. Therefore, when the first process is executed (in the scene information of FIG. 10, when the process for the scene with the clip ID = 1) is executed, the determination result is that there is no previous scene. If the result of this determination is that there is a previous scene, processing advances to the next step S503, and otherwise, processing advances to step S509, which will be described later.

Step S503: If the result of the determination in step S502 is that there is a scene immediately preceding the target scene, it is determined whether or not the scene has ended in pan. An example of this determination method is a method in which camera operation information (pan information) corresponding to the scene is acquired from the held information in step S501, and the determination is made from the acquired information. If the result of this determination is that the previous scene has ended with a pan, the operation proceeds to the next step S504,
If not, the process proceeds to step S509 described below.

Step S504: If the result of the determination in step S503 is that the scene ends one pan before the target scene, it is determined whether the length of the pan is greater than or equal to a predetermined length. The determination here indicates that "this rule is not applied if the bread is a bread that is too short and is not noticeable to human eyes."
The predetermined length of the bread may be a result obtained by an experiment or the like, and is not particularly limited, but for example, about 10 frames is desirable. If the result of this determination is that the previous scene has ended with a pan having a predetermined length, processing advances to the next step S505, and if not, processing advances to step S509 described below.

Step S505: If the result of the determination in step S504 is that the previous scene ends with pan having a predetermined length, it is determined whether or not the target scene starts with pan. The determination here can be performed using the camera operation information (pan information) held in step S501. As a result of this determination, if the target scene starts with pan, the process proceeds to the next step S506, and if not, the process proceeds to step S509 described later.

Step S506: If the result of the determination in step S505 is that the target scene starts with pan, it is determined whether or not the pan length is equal to or longer than a predetermined length.
The determination here is executed for the same reason as in step S504. As a result of this determination, if the target scene is a pan having a predetermined length, the process proceeds to the next step S507, and if not, the process proceeds to step S509 described later.

Step S507: If the result of the determination in step S506 is that the target scene is a pan having a predetermined length, it is determined whether or not the directions of the front and rear pans are different. For example, when the ending pan of the scene immediately before the target scene is a pan from right to left, it is determined whether the pan of the target scene is a pan from left to right. As a result of this determination, if this is the case (when both pan directions are different), this rule is violated, and therefore the process proceeds to the next step S508.

Step S508: If the result of the determination in step S507 is that the pan direction of the target scene is different from the pan direction of the scene immediately before it, details will be described later.
For example, as shown in FIG. 12, a transition effect 603 having a predetermined length is added to the joint between the target scene 601 and the scene 602 immediately before it. As a result, a hard-to-see part is hidden and becomes inconspicuous, and a good image is obtained. Then, it progresses to step S511 mentioned later.

Step S509: On the other hand, in any one of the determinations of Steps S502 to S507 described above, if the determination is “NO”, it is recognized that this rule is satisfied, and the target scene is It is determined whether or not a transition effect has already been added in the editing process of. If the result of this determination is that a transition effect has already been added to the target scene, the process proceeds to the next step S510, and if not, the process proceeds directly to step S511 described below.

Step S510: If the result of determination in step S509 is that a transition effect has already been added to the target scene, the transition effect will be removed from the target scene, and details will be described later, and the flow advances to step S511.

Step S511: It is determined whether or not the processing has been completed for all the scenes in the scene information, and if there is a scene that has not been processed yet, the processing for that scene is executed again. Step S501
Return to. On the other hand, when the processing is completed for all scenes,
This processing ends.

<Movie Editing Section 130: Transition Effect Adding Process> Here, step S307 shown in FIG. 8 above, specifically step S5 shown in FIG. 11 above.
In 08, the transition effect adding process executed by the scene rule matching unit 133 will be described.

First, as shown in FIG.
The transition effect 603 is the target scene 602.
Is added to the scene 601 immediately before that. The transition effect 603 here is a transition effect of a predetermined length, and the predetermined length is sufficient as long as it is possible to reduce the difficulty of seeing the joints of the scene, and is particularly specified. However, it is set to 60 frames, for example.

The order effect of the transition effect 603, the target scene 602, and the scene 601 immediately before it is the relationship shown in FIG. 12 (b), and is reproduced in the direction of the arrow in FIG. Apparently, it looks as if the transition effect 603 is added as shown in FIG.

For example, in the target video data having the scene information as shown in FIG. 10, clip ID =
The scene indicated by 2 is the target scene (the third scene), the scene immediately before it is the scene indicated by the clip ID = 4 (the second scene), and the transition effect between them is When added as shown in FIG. 12, the scene information becomes information as shown in FIG.

In the scene information shown in FIG. 13, "clip ID = 10000" is a transition effect clip, and the transition effect clip ID is "10000" here.
A special clip ID is given. This makes it possible to distinguish which clip ID is the transition effect.

As can be seen from the comparison between the scene information in FIG. 10 and the scene information after the transition effect is added in FIG. 13, the target scene (clip ID =
In the state where the end point of the scene immediately before the second scene (scene of clip ID = 4) is shortened by 30 frames and the start point of the target scene (scene of clip ID = 2) is shortened by 30 frames. , Clip ID = 10 between these
000, a transition effect is inserted.

Note that the transition effect clip can be generated by rendering using the frames of the last 30 frames of the scene immediately before the target scene and the first 30 frames of the target scene.

In step S312 shown in FIG. 8, specifically step S510 shown in FIG. 11,
In the transition effect removing process executed by the scene rule checking unit 133, an operation reverse to the above-described transition effect applying process is performed. As a result, the scene information is returned from the state shown in FIG. 13 to the state shown in FIG. Also, the removal process of the transition effect is executed when the transition effect has already been added in the previous process. As the ID, the special I
Since D is used, it can be easily performed by searching for the special ID.

[Second Embodiment] In the first embodiment, in the image editing apparatus 100 shown in FIG. 1, the camera operation information is read from the camera operation unit 113 and the sensor unit 114 at the time of image capturing by the image capturing unit 110. Configured to get. In the present embodiment, it is assumed that there is no means for obtaining camera operation information during shooting. Specifically, for example, in the first embodiment, the image capturing unit 110 is configured by a recording type video camera on a disc, and the moving image editing unit is a PC (and a moving image) that has a drive for reading recording information on the disc. Editing application). On the other hand, in the present embodiment, for example, it is configured by only the PC (application above), and the video capturing means is configured to connect the video camera to the PC by IEEE1394 or the like.

That is, in this embodiment, as shown in FIG. 14, the video camera 150 (10) and the VTR 150 are used.
The video interface (I / F) 150 (9) capable of capturing the image obtained in (12) is provided.

The video camera 150 (10), VTR
The 150 (12) and the external storage device 150 (6) may be arranged on a network to exchange data and the like by communication via the NIC 150 (8).

FIG. 15 is a functional block diagram of editing apparatus 150 in the present embodiment. That is, in addition to the configuration shown in FIG. 4, the moving image editing unit 150 of this embodiment has a moving image input unit 156 (0) and a frame extracting unit 156.
(1), cut detection unit 156 (2), moving image / index storage unit 156 (3), and camera operation determination unit 156.
(4) is provided.

The moving image input section 156 (0) has a video I / F.
The moving image (video) data obtained by the video camera 150 (10) or the VTR 150 (12) is read out via 115 (9) and taken into the moving image editing section 150. The frame extraction unit 156 (1) sequentially extracts the frames (frame images) constituting the video data captured by the moving image input unit 156 (0). Cut detection unit 156
(2) detects the cut point in the video from the frame image extracted by the frame extraction unit 156 (1).
A storage device such as the external storage device 150 (6) can be applied to the moving image / index storage unit 156 (3), and the frame image extracted by the frame extraction unit 156 (1) and camera operation information described later can be used. Etc. are accumulated.

The reading unit 151 reads data at an arbitrary position from the moving image / index storage unit 150 (3).
The camera operation determination unit 156 (4) performs image processing on the video data read by the reading unit 151 and determines camera operation information from the processed data.

Similar to the first embodiment, the display unit 152 displays to the user the status of editing the target video, the editing result, and the like. The scene rule matching unit 153 is the first
Similar to the embodiment described above, the scene rules stored in the scene rule storage unit 155 and the camera operation determination unit 156.
Based on the result of collation with the camera operation information obtained in (4), the transition effect is added to the scene joint. As in the first embodiment, the user operation unit 154 accepts an operation from the user necessary for video editing, such as designation of a section of a scene.

FIG. 16 shows the moving picture editing section 15 of this embodiment.
The whole operation of 0 is shown by a flowchart.

Step S701: In the image editing section 150 shown in FIG. 15, the moving image input section 156 (0)
Import video data. Frame extraction unit 156 (1)
Sequentially extracts the frame images constituting the video data captured by the moving image input unit 156 (0). The cut detection unit 156 (2) detects a cut point in the video from the frame image extracted by the frame extraction unit 156 (1). The moving image / index storage unit 156 (3) stores the frame image extracted by the frame extraction unit 156 (1).

Step S702: The reading unit 151 reads the video data to be edited from the moving image / index storage unit 156 (3). Camera operation determination unit 156 (4)
Performs image processing on the target video data obtained by the reading unit 151, determines the camera operation from the target image data, and outputs the information (camera operation information) of the determination result.
The moving image / index storage unit 156 (3) is stored.

Step S703: Reading unit 151, display unit 152, scene rule matching unit 153, user operation unit 15
4 and the scene rule storage unit 155 respectively operate in the same manner as in the first embodiment to execute the editing process of the target video data.

FIG. 17 shows the step S shown in FIG.
701 specifically shows the moving image capturing process of 701.

Step S721: Movie input section 156
(0) reads the moving image (video) data obtained by the video camera 115 (10) or the VTR 115 (12) via the video I / F 115 (9) and loads it into the moving image editing unit 150. Note that the video capture destination of the moving image input unit 156 (0) is not limited to the video camera 115 (10) or the VTR 115 (12), and may be, for example, a recording medium as in the first embodiment. It may be 120.

Step S722: Frame extraction unit 156
In (1), frames (frame images) constituting the video data captured by the moving image input unit 156 (0) are sequentially extracted and supplied to the cut detection unit 156 (2).

Step S723: Cut detection unit 156
(2) assigns a frame ID to each of the frame images extracted by the frame extraction unit 156 (1), and stores the frame image after the frame ID is assigned in the moving image / index storage unit 156 (3). The frame ID here is an ID starting from "1" for each clip unit.

Step S724: Cut detection unit 156
(2) detects the degree of similarity between adjacent frames in the frame image stored in the moving image / index storage unit 156 (3). To give an example of the method of detecting the similarity between adjacent frames, first, the similarity distance between frames is calculated, and assuming that this is 100% and completely the same, the percentage expressed as a “similarity”. . The algorithm for calculating the similarity distance between frames is not particularly limited, but an example thereof is, for example, as illustrated in FIG. 18, each frame (two adjacent frames) 800 to be compared is vertically and horizontally arranged. Is divided into a plurality of blocks, the average value of RGB is calculated for each block, and the R of the corresponding blocks to be compared is calculated.
The sum of squares of the differences between the GB channels is obtained and used as the inter-frame similarity distance. As for the inter-frame similarity distance, for example, the smaller the value is, the more similar the two frames are, and the larger the value is, the less similar the distances are. The similarity distance between frames is

[0121]

[Equation 1]

It is represented by the following (formula 1). The inter-frame similarity distance is converted into a similarity and used.

Step S725: Cut detector 156
In (2), it is determined whether or not the similarity between the adjacent frames obtained in step S724 is equal to or less than the predetermined value H. By this judgment, the transition point (cut point) of the clip will be detected. Here, the transition point (cut point) of the clip will be detected.
Then, since the frame image changes abruptly, the similarity between the frames before and after becomes low. That is, the predetermined value H is the lower limit value of the similarity that can be regarded as not a cut point. The predetermined value H is, for example, a value that can be obtained by an experiment, and is not particularly limited, but is, for example, 8
About 0% is desirable. As a result of the determination in this step S725, if the similarity between adjacent frames is less than or equal to the predetermined value H, the process proceeds to step S726, and if the similarity between adjacent frames is not less than or equal to the predetermined value H, the process proceeds to step S728.

Step S726: If the result of the determination in step S725 is that the similarity between adjacent frames is less than or equal to the predetermined value H, the cut detection unit 156 (2) recognizes that the frame image currently being processed is the cut point. Then, in order to use this as an index, the frame image is reduced to generate a thumbnail image. Step S727: The cut detection unit 156 (2) uses the thumbnail image generated in step S726 as index information, for example, as shown in FIG. 19, clip IDs and clip lengths (frames) sequentially assigned from the first clip. Number) and information consisting of thumbnail images,
It is stored in the moving image / index storage unit 156 (3).
Then, it progresses to the following step S728.

Step S728: Video input section 156
If the capture process of the video data to be captured in (0) is not yet completed, the process from step S721 is repeatedly executed again, and if the capture process is completed,
This processing ends.

FIG. 20 shows the step S shown in FIG.
7 specifically shows the camera operation information determination process 702.

Step S731: In order to execute the processing for all the clips in the information as shown in FIG. 19, the reading unit 151 sets the initial value "1" for the index N indicating the clip to be processed. "Set. Step S732: The reading unit 151 extracts the frame group corresponding to the clip having the ID of the index N from the moving image / index storage unit 156 (3).

Step S733: Camera operation determination unit 15
6 (4) determines the operation of the camera from the data of the frame group extracted by the reading unit 151. As the determination method here, for example, the vanishing point of the motion vector is obtained by using the Hough transform, and the vanishing point having the largest number of votes among the plural vanishing points is set as the vanishing point with respect to the background, and is related to the camera operation. It is possible to apply a method such as preaching a restraint formula. The relationship between the camera operation and the vanishing point, and the method for obtaining the camera operation are described in detail, for example, in “Kenichi Kanaya: Image Understanding (Morikita Publishing)”. Step S734: The camera operation determination unit 156 (4)
The determination result of the camera operation acquired in step S733 is
For example, the moving image / index storage unit 156 (3) may be stored in the footer area of the file by associating it with the corresponding frame ID as the camera operation information in the format shown in FIGS. 6 and 7 above. ).

Steps S735 and S736: If there is a clip that has not been processed yet in the information as shown in FIG. 19 (step S73).
5) The index N is incremented (step S736), the processing from step S732 is executed again, and when the processing for all clips is completed, this processing ends.

Incidentally, in the second embodiment, the above-mentioned FIG.
The moving image editing process of step S703 shown in FIG. 6 is the same as the moving image editing process in the first embodiment, and thus detailed description thereof will be omitted.

Further, in the first and second embodiments, as shown in FIG. 1, the photographing section 110 and the moving picture editing section 130 are separately provided, but the present invention is not limited to this. For example, by providing the image capturing unit 110 with sufficient computing capability, or by providing the moving image editing unit 130 with a photographing function including a lens for photographing,
It may be one component (device).

Further, in the first and second embodiments, the scene rule to be applied is not limited to the above-mentioned rule (1), and various rules such as a rule regarding tilt are also applied. It is possible.

Further, in the first and second embodiments, for example, in order to speed up the collation process of the scene information and the scene rule, the camera operation information given in frame units is previously shown in FIG. As shown, the scene section indicated by the start point and the end point, the type of camera operation information, and the parameter of the camera operation information may be collectively managed for each scene section.

Also, in the first and second embodiments, the frame ID is used to indicate the position of the frame.
For example, it may be a time code. That is, any frame may be used as long as the position of the frame in the target moving image can be specified.

Further, in the first embodiment, the operation of the photographing section 110 is shown by the flow chart of FIG. 5, but the operation is not limited to this. For example, the operation may be such that the setting of the camera can be changed during image capturing.

Further, in the first and second embodiments, the length of the transition effect is fixed, but the length of the transition effect is dynamically set, for example, 60% of the bread length. May be determined.

Further, in the first and second embodiments, the transition effect is configured to be applied so as to extend over two scenes, but the present invention is not limited to this, and the purpose is to hide an image that is difficult to see. It is also possible to adopt a configuration in which the transition effect is added to only one of the scenes as long as it can be achieved.

Further, in the first embodiment, the video is stored in clip units. In addition, in the second embodiment, the video is divided by the cut detection unit 156 (2) into clip units. This is in consideration of operability when the user specifies a scene. Therefore, if the decrease in operability is tolerated, it is not always necessary to divide the video clips into units, for example, in the first embodiment, all clips are saved as one file, and in the second embodiment, cuts Detector 156
It is possible to omit the process of (2).

Further, in the first and second embodiments, the format format of the camera operation information is configured to use the format format shown in FIG. 6 or FIG. 7, but the format is not limited to this. However, any format can be applied as long as it includes information such as the type of camera operation information, the existing section, and direction, and information such as strength as necessary. .

In the first and second embodiments, when the scene rule is violated, for example, in the UI (user interface) 400 as shown in FIG. 9, the scene corresponding to the scene designation window 440 is displayed. The message window may be displayed together with the message to prompt the user to press a confirmation button (not shown) at any time. This allows the user to easily confirm which scene corresponds to which scene and which these scene rules are violated. However, the present invention is not limited to such a configuration, and other display methods may be used as long as the display configuration is for the purpose of preventing the user from not knowing which scene corresponds to which scene when displayed at one time. It is also possible to use.

The present invention can also be applied to a system composed of a plurality of devices such as a host computer, an interface device, a reader, and a printer, for example, a device composed of one device (for example, a copying machine or a facsimile). It is also possible to apply to a device such as a device).

Further, an object of the present invention is to supply a storage medium storing a program code of software for realizing the functions of the host and the terminal of the first and second embodiments to a system or an apparatus, and the system or the apparatus. It is needless to say that it is also achieved by the computer (or CPU or MPU) of the device reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the first and second embodiments, and the storage medium storing the program code and the program code constitute the present invention. Becomes A ROM, flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, non-volatile memory card, or the like can be used as a storage medium for supplying the program code.

Further, by executing the program read by the computer, not only the functions of the first and second embodiments are realized, but also the computer operates on the basis of the instructions of the program code. It goes without saying that a case where an OS (operating system) or the like executes a part or all of the actual processing and the functions of the first and second embodiments are realized by the processing is also included. Further, after the program code read from the storage medium is written in the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. C on board and function expansion unit
Needless to say, this also includes the case where the PU or the like executes a part or all of the actual processing and the functions of the first and second embodiments are realized by the processing.

FIG. 22 shows the function 900 of the computer.
Is shown. As shown in FIG. 22, the computer function 900 includes a CPU 901 and a ROM 902.
A RAM 903, a keyboard controller (KBC) 905 of a keyboard (KB) 909, a CRT controller (CRTC) 906 of a CRT display (CRT) 910 as a display unit, and a hard disk (HD).
911 and a flexible disk (FD) 912 disk controller (DKC) 907, and a network 9
A network interface controller (NIC) 908 for connection with the
It is configured to be communicably connected to each other via.

The CPU 901 is a ROM 902 or HD.
By executing the software stored in 911 or the software supplied from the FD 912, each component connected to the system bus 904 is comprehensively controlled.
That is, the CPU 901 executes the processing program according to the predetermined processing sequence in the ROM 902 or the HD 91.
1 or by reading from FD912 and executing,
Control for realizing the operation in the first and second embodiments is performed.

The RAM 903 functions as the main memory or work area of the CPU 901. KBC905
Controls instruction input from the KB 909, a pointing device (not shown), or the like. CRTC906 is C
Control the display of RT910. The DKC907 is a boot program, various applications, edit files,
It controls access to the HD 911 and the FD 912 that store the user file, the network management program, and the predetermined processing program according to the present embodiment. N
The IC 908 bidirectionally exchanges data with a device or system on the network 920.

[0147]

As described above, according to the present invention, the image data obtained by the operation on the arbitrary device or system (the image data obtained by the operation by the video camera, etc.) Information about operations (operation information)
Based on the above, predetermined image processing (special effect processing at the time of image editing, etc.) is performed. As a result, appropriate image processing is performed according to the operation performed when the target image is acquired, and the user can easily obtain a good processed image without requiring specialized knowledge in handling the image. be able to.

Specifically, for example, according to the present invention, a plurality of sections designated by the user are sequentially recorded on moving image data (video data) obtained by photographing a subject with a video camera, When applied to a device that uses as an edited result, the matching process (check) between the rules of expression combination by video camera operation and the actual video camera operation
Since the transition effects can be automatically eliminated and the unnaturalness of the joints in the scene can be automatically eliminated, the user does not need specialized knowledge about image editing, and the amateur Even in this case, it is possible to easily obtain a high-level editing result. Further, since it is possible to check with the rule at any time during image editing, the user can proceed with image editing work in a free process.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an image editing apparatus to which the present invention has been applied in a first embodiment.

FIG. 2 is a block diagram showing a configuration of a photographing unit of the image editing apparatus.

FIG. 3 is a block diagram showing a configuration of a moving image editing section of the image editing apparatus.

FIG. 4 is a block diagram showing a functional configuration of the moving image editing unit.

FIG. 5 is a flowchart for explaining the operation of the image capturing unit.

FIG. 6 is a diagram for explaining an example of a format format of camera operation information obtained by the image capturing unit.

FIG. 7 is a diagram for explaining another example of the format format of the camera operation information.

FIG. 8 is a flowchart for explaining the operation of the moving image editing unit.

FIG. 9 is a diagram illustrating a user interface of the moving image editing unit.

FIG. 10 is a diagram for explaining scene information obtained by the image capturing unit.

FIG. 11 is a flowchart for explaining a process of collating scene information with a scene rule in the operation of the moving image editing unit.

FIG. 12 is a diagram for explaining a transition effect added to a scene in the matching process of the scene information and the scene rule.

FIG. 13 is a diagram for describing scene information after the transition effect is added.

FIG. 14 is a block diagram showing a configuration of the control unit in the second embodiment.

FIG. 15 is a block diagram showing a configuration of the moving image editing unit in the second embodiment.

FIG. 16 is a flowchart for explaining an overall operation of the moving image editing unit.

FIG. 17 is a flowchart for explaining a motion capturing process in the overall motion of the moving image editing unit.

FIG. 18 is a diagram for explaining an example of a method of detecting the degree of similarity between adjacent frames (detection of a cut point) in the operation capturing process.

FIG. 19 is a diagram for explaining an example of result information (index information) of the detection processing.

FIG. 20 is a flowchart for explaining camera operation information determination processing in the overall operation of the moving image editing unit.

FIG. 21 is a diagram for explaining another example of the camera operation information.

FIG. 22 is a block diagram showing a configuration of a computer that reads a program for causing a computer to realize the functions of the image editing apparatus from a computer-readable storage medium and executes the program.

FIG. 23 is a block diagram showing a configuration of a control unit of the photographing unit.

[Explanation of symbols]

100 image editing device 110 Imaging Department 111 Imaging unit 112 Image compression unit 113 Camera control section 114 sensor 115 control unit 116 recording section 120 recording media 130 Video Editing Department 131 Reader 132 display 133 Scene rule matching unit 134 User Operation Unit 135 Scene rule storage

Claims (40)

[Claims] 1. An image data obtained by an operation on an arbitrary device or a system, an acquisition means for acquiring information on the operation, and an operation means obtained on the basis of the operation information obtained by the acquisition means. An image processing apparatus, comprising: a processing unit that performs predetermined image processing on the obtained image data. 2. The arbitrary device or system has a function of acquiring moving image data by a shooting operation according to an operation made by a user, and the operation information includes information on the operation at the time of shooting. The image processing apparatus according to claim 1, wherein: 3. The arbitrary device or system includes a camera, and the operation information includes movement of the camera when acquiring the image data, a state of the camera, a type of operation of the camera, a direction of the camera, The image processing apparatus according to claim 1, further comprising information on at least one of a section in which the camera is operated. 4. The image processing apparatus according to claim 1, wherein the acquisition unit reads and acquires the image data and the operation information from an arbitrary recording medium. 5. The predetermined image processing includes special effect processing for the image data.
The image processing device described. 6. The image processing apparatus according to claim 5, wherein the special effect process includes a process of adding a transition effect to the image data. 7. The specifying means is provided for specifying a plurality of arbitrary sections among a plurality of sections forming the moving image data obtained as the image data by the obtaining means, and the processing means is obtained by the obtaining means. The image processing apparatus according to claim 1, wherein predetermined image processing is performed on the moving image data based on operation information corresponding to moving image data of a plurality of sections designated by the designating means. . 8. The processing means is based on a rule regarding a combination of operations on the arbitrary device or system,
The image processing apparatus according to claim 1, wherein predetermined image processing is performed on the image data. 9. The processing means sets a predetermined image for the image data based on a comparison result between a rule regarding a combination of the operations and information about an operation actually performed on the arbitrary device or system. The image processing apparatus according to claim 8, wherein the image processing apparatus performs processing. 10. An image processing device for moving image data obtained by an operation on an arbitrary device or system, wherein the obtaining means obtains information on the moving image data and the operation, and is obtained by the obtaining means. Specifying means for specifying any of a plurality of sections forming the moving image data and the order of the plurality of sections, and specifying by the specifying means by the operation information acquired by the acquiring means. Detection means for detecting operation information corresponding to the selected section, storage means for storing rules relating to operation on the arbitrary device or system, operation information detected by the detection means, and the storage means stored in the storage means. An image comprising a processing means for applying a transition effect to the moving image data obtained by the obtaining means, based on the result of matching the rules. Management apparatus. 11. The image processing apparatus according to claim 10, wherein the storage unit stores at least the existence section of the operation and a rule regarding a time-series arrangement thereof. 12. The processing means is characterized in that when the operation information detected by the detecting means is information about an operation in a section shorter than a predetermined length, it is not targeted for the collation. The image processing apparatus according to claim 10. 13. The image processing apparatus according to claim 10, wherein the processing means executes the processing based on the collation result at the timing of section designation by the designating means. 14. The processing means adds a transition effect prepared as moving image data different from the moving image data obtained by the obtaining means, to the moving image data obtained by the obtaining means. The image processing apparatus according to claim 10, wherein: 15. The processing means adds a transition effect to the moving image data when the operation based on the operation information violates the rule based on the collation result. Image processing device. 16. The moving image if the operation according to the operation information does not violate the rule based on the collation result and the transition effect has already been added to the moving image data. 11. The image processing apparatus according to claim 10, wherein the transition effect is removed from the data. 17. The image processing apparatus according to claim 10, wherein the transition effect has a fixed length or a length according to the operation information. 18. An image processing system comprising a plurality of devices communicably connected to each other, wherein at least one of the plurality of devices is an image processing device according to any one of claims 1 to 17. An image processing system having the function of. 19. An image processing method for moving image data obtained by a shooting operation, wherein predetermined image processing is performed on the moving image data based on information related to the shooting operation at the time of obtaining the moving image data. An image processing method comprising a processing step. 20. The image processing method according to claim 19, wherein the predetermined image processing includes special effect processing for the moving image data. 21. A step of designating a plurality of arbitrary sections among a plurality of sections constituting the moving image data is included, and the processing step includes a step of acquiring moving image data of the plurality of sections designated by the specifying step. 20. A step of subjecting the moving image data to predetermined image processing based on information regarding a shooting operation is included.
The described image processing method. 22. The processing step includes a step of performing predetermined image processing on the moving image data based on a result of comparison between a predetermined rule regarding a shooting operation and a shooting operation at the time of acquiring the moving image data. The image processing method according to claim 19, further comprising: 23. An image processing method for moving image data obtained by a video camera, comprising: moving image data obtained by the video camera; and information about operation of the video camera at the time of obtaining the moving image data. It includes a reproducing step of reproducing from an arbitrary recording medium, and a processing step of performing special effect processing on the moving image data reproduced by the reproducing step based on the operation information reproduced by the reproducing step. Image processing method. 24. An image processing method for moving image data obtained by a video camera, comprising: moving image data obtained by the video camera; and information on an operation of the video camera at the time of obtaining the moving image data. A reproducing step of reproducing from an arbitrary recording medium, a specifying step of specifying a plurality of arbitrary sections among a plurality of sections forming the moving image data reproduced by the reproducing step, and a plurality of specifying sections of the specifying step. And a processing step of performing special effect processing on the moving image data reproduced by the reproducing step based on the operation information corresponding to the moving image data of the section. 25. An image processing method for moving image data obtained by a camera operation, comprising a storage step of storing the moving image data and information about the camera operation, and a storage object by the storage step to be processed. A reading step for reading out the moving image data and the camera operation information corresponding thereto, a section specifying step for specifying a plurality of arbitrary sections for the target moving image data read out by the reading step, and a specifying step by the section specifying step. The camera operation information corresponding to the selected section from the camera operation information read by the reading step, a rule storing step of storing a rule regarding camera operation, and a camera operation detected by the detecting step. Information,
An image processing method, comprising: a processing step of applying a transition effect to the moving image data read by the reading step, based on a result of collation with the rule stored by the rule storing step. 26. An image processing method for moving image data obtained by operating a camera, comprising a storing step of storing the moving image data, and a step of acquiring the moving image data from the moving image data stored in the storing step. Extraction step of extracting the camera operation information, a designation step of designating a plurality of arbitrary sections for the moving image data stored in the storage step, and a camera operation corresponding to the section designated by the section designation step. A detection step of detecting information from the camera operation information extracted by the extraction step, a rule storage step of storing a rule regarding camera operation, and camera operation information detected by the detection step,
An image processing method, comprising: a processing step of applying a transition effect to the moving image data stored in the storing step, based on a matching result with the rule stored in the rule storing step. 27. An image processing method for moving image data obtained by photographing with a camera operation, comprising: a photographing step of photographing a subject by camera operation to generate moving image data of the subject; Acquisition step of acquiring setting information and camera status information for the camera as camera operation information, moving image data obtained in the above-mentioned shooting step, and recording for recording the camera operation information obtained in the above-mentioned acquisition step in an arbitrary recording medium Steps, a reading step for reading moving image data to be processed and camera operation information corresponding thereto from the recording medium, and a plurality of arbitrary sections are designated for the target moving image data read by the reading step. Corresponds to the section specification step and the section specified in the section specification step above The camera operation information to be detected from the camera operation information read by the reading step, a rule storing step for storing a rule relating to the camera operation, the camera operation information detected by the detecting step,
An image processing method, comprising: a processing step of applying a transition effect to the moving image data read by the reading step, based on a result of collation with the rule stored by the rule storing step. 28. The image processing method according to claim 25, further comprising an order specifying step of specifying an order of a plurality of sections specified by the section specifying step. 29. The camera operation information includes at least one piece of information about a camera movement at the time of acquiring moving image data, a camera state, a type of camera operation, a section in which the camera operation is performed, and a camera direction. Claim 2 characterized by the above-mentioned.
The image processing method according to any one of 5 to 27. 30. The image processing method according to claim 25, wherein the rule includes at least an existing section of the camera operation information and a time-series arrangement thereof. 31. The processing step includes a step of, when the camera operation indicated by the camera operation information is a camera operation in a section shorter than a predetermined length, not making the camera operation information the target of the collation. The image processing method according to any one of claims 25 to 27, which is characterized in that. 32. The image processing method according to claim 25, wherein the processing step includes a step of executing processing each time a section is specified by a user in the section specifying step. . 33. The transition effect,
28. The image processing method according to claim 25, further comprising a holding step of generating and holding as moving image data different from the moving image data to be processed. 34. According to the result of the collation, if the camera operation indicated by the camera operation information violates the rule, the step of executing the process of adding the transition effect, and the result of the collation include: If the camera operation indicated by the camera operation information does not violate the rules, it is determined whether or not a transition effect has already been added to the moving image data, and as a result of the determination, the transition effect has already been added. 28. The image processing method according to claim 25, further comprising a step of removing a transition effect from the moving image data. 35. The transition effect,
28. The image processing method according to claim 25, wherein the image processing method has a fixed length or a length according to the camera operation information. 36. The image processing method according to claim 25, wherein the detecting step includes a step of previously managing camera operation information for each section. 37. A computer-readable storage medium recording a program for causing a computer to realize the function of the image processing apparatus according to claim 1 or the function of the image processing system according to claim 18. . 38. A computer-readable storage medium recording a program for causing a computer to execute the processing steps of the image processing method according to claim 19. 39. A program for causing a computer to realize the function of the image processing apparatus according to claim 1 or the function of the image processing system according to claim 18. 40. A program for causing a computer to execute the processing steps of the image processing method according to any one of claims 19 to 36.