JP2007013479A - Camera work information application and evaluation device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem wherein it is impossible to easily search a video block photographed by such a photographic operation as "pan" from among photographic object photographed and recorded by various photographic operations such as "pan" or "tilt" by using a video camera, and to present it to a user. <P>SOLUTION: A sensor is loaded on a video camera, and the changing point of a photographic operation is detected by using the information of the sensor, and camera work information is derived for a video block divided by the changing point, and recorded so as to be associated with the pertinent video block. The video block in the photographic raw materials is prescribed by using recorded camera work information such as "pan" so that the video block photographed by such a desired photographic operation as "pan" is presented to the user. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a method for providing information at the time of shooting a moving image such as a television, a movie, and a video, and in particular, acquires information on movement of the camera up and down, left and right using a sensor, and the sensor information is obtained from the corresponding image The present invention relates to a method of giving to a section.

At the time of editing a moving image, there are frequently performed operations such as searching for a target scene, reproducing the scene, and extracting the scene. However, at the time of the conventional moving image editing, in order to find a target scene, it is necessary for the editor to reproduce all the target videos once to find the target scene, which takes a lot of work time.
In order to solve the above-described problem, a technique is disclosed in which a change point of a scene is detected by automatically detecting a change point of a video signal to facilitate a scene cueing operation or the like (see, for example, Patent Document 1). ). Also, in order to be able to search for videos using keywords related to “camera work”, which is information related to camera movement, such as “pan video”, camera work such as “pan” is automatically extracted from video by image processing. (See, for example, Patent Document 2). Also disclosed is a method that makes it easier to select an appropriate video segment during editing by extracting video segments with low quality during shooting, such as camera shake, and appropriate video segments, classifying them, and presenting them to the editor. (For example, see Patent Document 3).
Japanese Patent No. 2863818 Japanese Patent No. 2677312 JP 2000-261757 A

In order to reduce the workload when editing a moving image as in the above prior art, the video is automatically divided and the additional information is added to the divided video sections by adding the additional information. It is necessary to be able to use it for video search, and to automatically evaluate the quality of the video section, classify the video section and present it to the editor.
However, in the method described in Patent Document 1, since change point detection is performed using luminance and color information of a video signal, there is a correlation between camera movement at the time of shooting, that is, camera work, and luminance and color information. If not, the video cannot be divided in units of camera work.

  Further, in the method described in Patent Document 2, since the movement of the camera at the time of shooting is estimated using image processing, in the case of a moving image in which a subject occupying most of the shooting area is moving, the camera is actually Although the camera was shooting while moving the camera, it may not be possible to determine whether the subject was moving. Therefore, in the method described in Patent Document 2, an erroneous camera work may be detected.

  In the method described in Patent Document 3, as in the case of the camera work detection described above, the camera motion is estimated and evaluated using image processing. For example, the subject is slightly shaken. May be evaluated as camera shake. In addition, for each camera work, video shooting sections that are shot still have strict evaluation criteria for camera shake, and video sections that are shot while panning are looser than those that are shot stationary. There is no mention of selecting evaluation criteria.

  In view of the above problems, the present invention divides a video into video sections in units of camera work, detects camera work for each divided video section, and uses camera work information as additional information so that it can be searched later. By recording and associating, and evaluating the video section using an evaluation index that depends on camera work, and recording the evaluation value in relation to the video, the video section can be searched using the camera work information during editing. In addition, the video section is intended to be classified and presented by an evaluation value.

  An information processing apparatus according to claim 1 of the present invention is an information processing apparatus that detects movement caused by movement of an imaging apparatus and provides information at the time of image capturing, an imaging unit that captures an image, and an image frame that is captured Based on the motion information detected by the motion information detecting means, video information giving means for giving a video address based on time series to the motion information detecting means for detecting movement of the imaging means as motion information Change point detecting means for detecting a change in movement as a change point, video address acquiring means for acquiring a video address at the detected change point, and holding means for holding the change point and the video address.

  An information processing apparatus according to claim 2 of the present invention is an information processing apparatus that detects movement caused by movement of an imaging apparatus and provides information at the time of image capturing, an imaging unit that captures an image, and an image captured Video address giving means for giving a video address based on time series to a frame, motion information detecting means for detecting movement of the imaging means as motion information, and the calculation calculated based on the motion information detected by the motion information detecting means Change point detection means for detecting a time point at which the speed of the imaging means becomes a predetermined value or a time point when the speed changes from the value as a change point, video address acquisition means for acquiring a video address at the detected change point, and the change And holding means for holding the video address.

  The information processing apparatus according to claim 3 of the present invention is the configuration of claim 1 or claim 2, and further, based on camera work correspondence information that associates motion information and camera work information, between detected change points Camera work information determining means for determining camera work information from motion information is provided, and the holding means holds the video address between the change points and the camera work information in association with each other.

  The information processing apparatus according to claim 4 of the present invention is the configuration according to claim 3, and further, the motion between the detected change points based on the camera work evaluation index for evaluating the camera work from the motion information and the camera work information. Camera work evaluation means for determining camera work evaluation between change points based on information and camera work information, and the holding means further corresponds to video addresses between corresponding change points, camera work information, and camera work evaluation information Hold on. .

An information processing apparatus according to a fifth aspect of the present invention has the configuration according to the fourth aspect, further comprising the camera work evaluation index corresponding to each camera work.
An information processing apparatus according to a sixth aspect of the present invention is the configuration according to any one of the first to fifth aspects, wherein the change point detecting means calculates a moving speed of the imaging means from motion information. The time point at which the moving speed changes from 0, or the time point when it becomes 0 is taken as the changing point.

An information processing apparatus according to a seventh aspect of the present invention is the configuration according to any one of the first to fifth aspects, wherein the change point detecting means calculates a moving speed of the imaging means from motion information. The time point at which the moving speed is equal to or higher than a predetermined threshold value or the time point when the moving speed is equal to or lower than the threshold value is defined as a change point.
An information processing apparatus according to an eighth aspect of the present invention has the configuration according to the third aspect, further including an information update unit that updates the camera work correspondence information based on information input from the outside.

  An information processing apparatus according to an eleventh aspect of the present invention is an information processing apparatus that detects movement caused by movement of an imaging apparatus and provides information at the time of image capturing. Receiving means for receiving video frames to which a video address based on a sequence is assigned, movement information due to movement of the external imaging device, motion information detecting means for detecting movement of the imaging means as movement information, and the motion information detection Based on the motion information detected by the means, a change point detection means for detecting a change in the movement of the imaging means as a change point, a video address acquisition means for acquiring a video address at the detected change point, and the change point Holding means for holding the video address.

According to the present invention, it is possible to perform cueing playback of a video section in units of camera work.
In addition, since the camera work information is given to the video section divided by the cue point in association with the video, the camera work information can be searched using the search key to identify the corresponding video section. .
Each video section is evaluated by an evaluation index depending on camera work, and an evaluation value is given. Using this evaluation value and camera work information, it is possible to perform video segment classification and presentation processing, such as presenting only video segments with high video quality to the editor.

  Therefore, at the time of editing, it is possible to search using camera work information for video divided in units of camera work, and to classify and present video sections using camera work information and evaluation values. The video section used for editing by the editor can be easily specified, and efficient editing can be performed.

Embodiments of the present invention will be described below with reference to the drawings.
(Embodiment 1)
In this embodiment, a method of adding camera work information to a captured image based on information about the movement of the video camera to the left and right and back and forth at the time of shooting using the video camera Is disclosed. Here, “camera work information” is a name for how to move the video camera when shooting while changing the shooting position and direction of the video camera. This method of moving the video camera is called “shooting operation”. Commonly used camera work information includes “pan” for shooting horizontally while changing the direction of the video camera around the photographer, and “dolly” for shooting while chasing a moving subject. . Note that the camera work information is not limited to “pan” and “dolly”, and the correspondence between the shooting operation and the name is not limited to that given here. The user who operates the video camera or the like may separately define the shooting operation, its name, and their relationship. Also, camera work information may be given to an image shot with the video camera not moved at all. In the present embodiment, “still” is used as camera work information for an image shot with the video camera stationary.

(1. Overall configuration)
FIG. 1 shows the overall configuration of the present embodiment.
The imaging unit 1001 is configured using an image sensor such as a CCD, and performs imaging and outputs video information of the result to the video recording control unit 1002.
The video recording control unit 1002 controls whether to record video information captured by the imaging unit 1001 in the video holding unit 1003. The video obtained from the imaging unit 1001 is recorded in the video holding unit 1003 in response to a recording start instruction from a user or a connected device, and the video recording is ended in response to a recording end instruction from the user or a connected device. In addition, a video address for identifying a video frame is assigned to the video to be recorded, and the video is recorded in the video holding unit 1003. By using this video address, it is possible to specify a specific part of the video, so that any position of the video recorded in the video holding unit 1003 can be cued and played back, or unnecessary parts are deleted It is possible to perform editing and so on. In this embodiment, a time code defined by SMPTE (Society of Motion Picture and Television Engineer), which is one of the standards for synchronizing video and audio, is used as the video address. To do.

When the video recording control unit 1002 receives a recording start instruction, the video recording control unit 1002 issues an operation start instruction to the motion information acquisition unit 1004. When the video recording control unit 1002 receives a recording end instruction, the motion information acquisition unit An operation end instruction is issued to 1004.
The video holding unit 1003 records the video captured by the imaging unit 1001 together with the video address generated by the video recording control unit 1002 in accordance with an instruction from the video recording control unit 1002.

  The movement information acquisition unit 1004 acquires information related to the movement of the imaging unit 1001 from one or more sensors installed to measure the movement of the imaging unit 1001. In addition, information about the movement of the imaging unit 1001 is calculated using information from these sensors. Specifically, when an acceleration sensor is used to acquire information related to the movement of the imaging unit 1001, the acceleration information is acquired from the sensor, and the acceleration information is integrated to calculate the speed, and the noise from the sensor information. Processing for removal is performed. The sensor to be mounted is not limited to the acceleration sensor, but may be an angular velocity sensor or the like. Further, the value calculated from the sensor information is not limited to the speed, and the moving distance or the like may be obtained by calculation. The sensor information acquired in this way, information acquired by performing processing using the sensor information, and the like are transmitted to the change point detection unit 1005 and the motion section information acquisition unit 1007. These pieces of transmitted information are collectively called “motion information”.

The motion information acquisition unit 1004 only needs to operate from the start of video recording until the end of recording, and therefore starts and ends operation in accordance with an instruction from the video recording control unit 1002.
The change point detection unit 1005 detects a point at which the shooting operation is changed based on the motion information of the imaging unit 1001 acquired from the motion information acquisition unit 1004. In the present embodiment, speed information is acquired as motion information from the motion information acquisition unit 1004, and the magnitude of the speed, that is, the time when the speed becomes 0 from a value other than 0 and the time when the speed becomes a value other than 0 to 0 are obtained. This is detected as a change point of the shooting operation. The change point detection method is not limited to the current method. For example, when a person is shooting with a video camera in his / her hand, even when he intends to be stationary, there is some shaking, so when the speed changes below a certain value, The time of change over a certain value may be used as the change point. In addition, when shooting on a moving vehicle, the video camera has the same speed as the moving speed of the vehicle even if the camera is stationary. In order to deal with such a case, a three-dimensional orthogonal coordinate system is defined, the speed is decomposed into each axial component of the orthogonal coordinate system, and each value is a specific value, that is, the moving speed of the vehicle is changed to each of the orthogonal coordinate system. The change point may be the time when the value is the same as the value decomposed into the axial component. As described above, the change point detection unit 1005 uses the information that can be acquired from the motion information acquisition unit 1004 as a change point when a specific condition is satisfied. Further, the rules for detecting such a change point may be selected and set for each shooting or for each user.

When the change point can be detected in this way, the video address acquisition unit 1006 is notified in order to acquire the video address specifying the video position at the time when the change point is detected. Also, the motion section information acquisition unit 1007 is notified.
The video address acquisition unit 1006 acquires a video address from the video recording control unit 1002 in accordance with an instruction from the change point detection unit 1005. Then, the acquired video address is recorded in the information holding unit 1010.

  The motion section information acquisition unit 1007 is based on the change point information received from the change point detection unit 1005 and the motion information acquired from the motion information acquisition unit 1004 until the next change point is detected from a certain change point. The motion information of the section is held, and the motion information of the section from the held change point until the next change point is detected is transmitted to the camera work information deriving unit 1008 and the camera work evaluation unit 1009. This transmitted information is called “motion section information”.

  The camera work information deriving unit 1008 derives camera work information from the motion section information acquired from the motion section information acquiring unit 1007. The camera work information deriving unit 1008 holds in advance a video camera moving method (shooting operation) during shooting and the corresponding camera work information, and determines the shooting operation during shooting from the motion section information. Deriving corresponding camera work information.

The correspondence between the camera work information and the shooting operation is referred to as “camera work correspondence information”.
The derived camera work information is recorded in the information holding unit 1010.
The camera work evaluation unit 1009 obtains the motion section information using the motion section information acquired from the motion section information acquisition unit 1007 and the camera work information acquired from the camera work information derivation unit 1008 corresponding to the motion section information. The evaluation is given to the same video section as that of the current section. The evaluation is performed based on the motion section information, using an evaluation criterion depending on camera work information held in advance. This evaluation standard is referred to as a “camera work evaluation index”. The given evaluation value is called “camera work evaluation value”. Evaluation here means not only information such as video quality information that indicates whether or not to withstand viewing and availability information that indicates whether or not it is likely to be used for later editing, etc. It also includes camerawork supplementary information that supplements camerawork information such as the direction of movement and whether the moving speed is fast or slow.

A camerawork evaluation index that does not depend on camerawork information may be used.
The information holding unit 1010 holds a video address, camera work information, and camera work evaluation information. An example of information held by the information holding unit 1010 and the information management table 2000 is shown in FIG. A column 2001 describes the video address detected by the change point detection unit 1005 and acquired from the video recording control unit 1002 by the video address acquisition unit 1006, and the start address and end address of the video section divided by the change point. For example, the lines 2004, 2005, and 2006 are described so as to correspond to the video sections. A column 2002 describes camera work information obtained by the camera work information deriving unit 1008. A column 2003 describes camera work evaluation values obtained by the camera work evaluation unit 1009. For example, row 2004 is said to have camera work information “still” and camera work evaluation value “OK” for the video section specified by video addresses from 00:00:00 to 00: 00: 10: 00. Represents that. Also, a plurality of camera work evaluation values may be given to one video section as in column 2003 of row 2005.

In the information management table 2000, “stationary”, “OK”, and the like are described for easy understanding by looking at the drawings. However, the information management table 2000 can actually be used for processing in the apparatus and can be processed by the apparatus. It is described with a code, a code, and the like.
(2. Details of processing)
In the following, as an example, consider a case where all the configuration shown in FIG. 1 is mounted on the video camera 3001 shown in FIG. 3 and shooting is performed according to the flow shown in FIG.

  As shown in FIG. 3, in the video camera 3001 installed horizontally, the shooting direction is the positive y-axis direction, the upward vertical direction is the positive z-axis direction, and the horizontal right direction is 90 ° toward the positive y-axis. Let us consider a coordinate axis fixed to the video camera 3001 in the positive axis direction. Since the coordinate axis is fixed to the video camera 3001, when the posture of the video camera 3001 is changed, the z-axis may be lost in the vertical direction.

Here, the flow of FIG. 4 will be described.
A recording start instruction is issued to the video recording control unit, and video video recording is started (S4001).
Shooting is performed without moving the video camera 3001 up and down, left and right, etc. (S4002).

Next, in the recording state, the camera moves in the shooting direction (S4003).
After the movement in S4003 is completed, shooting is performed while changing the shooting direction of the video camera 3001 (S4004).
A recording end instruction is given to the video recording control unit, and the recording is ended (S4005).
Thus, the process ends (S4006).

Details of operations of typical components when shooting is performed according to the flow illustrated in FIG. 4 using the video camera 3001 illustrated in FIG. 3 in which all the configurations illustrated in FIG. 1 are mounted will be described.
Note that in this embodiment, all the configurations shown in FIG. 1 are mounted on the video camera 3001, but the present invention is not limited to such a mounting method. It may be divided into a plurality of devices and each function is implemented like a video camera, a personal computer, and a video recorder, and information and data may be transmitted and received between them by communication or the like.

(2-1. Example of operation of video recording control unit)
Details of the operation of the video recording control unit 1002 will be described. The recording start instruction in S4001 and the recording end instruction in S4005 are made to the video recording control unit 1002. The processing flow of the video recording control unit 1002 at this time will be described with reference to the flowchart shown in FIG.
A recording start instruction is received from the user or another device (S5001).

The motion information acquisition unit 1004 is instructed to start the operation (S5002).
One video frame is acquired from the imaging unit 1001 (S5003).
A video address for identifying the video frame acquired in S5003 is generated (S5004). As a video address, for example, a linear time code defined in SMPTE12M-1995 is used. The video address is generated in accordance with this standard, the number of video frames is counted from the recording start point, and the hour, minute, second, and frame are separated by punctuation marks. In the case of an image of 30 frames per second, the frame is continuously counted up from 0 to 29, and then the digit is increased to the second. In addition, in the NTSC television system, the number of frames is in accordance with the SMPTE standard, such as when 1 second is not 30 frames, when 1 second is 25 frames, and when 1 second is 24 frames.

The video frame acquired in S5003 and the video address generated in S5004 are recorded in the video holding unit 1003 (S5005).
If the recording end instruction has not been received, the process returns to S5002 to repeat the process. If a recording end instruction has been received, the process advances to step S5007 (S5006).
If it is determined in S5006 that an instruction to end recording has been received, the motion information acquisition unit 1004 is instructed to end the operation (S5007).

Above, the process of the video recording control part 1002 is complete | finished (S5008).
(2-2. Example of operation of motion information acquisition unit)
Next, the operation of the motion information acquisition unit 1004 will be described. In this embodiment mode, the video camera 3001 is provided with a sensor that can measure the acceleration in the x-, y-, and z-axis directions shown in FIG. 3 and the rotation speed about the x-, y-, and z-axes. Consider a case where a three-axis angular velocity sensor capable of measurement is installed. The motion information acquisition unit 1004 acquires sensor information from an acceleration sensor and an angular velocity sensor mounted so as to acquire the motion of the video camera 3001. The motion information acquisition unit 1004 calculates the speed by integrating information obtained from the acceleration sensor. At the time of this speed calculation, based on the information obtained from the angular velocity sensor, the acceleration due to the centrifugal force due to the rotational motion is excluded from the acceleration obtained from the acceleration sensor. For integration, for example, numerical integration is used. The acquired acceleration information and velocity information, which are motion information, are transmitted to the change point detection unit 1005 and the motion section information acquisition unit 1007. In addition, in the movement information acquisition unit 1004, a change inspection is performed by regarding the acquired sensor information and removing a minute vibration part or a specific part that is greatly deviated locally from sensor information acquired at a close time as noise. You may transmit to the output part 1005 and the motion area information acquisition part 1007. FIG. Information removal performed at this time is referred to as “noise removal”. As a method for removing noise, there are a frequency filtering method in which information from a sensor is decomposed into frequency components, and information on a specific frequency or a specific range of frequencies is deleted, or a moving average is taken. Since these noise removal methods are not the main point of the present invention, a detailed description thereof will be omitted. Further, the acceleration information from which the noise has been removed may be used in the speed calculation.

  The video recording control unit 1002 receives the recording start instruction in S4001 of FIG. 4 (S5001), the video recording control unit 1002 issues an operation start instruction (S5002), and the motion information acquisition unit 1004 receives the operation to operate. To start. Then, the video recording control unit 1002 receives the recording end instruction in S4005 of FIG. 4 (S5006), the video recording control unit 1002 issues an operation end instruction (S5007), and the motion information acquisition unit 1004 receives it. To finish the operation.

The motion information acquisition unit 1004 performs each shooting of S4002, S4003, and S4004 from the start to the end of the operation.
FIGS. 6A, 6B, and 6C illustrate the movement of the video camera 3001 at the time of shooting in S4002, S4003, and S4004 in FIG.
In the case of shooting in S4002, the video camera 3001 is shot without moving it left and right, back and forth, and up and down as shown in FIG.

When moving in the recording state in S4003, the gait of the photographer is modeled, and the video camera 3001 moves forward and periodically as shown in FIG. 6 (b). Think.
When shooting is performed while changing the orientation of the video camera 3001 in S4004, as shown in FIG. 6C, the origin is provided behind the video camera on the y axis (the negative direction of the y axis). The direction of shooting is changed by rotating around the origin.

  FIG. 7 shows the values of the three-axis acceleration sensor acquired by the motion information acquisition unit 1004 for a certain period of time for each frame interval when the images are taken sequentially in S4002, S4003, and S4004. Shown in FIG. 7A shows the time change of the acceleration in the x-axis direction. FIG. 7B shows the time change of acceleration in the y-axis direction. FIG. 7C shows the time change of acceleration in the z-axis direction.

  The number of frames per second is f, the number of frames from the recording start point is n, the speed of the nth frame is V (n), the speed of the n−1th frame is V (n−1), and the nth frame. If the acceleration of A is a (n), V (n) is obtained by the following (Equation 1). Here, V (0) is the speed at the recording start point, but this time, V (0) = 0 is set in order to obtain the speed on the basis of the recording start point.

In addition, when a rotational motion is performed, it is necessary to obtain the speed from the acceleration in consideration of the centrifugal force accompanying the rotational motion. For example, when there is a rotational motion component around the z axis, the acceleration W due to centrifugal force is obtained from the angular velocity ω obtained from the angular velocity sensor and the tangential velocity v of the rotating circle using the following (Equation 2).

The obtained acceleration W is added to the rotation center direction acceleration, and the speed is obtained from the acceleration.
The same applies to the case where rotational motion around other x-axis and y-axis is performed. In addition, the tangential direction and the rotational center direction of the rotational motion are determined based on the value of the triaxial acceleration sensor as the tangential direction when the acceleration value rises earlier and the slower direction as the rotational center direction.

  FIG. 8 shows the velocity information obtained by using (Equation 1) from the acceleration information shown in FIG. (A), (b), and (c) of FIG. 8 respectively represent changes in speed in the x-axis direction, the y-axis direction, and the z-axis direction in order. The section from point 8003 to point 8004 in FIG. 8B has acceleration in FIG. 7B, but the result of adding acceleration W due to centrifugal force to the acceleration in FIG. 7B is 0. , (Equation 1) was always 0 when the speed was obtained.

The motion information acquisition unit 1004 acquires information acquired from the sensors shown in FIGS. 7A, 7B, and 7C, and sensor information shown in FIGS. 8A, 8B, and 8C. The information obtained by processing is sequentially transmitted to the change point detection unit 1005 and the motion section information acquisition unit 1007.
In this case, numerical integration was performed using (Equation 1) and the speed was obtained, but since it is not the main point of the invention of the present application, detailed explanation is omitted, but a method generally called Simpson formula or trapezoid formula is used. Also good.

(2-3. Operation Example of Change Point Detection Unit)
Next, the operation of the change point detection unit 1005 will be described. The operation flow is shown as a flowchart in FIG.
It is determined whether speed and acceleration information can be received from the motion information acquisition unit 1004 (S9001). If it has been received, the process proceeds to S9002, and if it has not been received, the process is terminated (S9004).

If speed and acceleration information can be received from the motion information acquisition unit 1004, it is determined whether the point is a change point according to the change point detection rule (S9002). If it is determined that it is a change point, the process proceeds to S9003, and if it is determined that it is not a change point, the process proceeds to S9001.
When a change point is detected in S9002, a change point detection notification is transmitted to the video address acquisition unit 1006 and the motion section information acquisition unit 1007 (S9003).

In the present embodiment, the change point is determined by using the velocities in the x-, y-, and z-axis directions shown in (a), (b), and (c) of FIG. 8 received from the motion information acquisition unit 1004. It will be detected.
Further, as a rule for detecting the change point this time, paying attention to the speed of the video camera 3001, that is, the change in the speed of the video camera 3001 obtained by combining the speeds in the axial directions of x, y, and z, A point where the speed changes from 0 to a value other than 0 and a point where the speed changes from a value other than 0 to 0 are detected as change points. When the speed of the video camera 3001 is V, the speed V can be obtained by the following (Equation 3).

The processing flow at this time will be described with reference to the flowchart shown in FIG. This flowchart explains in detail the change point detection process of S9002 of the flowchart of FIG.

The velocity V is calculated using the above (Equation 3) from the velocities in the respective axial directions of x, y, and z acquired in S9001 (S9101).
It is determined whether or not the speed U calculated at the previous time is present (S9102). When the processing of the flowchart of FIG. 9B is performed for the first time immediately after the start of video recording, U does not exist. If the speed U exists, the process proceeds to S9103, and if not, the process proceeds to S9105.

If the speed U exists, it is determined whether the condition of “U = 0 and V ≠ 0” or “U ≠ 0 and V = 0” is satisfied using V and U calculated in S9101 (S9103). ). If either condition is satisfied, the process proceeds to S9104, and if neither condition is satisfied, the process proceeds to S9105.
When the condition is satisfied in S9103, the change point is set (S9104). If the condition of “U = 0 and V ≠ 0” is satisfied in S9103, the time point at which the speed U can be obtained, that is, the time point immediately before is set as the changing point. In the present embodiment, since this flowchart is processed for each frame, it means that the previous frame is the change point. If the condition of “U ≠ 0 and V = 0” is satisfied in S9103, the time point at which the speed V can be obtained, that is, the current frame is set as the changing point. As a result, it is determined in FIG. 9A that a change point has been detected in S9002, and the process proceeds to S9003.

The speed V obtained in S9101 is substituted for U (S9105).
The process is terminated (S9106).
The change point detection process when the change point detection unit 1005 acquires the axial speeds of x, y, and z shown in FIGS. 8A, 8B, and 8C using a specific example. explain. Here, a point 8001, a point 8002, a point 8003, and a point 8004 in FIG. 8 indicate a certain video frame. A point 8001 is a video frame at the video recording start point, a point 8002 and a point 8003 are video frames at a point where the temporal change of the axial velocity is 0 or changed from 0, and a point 8004 is the video recording end. Pointing to a video frame of points. The change point detection unit 1005 discretely acquires the x, y, and z axial velocities shown in FIGS. 8A, 8B, and 8C at the same intervals as the video frames. The speed V is calculated from the acquired speed information using (Equation 3) (S9102). The required speed V is “0” between points 8001 and 8002. Therefore, the value of U is also “0” between the points 8001 and 8002, “U = 0 and V = 0”, and the condition in S9103 is not satisfied.

  Next, the velocity V is calculated at the time point when the velocity information is acquired after the point 8002, that is, between the points 8002 and 8003, that is, at a point one frame after the point 8002. At this time, since the y-axis direction speed and the z-axis direction speed are not “0”, the speed V also has a value that is not “0”. Further, the speed V at the time of the point 8002 is “0”, and this value is currently held as the value of U. Therefore, the condition “U = 0 and V ≠ 0” is satisfied (S9103). Since the condition of “U = 0 and V ≠ 0” is satisfied, the time point at which the speed of U is acquired, that is, the time point of point 8002 is detected as a change point.

  Next, since the speed in the y-axis direction or the speed in the z-axis direction is not “0” between the points 8002 and 8003 not including the points 8002 and 8003, V always has a value that is not “0”. Accordingly, the value of U is not “0” between the point 8002 and the point 8002 one frame after and between the point 8002 and the point 8003 not including the point 8003. Therefore, “U ≠ 0 and V ≠ 0”, and the condition in S9103 is not satisfied.

  Next, the speed V is calculated at a point 8003. At this time, since the velocities in all the axial directions of x, y, and z are “0”, the speed V is “0”. Further, the speed V one frame before the point 8003, that is, the value of U at the time of the point 8003 is not “0” because the y-axis direction speed or the z-axis direction speed is not “0”. Therefore, “U ≠ 0 and V = 0” is satisfied, the condition in S9103 is satisfied, and the time point when the speed of V is acquired, that is, the time point 8003 is detected as the changing point.

Similar to the above processing, the point 8004 is detected as a change point.
Therefore, the changing point detection unit 1005 detects points 8002, 8003, and 8004 in order as changing points.
In addition, the video recording control unit 1002 issues an operation start instruction to the motion information acquisition unit 1004 to start the operation. As a result, the change point detection unit 1005 can receive information such as speed and acceleration. That is, the point 8001 which is the operation start point is also regarded as equivalent to the change point. In addition, this time, the operation end point is the same as the change point 8004. However, when the video recording control unit 1002 receives a recording end instruction while the video camera 3001 is moving, The end point and the last detected change point may not be the same. In this case, the operation end point is regarded as a change point and processed.

The change point detection unit 1005 sends the video address acquisition unit 1006 and the motion section information acquisition unit 1007 to the point 8001 that is the operation start point and the point 8002, the point 8003, and the point 8004 that are the change points. On the other hand, a change point detection notification is transmitted.
In this example, the change point detection rule is a point where the speed has changed from 0 to a value other than 0 and a point where the speed has changed from a value other than 0 to 0. However, it is not limited to this.

(2-4. Example of operation of video address acquisition unit)
Next, the operation of the video address acquisition unit 1006 will be described. The video address acquisition unit 1006 acquires a video address from the video recording control unit 1002 when receiving a change point detection notification from the change point detection unit 1005. The video recording control unit 1002 assigns a video address to the video being captured. Further, the acquired video address is transmitted to the information holding unit 1010.

This time, the video address acquired at the point 8001 as the operation start point is “00:00:00”, and the video address acquired when the points 8002, 8003, and 8004 as the change points are detected. In this order, “00: 00: 10: 00”, “00: 00: 20: 00”, and “00: 00: 30: 30” are set.
Note that this time, assuming that the processing of this system is sufficiently fast, the video recording control unit 1002 gives a change point detection notification from the change point detection unit 1005 when the video address acquisition unit 1006 receives it. I got an address. However, the video recording is changed in the video recording control unit 1002 while the system records the video and acquires the motion information, detects the change point, and the video address acquisition unit 1006 receives the change point detection notification. If processing takes longer than this, the video address is temporarily stored in the video recording control unit 1002 such as a FIFO so that the video address at the time when the change point is detected can be appropriately acquired. If a buffer is provided and stored, data can be read from this buffer.

(2-5. Operation example of motion section information acquisition unit)
Next, the operation of the motion section information acquisition unit 1007 will be described. The flow of operation of the motion section information acquisition unit 1007 will be described with reference to the flowchart of FIG.
A change point detection notification is received from the change point detection unit 1005 (S10001). The change point detection notification received first is a change point detection notification related to the operation start point.

The motion information is acquired from the motion information acquisition unit 1004 (S10002). If motion information can be acquired, the process proceeds to S10003. If it cannot be acquired, the process is terminated (S10007).
The motion information acquired from the motion information acquisition unit 1004 is accumulated (S10003). This accumulated information is referred to as “motion section intermediate information”.

When the change point detection notification is received from the change point detection unit 1005, the process proceeds to S10005. When the change point detection cannot be received, the process returns to S10002, and the processes of S10002 and S10003 are repeated until reception is possible (S10004).
When the change point detection notification is received in S10004, the motion section intermediate information accumulated in S10003 is transmitted to the camera work information deriving unit 1008 and the camera work evaluation unit 1009 as motion section information (S10005).

The motion section intermediate information accumulated in S10003 is discarded, and the process returns to S10002 (S10006).
Next, while receiving (a), (b), and (c) in FIG. 7 and (a), (b), and (c) in FIG. 8 as motion information, it is an operation start point from the change point detection unit 1005. Consider a case where notifications regarding a point 8001, a change point 8002, a change point 8003, and a change point 8004 are sequentially received.

In S10001, a notification regarding the point 8001 which is the operation start point is received.
In S10002, motion information shown in (a), (b), and (c) of FIG. 7 and (a), (b), and (c) of FIG. 8 is acquired. In S10002, the motion information acquisition unit 1004 does not receive all the motion information shown in (a), (b), and (c) of FIG. 7 and (a), (b), and (c) of FIG. Thus, it is received from time to time from the motion information acquired in frame units.

In S10003, the motion information received at any time in S10002 is accumulated in a buffer memory or the like (not shown) in the apparatus, and the motion section intermediate information is updated.
Until the change point detection notification related to the change point 8002 is received in S10004, the motion information acquisition in S10002 and the motion information accumulation in S10003, that is, the motion section intermediate information is updated. When the change point detection notification regarding the point 8002 which is the change point is received, the change point detection notification regarding the point 8002 which is the change point is received and the change point detection notification regarding the point 8002 which is the change point is received. The motion information accumulated in S10003, that is, the motion section intermediate information is transmitted to the camera work information deriving unit 1008 and the camera work evaluation unit 1009 as motion section information.

The motion section information 11000 transmitted at this time is shown in FIG.
Line 11001 describes acceleration information, and line 11002 describes speed information.
Column 11003 describes information in the x-axis direction, column 11004 describes information in the y-axis direction, and column 11005 describes information in the z-axis direction. That is, the acceleration in the x-axis direction is described in the column 11003 of the row 11001.

As illustrated in FIG. 11, the motion information acquired from the motion information acquisition unit 1004 is accumulated in S10003 for the section between the point 8001 that is the operation start point and the point 8002 that is the change point.
After the transmission to the camera work evaluation unit 1009 is completed, the information stored in S10003 is discarded.

Returning to S10002, by the processing of S10002 and S10003, the motion information from when the change point detection notification regarding the change point 8002 is received until the next change point detection notification regarding the change point 8003 is received. Will be accumulated.
FIG. 12 shows motion section information 12000 transmitted to the camera work information deriving unit 1008 and the camera work evaluation unit 1009 when a change point detection notification regarding the point 8003 that is a change point is received. The information description method is the same as in FIG.

FIG. 13 shows motion section information 13000 transmitted to the camera work information deriving unit 1008 and the camera work evaluation unit 1009 when processing is performed in the same manner and a change point detection notification regarding the change point 8004 is received. The information description method is the same as in FIG.
(2-6. Operation example of camera work information deriving unit)
Next, the operation of the camera work information deriving unit 1008 will be described. The camera work information deriving unit 1008 derives camera work information corresponding to the acquired motion section information using the motion section information acquired from the motion section information acquiring unit 1007 and the camera work correspondence information stored in advance. .

  An example of camerawork correspondence information is shown in FIGS. 14, 15, and 16 (a) and (b), and FIGS. 17 (a) and (b). FIG. 14, FIG. 15, FIG. 16, and FIG. 17 show the accelerations and velocities in the respective axial directions when the camera work is stationary, dolly, pan, and tilt, respectively. In addition, these are examples and are not limited only to these. In addition, this time, the motion section information composed of the velocity and acceleration in the x, y, and z axes and the camera work information are described in correspondence with each other. The information to be used is not limited to these. The conditional expression for evaluating the motion section information may be described in correspondence with the camera work information. Camera work information is described in a row 14001 in FIG. In line 14002 and line 14003, motion section information corresponding to the camera work information described in line 14001 is described. The motion section information described in the camera work correspondence information is given as numerical data, but in FIGS. 14, 15 and 16 (a) and (b), and in FIGS. 17 (a) and (b). The numerical data is shown for easy understanding. The row 14002 describes acceleration, the row 14003 describes velocity, the column 14004 contains information about the x-axis direction, the column 14005 contains information about the y-axis direction, and the column 14006 contains information about the z-axis direction. is described. That is, in the column 14004 of the row 14002, information in the x-axis direction of acceleration in the motion section information corresponding to the camera work information “still” described in the row 14001 is described.

Even if the camera work information is “Pan”, there are various movements such as panning to the right, panning to the left, and slowly and quickly. Therefore, as shown separately in FIGS. 16A and 16B, the same camera work information “pan” may be associated with a combination of different motion section information.
The process flow of the camera work information deriving unit will be described with reference to the flowchart of FIG.

The motion section information is acquired from the motion section information acquisition unit 1007 (S18001).
The time change of the acceleration in the x-axis direction is extracted from the motion section information acquired in S18001, and collated with the time change of the x-axis direction acceleration in the camera work correspondence information held in advance. The collation method at this time is such that the time change value of the x-axis direction acceleration held in the camera work correspondence information is the same as the time change time of the x-axis direction acceleration of the motion section information acquired in S18001. The acceleration is enlarged or reduced in the time axis direction, and the acceleration is also enlarged or reduced so that the maximum value of the acceleration becomes the same. Thereafter, using the value of the temporal change in the x-axis direction acceleration described in the enlarged or reduced camera work correspondence information and the motion section information acquired in S18001, a correlation coefficient is obtained, and the value is If it is greater than a certain value, it is considered as a match. The correlation coefficient r is obtained using, for example, the following (Equation 4). Here, n is the number of data to be collated, d (i) and e (i) are i-th values of data d and e to be collated, and D and E are average values of d and e, respectively. is there. In the present embodiment, d (i) and e (i) are values at each time of x-axis direction acceleration held by the motion section information, and at each time of x-axis direction acceleration held by the camera work correspondence information. Is the value of

In the present embodiment, r = 1 is set as a condition that is considered to be the same. That is, a case where the time change of acceleration completely matches between the movement section information and the information held in the camera work correspondence information is regarded as a match. Note that, as a condition to be regarded as coincidence, k can be freely set from 0 to 1 with r> k. As the value of k approaches 0, the condition to be considered to be consistent can be relaxed, and even if the correlation between the motion section information and the acceleration in the x-axis direction held in the camera work correspondence information is low, it is considered to be consistent. It will be said.

Similarly, collation is performed for the velocity in the x-axis direction, the velocity acceleration in the y-axis direction, and the z-axis direction (S18002).
In S18002, it is determined whether there is camerawork correspondence information in which the temporal changes in all of the x-, y-, and z-axis velocities and accelerations coincide with the motion section information (S18003). When it exists, it progresses to S18005, and when it does not exist, it progresses to S18004.

  If it is determined in S18003 that it does not exist, the camerawork relaxation condition is applied and collation is performed again with the camerawork correspondence information (S18004). Camera work relaxation conditions will be described later. After applying the camera work relaxation condition, it is determined whether there is camera work correspondence information in which the temporal changes in the speed and acceleration of the x, y, and z axes all match the motion section information acquired in S18001. If there is a match, the process proceeds to S18005, and if not, the process proceeds to S18006.

If it is determined in S18003 or S18004, the camerawork information described in the camerawork correspondence information is acquired as the camerawork information of the motion section information acquired in S18001 (S18005).
The camera work information acquired in S18005 is transmitted to the information holding unit 1010 and is transmitted to the camera work evaluation unit 1009 (S18006).

Thus, the process ends (S18007).
An example of the camera work relaxation condition table holding camera work relaxation conditions is shown in FIG. In each row of the camera work relaxation condition table 19000, information for relaxing the conditions of the camera work correspondence information is described. In a column 19005, camera work information is described in order to identify the corresponding camera work correspondence information. A column 19006 describes information for relaxing the conditions of the camera work correspondence information. For example, in the row 19001, “pan” is displayed in the column 19005, no information is described in the columns 19008 and 19209 in the row 19011, and “N / A” is described in the column 19010. In addition, no information is described in the columns 19008 and 19209 of the row 19012, and “N / A” is described in the column 19010.

  Since the line 19001 “Pan” is described, this represents a relaxation condition for the camera work correspondence information 16100 and the camera work correspondence information 16200 defining the camera work of “Pan”. As for the acceleration and speed in the x-axis direction and the acceleration and speed in the y-axis direction, which are not described at all, the x-axis direction and the y-axis described in the columns 16103 and 16104 of the camera work correspondence information 16100 The direction acceleration and speed change with time and the motion section information are collated. The same applies to the camera work correspondence information 16200. Regarding the acceleration and speed in the z-axis direction described as “N / A”, the speed in the z-axis direction described in the column 16102 of the camera work correspondence information 16100 and the column 16202 of the camera work correspondence information 16200 In addition, when collating the time change of acceleration and the motion section information, it is determined that they always match. The same applies to other camera work relaxation conditions. Here, the camera work information deriving unit 1008 holds the camera work correspondence information shown in FIGS. 14, 15, and 16 (a) and (b), and FIGS. 17 (a) and 17 (b). While the information described in the camera work relaxation condition table 19000 shown in FIG. 19 is held as the work relaxation condition, the motion section information shown in FIGS. 11, 12, and 13 is acquired from the motion section information acquisition unit 1007. The operation in this case will be described below.

The motion section information in FIG. 11 is acquired (S18001).
The camera work information deriving unit 1008 is motion section information described in the camera work correspondence information shown in FIGS. 14, 15 and 16 (a) and (b) and FIGS. 17 (a) and 17 (b). Is collated (S18002). First, the time change of the acceleration in the x-axis direction described in the column 11003 of the row 11001 of the motion section information 11000, and FIGS. 14, 15, 16A, 16B, and 17A. The correlation coefficient is obtained from the time change of the acceleration in the x-axis direction described in the camera work correspondence information shown in (b). As a result of obtaining the correlation coefficient, the correlation coefficient between the acceleration in the x-axis direction described in (a) and (b) of FIG. 14, FIG. 15, and FIG. . Similarly, the velocity in the x-axis direction described in the column 11003 in the row 11002 of the motion section information 11000, the acceleration and velocity in the y-axis direction described in the column 11004, and the acceleration in the z-axis direction described in the column 11005 Similarly, regarding the speed and the speed, the temporal change of the speed and acceleration in each axis direction described in the camera work correspondence information and the correlation coefficient are obtained, and it is determined whether they match.

  In the present embodiment, the time change information of the acceleration in each axis direction described in the line 14002 of the camera work correspondence information 14000 shown in FIG. 14 and the acceleration time described in the line 11001 of FIG. The change information matches, and the time change information of the speed in each axis direction described in the line 14003 of the camera work correspondence information 14000 and the time change information of the speed described in the line 11002 of FIG. It is determined that they match (S18003).

The camera work information corresponding to the motion section information in FIG. 11 is determined to be “still” described in the row 14001 of the camera work correspondence information 14000 (S18005).
The derived camera work information “still” is transmitted to the information holding unit 1010 and is also transmitted to the camera work evaluation unit 1009 (S18006).
The motion section information 12000 in FIG. 12 is acquired (S18001).

  The camera work information deriving unit 1008 is motion section information described in the camera work correspondence information shown in FIGS. 14, 15 and 16 (a) and (b) and FIGS. 17 (a) and 17 (b). Is collated (S18002). As in the case of the motion section information 11000 described above, the temporal changes in the acceleration and velocity in the x, y, and z axes, and FIGS. 14, 15, 16 (a) and (b), and FIG. 17. A correlation coefficient is obtained from the temporal change in acceleration and velocity in the x, y, and z axes described in the camera work correspondence information shown in (a) and (b), and it is determined whether or not they match. To do.

However, in this case, there is no camera work correspondence information in which the movement section information 12000 and the temporal changes in the acceleration and velocity in the respective x, y, and z axes coincide with each other (S18003).
Accordingly, the camera work relaxation condition held in the camera work relaxation condition table 19000 shown in FIG. 19 is applied to the camera work correspondence information, and collation is performed again (S18004). The information described in the column 12003 and column 12004 of the motion section information 12000 matches the information described in the column 15004 and column 15005 of the camera work correspondence information 15000. Also, the time change information of the acceleration and speed in the z-axis direction described in the column 12005 of the motion section information 12000, and the time of acceleration and velocity in the z-axis direction described in the column 15006 of the camera work correspondence information 15000. Change is different. However, from the camera work information “Dolly” described in the line 15001 of the camera work correspondence information 15000, the line 19003 described as “Dolly” in the column 19005 of the camera work relaxation condition table 19000 is found. When the camera work relaxation condition described in 19003 is applied to the camera work correspondence information 15000, the time change in acceleration and velocity in the z-axis direction described in the column 15006 is equal to one regardless of the value of the correlation coefficient. You can judge that you are doing it. Therefore, it is determined that the motion section information 12000 and the camera work correspondence information in which the temporal changes in acceleration and velocity in the respective x, y, and z axes coincide with each other exist.

The camera work information corresponding to the motion section information 12000 is determined to be “dolly” described in the row 15001 of the camera work correspondence information 15000 (S18005).
The derived camera work information “Dolly” is transmitted to the information holding unit 1010 and is also transmitted to the camera work evaluation unit 1009 (S18006).

In this way, even if the information described in the camera work correspondence information does not match the motion section information, the camera work correspondence information is partially relaxed according to the rules using the camera work relaxation conditions, It is determined whether it matches the section information, and camera work information is determined.
When the motion section information 13000 in FIG. 13 is acquired, the camera work information deriving unit 1008 performs the same processing as that in the case of acquiring the motion section information 11000, and the camera work correspondence information 16100 in FIG. It is determined that it is “pan” described in the row 16101.

The derived camera work information “pan” is transmitted to the information holding unit 1010 and also transmitted to the camera work evaluation unit 1009.
(2-7. Example of camera work evaluation unit operation)
Next, the operation of the camera work evaluation 1009 part will be described. The flow of processing will be described using the flowchart of FIG.

The motion section information is acquired from the motion section information acquisition unit 1007 (S20001).
Next, camera work information corresponding to the motion section information in S20001 is acquired from the camera work information deriving unit 1008 (S20002).
Using the camera work information acquired in S20002, the camerawork evaluation index used this time is selected from a plurality of camerawork evaluation indices held in advance (S20003).

Using the camerawork evaluation index selected in S20003, the motion section information acquired in S20001 is evaluated to obtain an evaluation value (20004).
The evaluation value derived in 20004 is transmitted to the information holding unit 1010 (20005).
The process ends (20006).
FIG. 21 shows an example of the camerawork evaluation index.

  A column 21005 describes camera work information, a column 21006 describes an evaluation index, and a column 21007 describes an evaluation value. A plurality of evaluation indices and evaluation values may exist for one camera work information. For example, in the case of the camera work information “still” described in the column 21005 of the row 21004, there are two types of evaluation indexes, that is described in the row 21010 and that described in the row 21011. If the motion section information matches the evaluation index described in the column 21006 of the row 21010, the evaluation value described in the column 21007 of the row 21010 is given. The same applies to the evaluation indexes described in other rows. Here, “A”, “B”, “C”, “D”, “E”, and “F” used in the row 21006 are predetermined constants. The following (Equation 5), (Equation 6), and (Equation 7) represent the magnitude of the x-axis direction speed, the y-axis direction speed, and the z-axis direction speed, respectively.

It should be noted that the evaluation index and the evaluation value exemplified this time are contradictory for each camera work information, and the evaluation value is alternatively selected as the result of the evaluation. It is also conceivable to give an evaluation index in which there are overlapping portions with respect to information, or an evaluation index based on a completely different standard. In such a case, one may be selected instead of one as the evaluation value.

In the state where the camera work evaluation unit 1009 holds the camera work evaluation index shown in FIG. 21 in advance, the motion section information 11000 is acquired from the motion section information acquisition unit 1007, and “still” is set as the camera work information. The processing flow of the camera work evaluation unit 1009 when acquiring from the deriving unit 1008 will be described with reference to the flowchart of FIG.
The motion section information 11000 is acquired from the motion section information acquisition 1007 (S20001).
“Still” is acquired from the camera work information deriving unit 1008 as camera work information (S20002).

The camera work evaluation index when the camera work information is “still” described in the rows 21010 and 21011 is selected from the plurality of camera work evaluation indexes in FIG. 21 (S20003).
The motion section information 11000 is evaluated using the camera work evaluation indices described in the rows 21010 and 21011 in FIG. FIG. 22A illustrates the speed in each axial direction described in the row 11002 of the motion section information 11000 and the threshold values of the camera work evaluation indices described in the rows 21010 and 21011 in FIG. From FIG. 22A, it can be seen that the evaluation index described in the row 21010 is satisfied and the evaluation index described in the row 21011 is not satisfied. Therefore, “OK” described in the column 21007 of the row 21010 is obtained as the evaluation value (S20004). This “OK” means, for example, a video with high video quality. Then, the evaluation value “OK” is transmitted to the information holding unit 1010 as a camera work evaluation value (S20005).

  The same processing is performed when the movement section information 12000 and the camera work information “Dolly” are acquired. For reference, FIG. 22B shows a diagram when the evaluation is performed in S20004. Since the two evaluation indexes described in the row 21003 in FIG. 21 are both evaluation indexes for the velocity in the z-axis direction, the z-axis described in the column 22202 in the row 22201 in FIG. Evaluate for changes in direction velocity. In this case, it can be seen that the evaluation index described in the row 21209 is satisfied and the evaluation index described in the row 21008 is not satisfied. Therefore, “NG” is obtained as the evaluation value. Then, this value is transmitted to the information holding unit 1010 as a camera work evaluation value. This “NG” means, for example, a video with low video quality.

It is assumed that the same processing is performed when the motion section information 13000 and the camera work information “pan” are acquired, and “OK” is obtained as the evaluation value. This value is transmitted to the information holding unit 1010 as a camera work evaluation value.
(2-8. Example of operation of information holding unit)
The information holding unit 1010 will be described. The information acquisition unit 1010 records the video address of the change point acquired from the video address acquisition unit 1006, the camera work information acquired from the camera work information deriving unit 1008, and the camera work evaluation value acquired from the camera work evaluation unit 1009. Holds an information management table. As an example, an information management table is shown in FIG. In the information management table 23000, the video address is recorded in the column 23001, the camera work information is recorded in the column 23002, and the camera work evaluation value is recorded in the column 23003 in association with each row. For example, in the row 23004, the camera work information in the section from the video address “00: 00: 00: 00” to “00: 00: 10: 00” is “still”, and the camera work evaluation value is “OK”. It represents that. The other lines can be interpreted similarly.

(3. Effect)
Through the above processing, the video address is recorded at the change point of the shooting operation. By using this video address for the cue reproduction, the cue playback of the video can be performed in units of the shooting operation. In addition, camera work information is given to the video section divided by the cue point in association with the video, and the camera work information is searched using the search key as a search key to identify the corresponding video section. Editing operations such as playback and clipping can be performed on the recorded video. Each video section is evaluated by an evaluation index depending on camera work, and an evaluation value is given. Using this evaluation value, it is possible to perform processing such as presenting only a video section with high video quality, such as a video section with less camera shake, to the user.

  In this embodiment, an imaging unit is provided, a sensor is provided so that the motion of the imaging unit can be acquired, the video recording control unit acquires video from the imaging unit, and the motion information acquisition unit acquires motion information from the sensor. Although the explanation has been given using an example, the video to which the video address is assigned and the motion information of the device that captured the video are recorded in advance, and the motion information acquisition unit reads the recorded motion information, and the video The address acquisition unit may be operated to read the recorded address information.

(Embodiment 2)
In this embodiment, when shooting is performed using the video camera described in Embodiment 1, based on the information about the movement of the video camera from side to side and back and forth during shooting, A method and system for generating camerawork correspondence information held by a camerawork information deriving unit in a system that implements a method for assigning camerawork information will be described.

(1. Overall configuration)
FIG. 24 shows the overall configuration of the present embodiment. Since the imaging unit 1001, the video recording control unit 1002, the video holding unit 1003, the motion information acquisition unit 1004, the change point detection unit 1005, and the motion section information 1007 are the same as those in the first embodiment, description thereof is omitted. Further, the video address acquisition unit 1006 and the camera work evaluation unit 1009 are the same as those in the first embodiment except that the information recording unit is not the information holding unit 1010 but the information holding unit 24004. Description is omitted.

The camera work information deriving unit 24001 derives camera work information in the same manner as the camera work information deriving unit 1008 described in Embodiment 1, and records it in the information holding unit 24002. In addition, the motion section information used when the camera work information is derived is also recorded in the information holding unit 24002.
Similar to the information holding unit 1010 described in the first embodiment, the information holding unit 24002 holds a video address, camera work information, and camera work evaluation information. In addition to these pieces of information, motion section information received from the camera work information deriving unit 24001 is also held.

The camera work information definition unit 24003 has an input unit from the user such as a keyboard and a display unit that presents information to the user, reads out the movement section information held in the information holding unit 24002, and presents it to the user. A new camera work information definition is received from the user for the read motion section information.
The camera work correspondence information update unit 24004 generates motion section correspondence information from the camera work information newly defined by the camera work information definition unit 24003 and the motion section information corresponding to the camera work information, and the camera work information It is added to the camera work correspondence information held in the deriving unit.

(2. Details of processing)
Hereinafter, as an example, consider a case where the configuration shown in FIG. 24 is all installed in the video camera 3001 shown in FIG. 3 and shooting is performed according to the flow shown in FIG.
(2-1. Example of camera work information deriving unit)
The operation of the camera work information deriving unit 24001 will be described. Similarly to the camera work information deriving unit 1008, the camera work information deriving unit 24001 uses the motion section information acquired from the motion section information acquiring unit 1007 and the camera work correspondence information stored in advance to acquire the motion. The camera work information corresponding to the section information is derived.

In addition to the derived camera work information, the motion section information used when deriving the camera work information is also transmitted to the information holding unit 24002.
A processing flow of the camera work information deriving unit 24001 will be described with reference to a flowchart of FIG.
Since S18001, S18002, S18003, S18004, S18005, and S18007 are the same as those in the flowchart of FIG. 18 described when the processing of the camera work information deriving unit 1008 is described, the description thereof will be omitted.

After the camera work information is acquired from the camera work correspondence table in S18005, the motion section information acquired in S18001 and the camera work information acquired in S18005 are transmitted to the information holding unit 24002 (S25001).
Here, the camera work correspondence relationship information shown in FIGS. 14, 15 and 16 (a) and (b), FIG. 17 (a) and (b), and the camera work relaxation condition table shown in FIG. The operation when the motion section information 11000 shown in FIG. 11 is acquired from the motion section information acquisition unit 1007 in the state where the information is held will be described.

The processing from obtaining the motion section information 11000 in FIG. 11 to determining “still” as the camera work information is the same as the case of the camera work information deriving unit 1008, and is therefore omitted.
The motion section information 11000 shown in FIG. 11 acquired in S18001 and the camera work information “still” derived in S18005 are transmitted to the information holding unit 24002, and the camera work information “still” is transmitted to the camera work evaluation unit 1009. (S25001)
When the camera work information deriving unit 24001 acquires the motion section information 12000 or the motion section information 13000 shown in FIG. 12 or 13, the camerawork information deriving unit 24001 operates in the same manner as when the motion section information 11000 shown in FIG. Until the camera work information “dolly” or “pan” is derived, the camera work information deriving unit 1008 operates in the same manner. In S24001, the motion section information 12000 or the motion section information 13000 shown in FIG. 12 or 13 acquired in S18001 and the camera work information “Dolly” or “Pan” acquired in S18005 are transmitted to the information holding unit 24002. At the same time, the camera work information “Dolly” or “Pan” is transmitted to the camera work evaluation unit 1009.

(2-2. Example of operation of information holding unit)
The information holding unit 24002 will be described. The information holding unit 24002 receives the video address of the change point acquired from the video address acquisition unit 1006, the camera work information and motion section information acquired from the camera work information deriving unit 24001, and the camera work evaluation value acquired from the camera work evaluation unit 1009. An information management table is held for recording. As an example, FIG. 26 shows an information management table to be held. In the information management table 26000, the video address is recorded in the column 26001, the camera work information is recorded in the column 26002, the camera work evaluation value is recorded in the column 26003, and the motion section information is recorded in the column 26004 in association with each row. For example, the row 26005 includes the video address acquired from the video address acquisition unit 1006 in the column 26001, the camera work information acquired from the camera work information deriving unit 24001 in the column 26002, and the camera acquired from the camera work evaluation unit 1009 in the column 26003. The work evaluation value is stored in the column 26004 in the motion section information acquired from the camera work information deriving unit 24001, and the camera work in the section from the video address “00: 00: 00: 00” to “00: 00: 10: 00” is stored. The information is “still”, the camerawork evaluation value is “OK”, and the motion section information is the motion section information 11000 shown in FIG.

Similarly, the video address “00: 00: 10: 00” “00: 00: 20: 00”, camera work information “Dolly”, and camera work evaluation value “NG” are recorded in the row 26006 together with the motion section information 12000. ing.
Similarly, the video address “00: 00: 20: 00”, “00: 00: 30: 30”, camera work information “pan”, and camera work evaluation value “OK” are recorded in the row 26007 together with the motion section information 13000. ing.

(2-3. Operation example of camera work information definition unit)
The camera work information definition unit 24003 has an input unit from the user such as a keyboard and a display unit that presents information to the user, reads out the movement section information held in the information holding unit 24002, and presents it to the user. A new camera work information definition is received from the user for the read motion section information.

Details of the processing will be described with reference to the flowchart shown in FIG.
In accordance with an instruction from the user, one row is read from the information management table held in the information holding unit 24002 (S27001).
The video address, camera work information, camera work evaluation value, and motion section information read in S27001 are presented to the user, and input of camera work information newly defined by the user is accepted (S27002).

The motion section information read in S27001 and the camera work information input and received by the user in S27002 are transmitted to the camera work correspondence information update unit (S27003).
The process ends (S27004).
This will be described using a specific example.

Consider a state in which the information holding unit 24002 holds the information management table 26000 shown in FIG.
It is assumed that an instruction is received from the user to take out information in the row 26006 of the information management table 26000. At this time, the information in the row 26006 of the information management table 26000 is taken out according to the instruction from the user (S27001).

  The video address “00: 00: 10: 00”, “00: 00: 20: 00”, camera work information “Dolly”, camera work evaluation information “NG”, and motion section information 12000 extracted in S27001 are presented to the user. The camerawork information is newly input (S27002). In the present embodiment, it is assumed that an input “walking” is received. At this time, the video section indicated by the video address may be reproduced and presented to the user using the video address information and the video holding unit 1003 information.

The camera work information “walking” received in S27002 and the motion section information 12000 extracted in S27001 are transmitted to the camerawork correspondence information update unit 24004 (S27003).
Then, the process ends (S27004).
Note that the camera work information defined and input by the user in S27002 may be transmitted to the information holding unit 24002, and the camera work information recorded in the corresponding row may be replaced and recorded.

(2-4. Example of operation of camerawork correspondence information update unit)
The camera work correspondence information update unit 24004 generates motion section correspondence information from the camera work information newly defined by the camera work information definition unit 24003 and the motion section information corresponding to the camera work information, and the camera work information It is added to the camera work correspondence information held in the deriving unit.

Details of the processing will be described with reference to the flowchart shown in FIG.
Camerawork information and motion section information are acquired from the camerawork information definition unit 24003 (S28001).
The camera work correspondence information is created from the camera work information and the motion section information acquired in S28001 (S28002).

The camera work correspondence information created in S28002 is added to the camera work correspondence information held by the camera work deriving unit 24001 (S28003). When the same motion section information has already been defined, it may be replaced with the camera work correspondence information.
The process is terminated (S28004).
This will be described using a specific example.

Consider a case in which “walking” is acquired as camera work information and motion section information 12000 is acquired as motion section information from the camera work information definition unit 24003. At this time, the camera work information deriving unit 24001 holds the camera work correspondence information shown in FIGS. 14, 15 and 16 (a) and (b) and FIGS. 17 (a) and (b). Suppose that
From the camera work information definition unit 24003, camera work information “walking” and motion section information 12000 are acquired (S28001).

  The camera work correspondence information is created from the camera work information “walking” and the motion section information 12000 acquired in S28001 (S28002). FIG. 29 shows camerawork correspondence information 29000 created at this time. Line 29001 describes camera work information “walking”, and lines 29002 and 29003 describe motion section information 12000.

  The camera work correspondence information 29000 created in S28002 is added to the camera work correspondence information of the camera work information deriving unit 24001 (S28003). As a result, the camera work information deriving unit performs camera work along with the camera work correspondence information shown in FIGS. 14, 15, 16 (a) and (b), and FIGS. 17 (a) and 17 (b). Correspondence relationship information 29000 is also held.

Thus, the process ends (S28004).
Note that when creating the camerawork correspondence information, means for receiving input from the user may be provided, and the motion section information may be corrected by input from the user. Further, the camera work correspondence information may be created after approximating the motion section information with some straight lines or quadratic curves.

(3. Effect)
Through the above processing, new camera work correspondence information can be defined using information held in the information holding unit after the photographing is completed. The camera work correspondence information can be derived by using the defined camera work correspondence information at the next shooting. Also, by registering these camera work information in advance to the camera work information deriving section of other cameras before shooting, even the camera that has registered the camera work correspondence information has the same camera work as the camera that defined it. Information can be derived.

When camerawork correspondence information is newly defined and added to the camerawork information deriving unit, the motion section information in the information management table held in the information holding unit is transmitted to the camerawork information deriving unit, The camera work information may be derived again using the camera work correspondence information including the new camera work correspondence information.

  In this embodiment, after shooting, camera work information is defined using information held in the information holding unit, camera work correspondence information is created, and added to the camera work information deriving unit. However, if at least the movement section information can be acquired, the camera work information is defined in the camera work information definition section using the movement section information, and the camera work correspondence information update section is used in the camera work correspondence information update section. Information can be created.

The camerawork correspondence information described in this application may be stored in a memory in the apparatus, or stored in an external memory such as a memory card, USB memory, optical disk, or magnetic disk, or a personal computer. It may be a memory.
Also, if camerawork correspondence information is stored in an external memory or the like, and this camerawork correspondence information is used in another device, information on camerawork can be obtained under the same conditions even if the device is changed. Variation in quality can be reduced.

  The present invention is not limited to video cameras used at the time of shooting assuming that editing is performed in program production such as TV and movies. It can also be used in video cameras.

The block diagram which shows an example of the structure in one embodiment of this invention The figure which shows an example of the information management table which the information holding part in one embodiment of this invention hold | maintains The figure which shows an example of the coordinate axis fixed to the video camera in one embodiment of this invention The flowchart which shows an example of the imaging | photography procedure in one embodiment of this invention The flowchart which shows an example of operation | movement of the video recording control part in one embodiment of this invention The figure which shows an example of the motion of the video camera at the time of imaging | photography in one embodiment of this invention The figure which shows an example of the time change of the motion information acquired in the motion information acquisition part in one embodiment of this invention The figure which shows an example of the time change of the motion information acquired in the motion information acquisition part in one embodiment of this invention The flowchart which shows an example of operation | movement of the change point detection part in one embodiment of this invention. The flowchart which shows an example of operation | movement of the motion area information acquisition part in one embodiment of this invention. The figure which shows an example of the motion area information in one embodiment of this invention The figure which shows an example of the motion area information in one embodiment of this invention The figure which shows an example of the motion area information in one embodiment of this invention The figure which shows an example of the camera work corresponding | compatible relationship information in one embodiment of this invention The figure which shows an example of the camera work corresponding | compatible relationship information in one embodiment of this invention The figure which shows an example of the camera work corresponding | compatible relationship information in one embodiment of this invention The figure which shows an example of the camera work corresponding | compatible relationship information in one embodiment of this invention The flowchart which shows an example of operation | movement of the camera work information derivation | leading-out part in one embodiment of this invention The figure which shows an example of the camera work relaxation condition table | surface in one embodiment of this invention The flowchart which shows an example of operation | movement of the camera work evaluation part in one embodiment of this invention The figure which shows an example of the camerawork evaluation parameter | index in one embodiment of this invention The figure which shows an example of the mode of the evaluation of the camera work evaluation part in one embodiment of this invention The figure which shows an example of the information management table which the information holding part in embodiment of this invention hold | maintains The block diagram which shows an example of a structure in embodiment of this invention The flowchart which shows an example of operation | movement of the camera work information derivation | leading-out part in one embodiment of this invention The figure which shows an example of the information management table which the information holding part in one embodiment of this invention hold | maintains The flowchart which shows an example of operation | movement of the camera work information definition part in one embodiment of this invention The flowchart which shows an example of operation | movement of the camera work corresponding | compatible relationship information update part in one embodiment of this invention The figure which shows an example of the camera work corresponding | compatible relationship information in one embodiment of this invention

Explanation of symbols

1001 Imaging unit 1002 Video recording control unit 1003 Video holding unit 1004 Motion information acquisition unit 1005 Change point detection unit 1006 Video address acquisition unit 1007 Motion section information acquisition unit 1008 Camera work information derivation unit 1009 Camera work evaluation unit 1010 Information storage unit

Claims (11)

An information processing apparatus that detects movement caused by movement of an imaging apparatus and provides information at the time of image capturing using the detected information,
Imaging means for imaging video;
Video address giving means for giving a video address based on time series to the captured video frame;
Motion information detection means for detecting movement of the imaging means as motion information;
Based on the movement information detected by the movement information detection means, a change point detection means for detecting a change in the movement of the imaging means as a change point;
Video address acquisition means for acquiring a video address at the detected change point;
Holding means for holding the change point and the video address;
An information processing apparatus comprising: An information processing apparatus that detects movement caused by movement of an imaging apparatus and provides information at the time of image capturing using the detected information,
Imaging means for imaging video;
Video address giving means for giving a video address based on time series to the captured video frame;
Motion information detection means for detecting movement of the imaging means as motion information;
Change point detection means for detecting, as a change point, a time point at which the speed of the imaging means calculated based on the motion information detected by the motion information detection means becomes a predetermined value or a time point when the speed changes from the value;
Video address acquisition means for acquiring a video address at the detected change point;
Holding means for holding the change point and the video address;
An information processing apparatus comprising: Based on camera work correspondence information that associates motion information and camera work information, camera work information determining means for determining camera work information from motion information between detected change points,
The information processing apparatus according to claim 1, wherein the holding unit holds a video address between change points and camera work information in association with each other. Based on the camera work evaluation index for evaluating the motion information and camera work, the camera work evaluation means for determining the camera work evaluation between the change points from the motion information between the detected change points,
4. The information processing apparatus according to claim 3, wherein the holding unit further holds a video address between corresponding change points, camera work information, and camera work evaluation information in association with each other.   5. The information processing apparatus according to claim 4, further comprising the camera work evaluation index corresponding to each camera work.   The change point detection means calculates a moving speed of the image pickup means from motion information, and uses a time point when the moving speed changes from 0 or becomes 0 as a change point. 6. The information processing apparatus according to any one of 5 above.   The change point detecting means calculates a moving speed of the imaging means from motion information, and uses a time point when the moving speed is equal to or higher than a predetermined threshold value or a time point when the moving speed is equal to or lower than the threshold value as a changing point. The information processing apparatus according to any one of claims 1 to 5.   4. The information processing apparatus according to claim 1, further comprising an information update unit configured to update camerawork correspondence information based on information input from the outside. 5. An information processing method for detecting movement caused by movement of an imaging device and deriving information at the time of image capturing using the detected information,
An imaging step of imaging an image with an imaging device;
A video address giving step for giving a video address based on a time series to the captured video frame;
A motion information detection step of detecting movement of the imaging device as motion information;
Based on the motion information detected in the motion information detection step, a change point detection step for detecting a change in motion of the imaging device as a change point;
A video address acquisition step of acquiring a video address at the detected change point;
Holding step for holding the change point and the video address;
An information processing method comprising: An information processing program for detecting movement caused by movement of an imaging device and deriving information at the time of image shooting using the detected information,
An imaging step of imaging an image with an imaging device;
A video address giving step for giving a video address based on a time series to the captured video frame;
A motion information detection step of detecting movement of the imaging device as motion information;
Based on the motion information detected in the motion information detection step, a change point detection step for detecting a change in motion of the imaging device as a change point;
A video address acquisition step of acquiring a video address at the detected change point;
Holding step for holding the change point and the video address;
An information processing program characterized by comprising: An information processing device that derives information at the time of image capturing in response to movement information due to movement of the imaging device,
Receiving means for receiving a video frame in which a video address based on a time series is given to a video frame captured by an external imaging device and movement information due to movement of the external imaging device;
Motion information detection means for detecting movement of the imaging means as motion information;
Based on the movement information detected by the movement information detection means, a change point detection means for detecting a change in the movement of the imaging means as a change point;
Video address acquisition means for acquiring a video address at the detected change point;
Holding means for holding the change point and the video address;
An information processing apparatus comprising: