JP2008141354A - Image coding apparatus and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To resolve a problem where the data amount of coded image streams capable of being specially reproduced with high quality is apt to increase. <P>SOLUTION: A coding apparatus 100 for coding a picked-up moving image refers to additional pictures picked up separately from pictures for recording to adaptively control a frame rate of the pictures for recording. In this case, a change in feature quantity between at least two additional pictures inserted between the pictures for recording with a frame period shorter than a frame period of the pictures for recording is referred to, and the frame rate is changed to a frame rate higher than the frame rate of the pictures for recording when the change exceeds a set change amount. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image encoding device and an imaging device for encoding a captured moving image.

  Digital video cameras and the like have become widespread, and general users can easily capture moving images. On the other hand, apparatuses for decoding and reproducing encoded moving images have become widespread, and it has become possible to easily reproduce and edit captured moving images.

  Such a playback apparatus is generally equipped with a special playback function such as slow playback or fast forward playback. In order to perform high-quality slow reproduction, when capturing a moving image, encoded image data captured at a high speed, that is, with a high frame rate is required.

Patent Document 1 discloses an editing system in which playback data played back at high speed is temporarily recorded on a disk drive, played back, subjected to special effect processing, recorded on the disk drive again, played back at high speed, and recorded at high speed. Is disclosed.
JP 2003-317448 A

  The data amount of the encoded image stream imaged at high speed is larger than the data amount of the encoded image stream imaged at a frame rate corresponding to reproduction at the normal speed. In particular, a portable and small digital video camera has a strong demand for reducing the amount of data.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image encoding device and an imaging device capable of suppressing the enlargement of the data amount while enabling high-quality special reproduction. is there.

  In order to solve the above problems, an image encoding device according to an aspect of the present invention is an image encoding device that encodes a captured moving image, and refers to an additional picture captured separately from a recording picture. Then, the frame rate of the recording picture is adaptively controlled. “Picture” is a unit of encoding, and the concept may include a frame, a field, a VOP (Video Object Plane), and the like. The frame rate may be adaptively changed with reference to the feature amount of “additional picture”. The “feature amount” may include a motion vector amount. The frame period of the “additional picture” may be shorter than the frame period of the “recording picture”. An interpolation picture may be generated from a plurality of “additional pictures” and recorded.

  According to this aspect, by adaptively controlling the frame rate, the frame rate can be increased in a certain section and the frame rate can be decreased in another certain section. Therefore, it is also possible to generate a well-balanced encoded image stream that can suppress the enlargement of the data amount while enabling high-quality special reproduction. In addition, by referring to the feature amount of the additional picture, the frame rate can be searched without depending on the frame rate of the recording picture.

  When the change in the feature amount between the additional pictures taken at least two images with a frame period shorter than the frame period of the recording picture between the recording pictures and the change exceeds the set change amount, The frame rate may be changed to a frame rate faster than the frame rate of the recording picture. The “additional picture” may be set to the shortest frame period for detecting a high-speed motion of the subject to be changed to the fastest frame rate set in the system. According to this, it is possible to detect a high-speed movement of the subject that is difficult to detect from the recording frame.

  You may change the imaging interval of the additional picture imaged synchronizing with the frame period of the picture for recording. The number of “additional pictures taken in synchronization with the frame period of the recording picture” may be one or more. According to this, it is possible to avoid a situation in which the movement of the subject that moves periodically is missed.

  When determining the frame rate to be transitioned by referring to a plurality of sets of additional pictures each having a different imaging interval, a change amount of the feature amount between the additional pictures is obtained for each set of additional pictures, and the set change amount is obtained. The closest set may be specified, and the frame rate corresponding to the imaging interval of the set may be determined. According to this, the transition time to the optimum frame rate can be shortened as compared with the case where the transitionable operation points are shifted in stages.

  Another aspect of the present invention is an imaging apparatus. This apparatus includes an imaging unit that captures a moving image, and a control unit that adaptively controls the frame rate with reference to a feature amount of a picture captured by the imaging unit. The control unit causes the imaging unit to capture an additional picture used for determining the frame rate, separately from the recording picture.

  According to this aspect, by adaptively controlling the frame rate, the frame rate can be increased in a certain section and the frame rate can be decreased in another certain section. Therefore, it is also possible to generate a well-balanced encoded image stream that can suppress the enlargement of the data amount while enabling high-quality special reproduction. Further, by using the feature amount of the additional picture, the frame rate can be searched without depending on the frame rate of the recording picture.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, the enlargement of data amount can be suppressed, enabling high quality special reproduction | regeneration.

  FIG. 1 is a configuration diagram of an imaging apparatus 500 equipped with the encoding apparatus 100 according to the first embodiment. These configurations can be realized in hardware by a CPU, memory, or other LSI of an arbitrary computer, and in terms of software, it is realized by a program having an image encoding function loaded in the memory. So, functional blocks that are realized by their cooperation are drawn. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The imaging device 500 includes an imaging unit 400 and an encoding device 100. The imaging unit 400 includes an imaging element such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor, and a driver that drives the imaging element. The imaging unit 400 converts the captured image into an electrical signal and encodes the encoding device 100. Output to. The encoding apparatus 100 receives an input of a moving image in units of frames, encodes the moving image, and outputs an encoded stream.

  The encoding apparatus 100 according to the present embodiment includes MPEG (Moving Picture Experts Group) series standards (MPEG-1, MPEG-) standardized by ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission). 2 and MPEG-4), standardized by ITU-T (International Telecommunication Union-Telecommunication Standardization Sector) which is an international standard organization for telecommunications. 26x series standards (H.261, H.262 and H.263), or H.264, the latest video compression coding standard standardized jointly by both standards organizations. H.264 / AVC (official recommendation names in both organizations are MPEG-4 Part 10: Advanced Video Coding and H.264 respectively).

  In the MPEG series standard, an image frame for intra-frame encoding is an I (Intra) frame, an image frame for forward inter-frame predictive encoding with a past frame as a reference image, a P (Predictive) frame, and past and future An image frame that performs bidirectional inter-frame predictive coding using this frame as a reference image is referred to as a B frame.

  On the other hand, H. In H.264 / AVC, the frame that can be used as a reference image may be a past two frames as a reference image or a future two frames as a reference image regardless of the time. Further, three or more frames can be used as the reference image regardless of the number of frames that can be used as the reference image. Therefore, in MPEG-1 / 2/4, the B frame refers to a Bi-directional prediction frame. Note that in H.264 / AVC, the B frame refers to a bi-predictive prediction frame because the time of the reference image does not matter before and after.

In the specification of the present application, a frame and a picture are used in the same meaning, and an I frame, a P frame, and a B frame are also called an I picture, a P picture, and a B picture, respectively.
In this specification, a frame is used as an example of the encoding unit. However, the encoding unit may be a field. The unit of encoding may be a VOP in MPEG-4.

  The block generation unit 10 divides the image frame input from the imaging unit 400 into macro blocks. Macroblocks are formed in order from the upper left to the lower right of the image frame. The block generation unit 10 supplies the generated macroblock to the differentiator 12, the motion compensation unit 60, and the frame rate control unit 70.

  If the image frame supplied from the block generation unit 10 is an I frame, the differentiator 12 outputs it to the DCT unit 20 as it is, but if it is a P frame or a B frame, the difference image 12 is supplied from the motion compensation unit 60. Is calculated and supplied to the DCT unit 20.

  The motion compensation unit 60 uses a past or future image frame stored in the frame memory 80 as a reference image, and has the smallest error for each macroblock of the P frame or the B frame input from the block generation unit 10. A prediction area is searched from the reference image, and a motion vector indicating a deviation from the macroblock to the prediction area is obtained. The motion compensation unit 60 performs motion compensation for each macroblock using the motion vector, and generates a predicted image. The motion compensation unit 60 supplies the generated motion vector to the frame rate control unit 70 and the variable length encoding unit 90, and supplies the prediction image to the difference unit 12 and the adder 14.

  The motion compensation unit 60 can apply both bidirectional prediction and unidirectional prediction. In the unidirectional prediction, the motion compensation unit 60 generates a forward motion vector indicating the motion with respect to the forward reference frame. In the bi-directional prediction, in addition to the forward motion vector, two motion vectors of a backward motion vector indicating motion with respect to the backward reference frame are generated.

  The differentiator 12 obtains a difference between the current image output from the block generation unit 10, that is, the image to be encoded, and the predicted image output from the motion compensation unit 60, and outputs the difference to the DCT unit 20. The DCT unit 20 performs a discrete cosine transform (DCT) on the difference image given from the differentiator 12 and gives a DCT coefficient to the quantization unit 30.

  The quantization unit 30 quantizes the DCT coefficient and provides it to the variable length coding unit 90. The variable length coding unit 90 performs variable length coding on the quantized DCT coefficient of the difference image together with the motion vector supplied from the motion compensation unit 60, and generates a coded stream CS.

  The quantization unit 30 supplies the quantized DCT coefficient of the image frame to the inverse quantization unit 40. The inverse quantization unit 40 inversely quantizes the supplied quantized data and supplies the quantized data to the inverse DCT unit 50. The inverse DCT unit 50 performs inverse discrete cosine transform on the supplied inverse quantized data. Thereby, the encoded image frame is restored. The restored image frame is input to the adder 14.

  If the image frame supplied from the inverse DCT unit 50 is an I frame, the adder 14 stores it in the frame memory 80 as it is. If the image frame supplied from the inverse DCT unit 50 is a P frame or a B frame, the adder 14 is a difference image, and thus is supplied from the difference image supplied from the inverse DCT unit 50 and the motion compensation unit 60. By adding the predicted image, the original image frame is reconstructed and stored in the frame memory 80.

  In the case of P frame or B frame encoding processing, the motion compensation unit 60 operates as described above. However, in the case of I frame encoding processing, the motion compensation unit 60 does not operate and is not shown here. The I frame is supplied to the DCT unit 20 after intra prediction.

  The frame rate control unit 70 refers to the motion vector given from the motion compensation unit 60 and the color information and luminance information given from the block generation unit 10 to adaptively determine the frame rate when the imaging unit 400 captures a moving image. To control.

  FIG. 2 is a diagram illustrating the configuration of the frame rate control unit 70 according to the present embodiment. The frame rate control unit 70 includes a feature amount acquisition unit 72, a feature amount analysis unit 74, a condition setting unit 76, and a frame rate setting unit 78.

  The feature amount acquisition unit 72 acquires parameters indicating image features from the block generation unit 10 and the motion compensation unit 60. For example, color information (R, G, B) and luminance information (Y) are acquired from a macroblock given from the block generation unit 10. Also, a motion vector is acquired from the motion compensation unit 60.

  The feature amount analysis unit 74 analyzes the parameters acquired by the feature amount acquisition unit 72 according to the conditions set by the condition setting unit 76. For example, from the obtained motion vector, the movement amount in the x direction and the y direction of each macroblock is expressed by an absolute value, and the value obtained by adding the above movement amounts of all the macroblocks for each frame (hereinafter referred to in this specification). Observe the change in the vector quantity.) It is also possible to observe a change in the direction of the motion vector.

  The feature quantity analysis unit 74 can manage changes in various parameters as a history and perform statistical analysis. For example, when obtaining the magnitude of the increase / decrease tendency of the parameter to be observed, the state duration, the number of times the parameter exceeded the threshold, the frequency, etc., by removing the maximum value, the minimum value, Outliers can be eliminated. Based on these analyses, the feature amount analysis unit 74 instructs the frame rate setting unit 78 to change the frame rate so that the observed parameter falls within a predetermined range.

  The feature amount analysis unit 74 can also observe the change in the acquired color information and the change in luminance information. Also, by analyzing the motion vector, it is possible to observe the moving speed and moving distance of a subject such as a human being or an animal. Note that the motion vector and the movement speed of the subject can also be obtained from the autofocus control system.

  The condition setting unit 76 is a parameter type to be acquired by the feature amount acquisition unit 72, an analysis condition necessary for the feature amount analysis unit 74 to analyze the image, and a frame rate setting unit 78 that is necessary for setting the frame rate. Set appropriate conditions and parameters. The condition setting unit 76 can set weights for various parameters as image analysis conditions. For example, with respect to changes in the motion vector, color, brightness, and subject, the change in the central region of the image is more important than the change in the peripheral region, so that 1 for the change in various parameters in the macroblock near the center It is also possible to set so that a coefficient less than 1 is multiplied to a change in various parameters in the macroblocks in the surrounding area.

  The condition setting unit 76 can also limit the range itself in which changes in various parameters are to be observed. For example, changes in various parameters in macroblocks in the peripheral area can be ignored. It is also possible to limit the subject to be observed. For example, only a change in a subject determined as a human face by pattern matching may be observed.

  The condition setting unit 76 can perform mode setting that determines which of various parameters is used. Although the observation is centered on the amount of motion vector, it is possible to make a complex setting such as observing a color change as an auxiliary. In addition, as a final determination for changing the frame rate, it is possible to set which degree of contribution should be given to which parameter.

The condition setting unit 76 sets a basic frame rate, a transitionable frame rate, a transitionable range, a change period, and a threshold value indicating a change amount of the parameter to be changed. Specific examples will be described below. The condition setting unit 76 sets the frame rate for playback at normal speed to the basic frame rate. Note that the frame rate for slow playback or fast-forward playback may be set to the basic frame rate depending on the user's intention to shoot. As a frame rate transition range, a frame rate change at an integral multiple of the basic frame rate is used as a constraint condition, and a frame rate that can transition in advance under this condition is selected.
Basic frame period: T0
Frame period: t = T0 / N (N is a natural number)
Frame rate: f = 1 / t [fps]

  The basic frame period T0 is the reciprocal of the basic frame rate. The frame rate change period is set to a time interval that is an integral multiple of the basic frame period. The feature amount analysis unit 74 determines a frame rate to be changed next in units of video within this time interval. That is, it has a frame rate change point at a time interval synchronized with the basic frame period. The maximum value of the frame rate at the frame rate change point is set to a value that guarantees the minimum viewing time of each frame reproduced at the frame rate.

  If the basic frame rate is the minimum unit and the frame rate change point is set to an integral multiple of the basic frame rate, it is possible to determine at which frame rate the moving image data is captured even with a low-spec playback device that can only play back at a limited frame rate. Judgment can be easily made, and confusion in viewing and editing can be suppressed. Note that if it is allowed to provide a frame rate change point within the basic frame period, the setting conditions and timing control for dynamically changing the frame rate become complicated and editing becomes difficult. Can be reduced. Therefore, provision of a frame rate change point within the basic frame period is not excluded.

  The condition setting unit 76 can also set a trigger condition that triggers an adaptive change process of the frame rate. Set to activate when various parameters meet certain conditions such as detecting a specific color, when the moving speed or distance of the subject exceeds a preset value, or when there is an external input from the user Can do.

  The frame rate setting unit 78 adaptively changes the frame rate in accordance with an instruction from the feature amount analysis unit 74 in accordance with the conditions set in the condition setting unit 76. Specifically, the exposure time of the image sensor provided in the imaging unit 400 is controlled. The exposure time can be adjusted by instructing a driver that drives the image sensor.

  FIG. 3 is a diagram showing three types of frame periods t. In FIG. 3, the upper frame indicates a frame having the same frame period t as the basic frame period T0. The middle frame indicates a frame whose frame rate t is 1/4 (T0 / 4) of the basic frame period. The lower frame indicates a frame whose frame period t is 1/2 (T0 / 2) of the basic frame period. Therefore, it can be seen that the middle frame is imaged at 4 × speed and the lower frame is imaged at 2 × speed with respect to the upper frame imaged at the normal speed.

  FIG. 4 is a diagram illustrating a relationship between a change in motion vector amount and a change in frame rate. In FIG. 4, the upper uneven block indicates a change in the motion vector amount. When the photographer is shooting a certain point and the movement of the subject is small between successive frames, the amount of motion vector is also small. In the opposite case, the motion vector amount becomes large. When slow playback is performed in a section where the change in the image is large, if a frame group captured at a normal frame rate is played back, frames may be dropped and the image may not be smoothly connected. In this regard, if the frame rate of the section is increased and the number of frames is large, a smooth image can be reproduced.

  FIG. 4 shows an example in which this knowledge is embodied. Imaging starts at the normal frame rate, but the amount of motion vector at the basic frame period T0 corresponding to the period between the frame rate change timings is large, so the frame rate is changed to the quadruple speed frame rate at the next frame rate change timing. ing. Since the amount of motion vector is decreasing while the basic frame period T0 has passed twice, the frame rate is changed to the double speed at the next frame rate change timing. Thereafter, while the basic frame period T0 has passed three times, the amount of motion vector has decreased, so the frame rate is changed to the normal frame rate at the next frame rate change timing.

  FIG. 5A shows an arrangement of frame data on the time axis when imaging. FIG. 5B shows the arrangement of the frame data on the time axis when reproducing. In FIG. 5, frame data captured at a normal frame rate and frame data captured at a quadruple frame rate are mixed. As described above, in the present embodiment, the frame rate adaptively changes during capturing of a moving image. Therefore, as illustrated in FIG. 5A, the captured moving image stream has sections in which time intervals between frames are different. Will be mixed. On the other hand, in normal playback, frame data is played back at regular time intervals as shown in FIG. The frame data captured at high speed becomes high-quality slow motion video.

  FIG. 6 is a flowchart illustrating an example of a frame rate change process according to the first embodiment. First, the condition setting unit 76 sets a plurality of transitionable frame rates, that is, frame rate change points in the frame rate setting unit 78 (S10). Also, an initial frame rate is set (S12). The condition setting unit 76 sets a unit analysis period of the motion vector amount corresponding to the basic frame period T0 (S14). Also, an upper limit threshold and a lower limit threshold for the motion vector amount are set (S16). These threshold values can be set to values obtained by the designer through experiments and simulations. Further, the user may select from a plurality of threshold candidates obtained by the designer.

  The setting process from step S10 to step S14 may be performed by the user instructing the condition setting unit 76 from an operation unit (not shown) provided in the imaging device 500 or a PC connected to the imaging device 500, The system may do this automatically. Note that the set value once set may be taken over unless there is a change instruction from the user.

  The motion compensation unit 60 extracts the motion vector of each macroblock and provides it to the feature amount acquisition unit 72, and the feature amount analysis unit 74 calculates the motion vector amount from the motion vector supplied from the feature amount acquisition unit 72 (S18). ).

  When the unit analysis period of the motion vector amount ends (Y in S20), the feature amount analysis unit 74 detects how many times the motion vector amount has exceeded the upper limit threshold or the lower limit threshold within the period (S22). If the number of times the upper limit threshold or the lower limit threshold has been exceeded is less than the predetermined number set in advance (N in S22), the process proceeds to step S18 without changing the frame rate. The set predetermined number of times can also be set to a value obtained by the designer through experiments and simulations.

  In step S22, when the predetermined number of preset times has been exceeded (Y in S22), the feature amount analysis unit 74 determines that the current value exceeds the predetermined number of times by the lower threshold (Y in S24). The frame rate is changed to a lower direction (S26). When the upper limit threshold is exceeded the predetermined number of times (Y in S28), the current frame rate is changed in the high direction (S30). Even after the change, while the frame rate changing process continues (N in S32), the process proceeds to step S18 to continue the above-described process.

  In addition, when the change point of the frame rate to be changed in the low direction or the high direction no longer exists in the low direction or the high direction, the frame rate is not changed even when the predetermined number of times is exceeded. The frame rate change is basically a transition to adjacent change points in stages, but when the frequency of motion vector amounts exceeding the threshold is very high, for example, exceeding a predetermined number of times. The frame rate may be changed to a change point that is two or more steps away from the current operating point. Note that an upper limit threshold and a lower limit threshold may be set in advance for each change in frame rate.

  As described above, according to the present embodiment, by appropriately changing the frame rate while capturing a moving image, it is possible to suppress the enlargement of the data amount while enabling high-quality special reproduction. . Therefore, it is possible to shoot for a long time while enabling high-quality special reproduction.

  In addition, it is difficult to appropriately specify the timing for switching to high-speed imaging necessary for high-quality special reproduction by manual operation. This is because changes in the movement of the subject are difficult to catch by human recognition. Unnecessary high-speed imaging enlarges the amount of data and shortens the time that can be taken. In this respect, in the present embodiment, since the system adaptively changes the frame rate, such a situation hardly occurs.

  Next, Embodiment 2 will be described. In the second embodiment, a search frame for searching for an optimum frame rate is added. The configuration of the encoding apparatus according to the second embodiment is the same as that of the encoding apparatus 100 according to the first embodiment shown in FIGS. The search frame is inserted in various manners with respect to the imaging frame rate. For example, the insertion interval, the frame period, or the number inserted at one time may be changed while being inserted. The search frame is intended to search for a frame rate and does not need to be processed as a captured image for recording. Therefore, exposure adjustment is unnecessary or minimal. That is, it is only necessary to basically maintain the exposure adjusted for the frame rate currently being imaged.

  FIG. 7 shows an example of a search frame insertion pattern. In FIG. 7, a period between search frames is a recording frame. Here, the recording frame is a concept indicating a period between search frame periods or an image during that period, and includes a case where the period is shorter than a frame rate during imaging or a case where the period is longer. Moreover, the area which straddles several imaging frames may be sufficient. For convenience, the period or video between the search frames is expressed as a recording frame, but the recording frame merely indicates that it is not used for searching the frame rate, and the frame to be recorded However, this does not mean that the recording frame is limited to the recording frame, and does not necessarily mean that all the recording frames must be recorded. FIG. 7 shows an example in which search frames having the shortest period t0 are inserted in pairs, while the insertion interval is changed. The reason why two search frames having the shortest period t0 are continuously inserted is to detect high-speed movement of the subject. The shortest period t0 may be set to the shortest time that the imaging unit 400 can capture images, or may be arbitrarily set by the designer.

  In FIG. 7A, two first and second search frames A1 and A2 are successively inserted before the first recording frame B1, and the first recording frame B1 and the second recording frame B2 are inserted. Two third and fourth search frames A3 and A4 are successively inserted between the second and fifth recording frames A5 and A6 after the second recording frame B2. Inserted. Also, second and third search frames A2 and A3 are added before and after the first recording frame B1, and fourth and fifth search frames A4 and A5 are added before and after the second recording frame B2. It may be taken as. Note that the search frame may be an arbitrary short period extracted from the recording frame period. Also, in the case of continuous search frames, short periods of individual search frames may be extracted continuously from one search frame period. When recording frames to which search frames are added before and after are arranged, two search frames are consecutive.

  The first recording period B1 to which the second and third search frames A2 and A3 are added before and after, the first frame period t1 and the second recording to which the fourth and fifth search frames A4 and A5 are added before and after. The second frame period t2 of the work frame B2 is arbitrarily set by the designer. In FIG. 7A, the frame period that is the sum of the first frame period t1 and the second frame period t2 is set to be a normal-period frame period T0.

  In FIG. 7B, two seventh and eighth search frames A7 and A8 are successively inserted before the third recording frame B3, and two ninth ninth frames B3 are inserted after the third recording frame B3. The tenth search frames A9 and A10 are continuously inserted. The third frame period t3 of the third recording frame B3 to which the eighth and ninth search frames A8 and A9 are added before and after is set to be equal to the normal speed frame period T0.

  In FIG. 7C, two eleventh and twelfth search frames A11 and A12 are successively inserted in front of the fourth recording frame B4, and the fourth recording frame B4 and the fifth recording frame B5 are inserted. Two thirteenth and fourteenth search frames A13 and A14 are inserted in succession, and a fifteenth search frame A15 is inserted after the fifth recording frame B5. The frame period, which is the sum of the fourth frame period t4 and the fifth frame period t5, is set so as to be a normal frame period T0. In FIG. 7C, it is assumed that the search frame is not added to the recording frame next to the fifth recording frame B5.

  FIG. 8 shows an example of a data stream in which a search frame is inserted. The data stream shown in FIG. 8 transitions from a normal frame rate to a triple frame rate. In this data stream, a search frame for determining the frame rate is inserted. A search frame is inserted into the normal-speed frame with an insertion pattern in which the insertion patterns shown in FIGS. 7A to 7C are arranged in order from FIG. The sixth search frame A6, the seventh search frame A7, the tenth search frame A10, and the eleventh search frame A11 in FIG. 7 are an eighth search frame A8, a fifth search frame A5, and a twelfth search, respectively. The frame A12 and the eleventh search frame A11 are replaced. Note that a search frame may also be inserted into a triple speed frame. There may also be a frame in which no search frame is inserted.

  FIG. 9 is a flowchart illustrating an example of a frame rate change process according to the second embodiment. In FIG. 9, description will be made in accordance with the data stream example shown in FIG. First, the motion compensation unit 60 extracts the motion vector of each macroblock of the search frame and provides it to the feature amount acquisition unit 72, and the feature amount analysis unit 74 calculates the motion vector from the motion vector supplied from the feature amount acquisition unit 72. The amount is calculated (S40). Here, the motion compensation unit 60 calculates the motion vector amount of the search frame inserted before and after the recording frame.

  The feature amount analysis unit 74 compares the motion vector amounts of successive search frames and obtains the amount of change (S42). If no continuous search frame appears in the initial stage, the process skips to step S46. The feature amount analysis unit 74 determines whether or not the amount of change exceeds a predetermined threshold (S44). The predetermined threshold value can be set to a value obtained by the designer through experiments or simulations. When the predetermined threshold is exceeded (Y in S44), the current frame rate is changed to the fastest frame rate (S54).

  When the predetermined threshold is not exceeded (N in S44), the feature amount analysis unit 74 compares the motion vector amounts of the search frames inserted before and after the recording frame, and obtains the change amount (S46). While the unit evaluation period does not end (N in S48), the process proceeds to step S40, and the above-described processing is executed. Here, the unit evaluation period refers to a period in which the amount of change in the motion vector amount between search frames straddling recording frames having different frame periods is collected in order to search for an optimal frame rate.

  When the unit evaluation period ends (Y in S48), the feature amount analysis unit 74 determines the change amount closest to the preset appropriate change amount of the motion vector among the change amounts of the motion vector amounts collected during that period. A frame period is selected (S50). This appropriate change amount can also be set to a value obtained by the designer through experiments and simulations. It is set based on a request for reducing the amount of data and quality in the case of slow playback. For example, when importance is attached to reducing the amount of data, the appropriate change amount is set to be relatively high, and when importance is attached to quality, the appropriate change amount is set to be relatively low.

  The feature amount analysis unit 74 changes the current frame rate to a preset transitionable frame rate that is closest to the frame rate of the selected frame period (S52). When the frame period of the recording picture set within the unit evaluation period is matched with the frame period corresponding to the frame rate at which the system can transition, the frame rate of the frame period of the selected recording picture is simply set. Change it. When the subject is stationary, the amount of change in each motion vector amount is zero or very small. In such a case, processing for changing the current frame rate to the basic frame rate, that is, the normal frame rate may be added.

  Explaining in accordance with FIG. 8, the condition setting unit 76 sets a period obtained by combining the frame period t1 to the frame period t5 as a unit evaluation period. The feature amount analysis unit 74 changes the amount of change in the first motion vector amount between the second search frame A2 and the third search frame A3, and between the fourth search frame A4 and the fifth search frame A5. The change amount of the second motion vector amount, the change amount of the third motion vector amount between the eighth search frame A8 and the ninth search frame A9, and the twelfth search frame A12 and the thirteenth search frame A13 The amount of change in the amount of the fourth motion vector in the meantime and the amount of change in the amount of the fifth motion vector between the fourteenth search frame A14 and the fifteenth search frame A15 are obtained. If the change amount of the first motion vector amount is closest to the appropriate change amount, the moving image captured in the first frame period t1 is closest to the moving image captured at the optimal frame rate. Therefore, the feature amount analysis unit 74 changes the current frame rate to a preset frame rate that can be transitioned and is closest to the frame rate of the first frame period t1. That is, the optimum frame rate can be estimated from the evaluation of the period video of the first frame period t1 or the video of the search frame before and after the period video. This narrows down the search range to the optimum frame rate, and when the frame rate that can be changed is fixed, the frame rate is selected from among them, so that the evaluation is more direct and easy. Therefore, the rate change process can be estimated easily and appropriately. In FIG. 8, the frame rate is changed to the triple speed closest to the frame rate of the first frame period t1.

  FIG. 10 shows another example of the data stream in which the search frame is inserted. Frame periods t1 to t5 in FIG. 10 correspond to frame periods t1 to t5 in FIG. In a recording frame stream to which search frames are added before and after, a plurality of recording frames, for example, two recording frames having the same frame period are continued. Thereby, each frame period can be evaluated more accurately. For example, when the amount of change in the motion vector amount in a plurality of recording frames with the same frame period is approximate, the detected amount of change is treated as valid data, and when the amount of change is significantly different, the amount of change is treated as invalid data May be performed. By this process, the continuity of the movement of the subject can be ensured.

  FIG. 11 is an image diagram illustrating a process of generating an interpolation frame from a search frame. FIG. 11 shows a process of generating an interpolation frame from a data stream in which a search frame is inserted with the insertion pattern of FIG. Each of the ninth search frame A9, the tenth search frame A10, the fourth recording frame B4, the eleventh search frame A11, the twelfth search frame A12, and the fifth recording frame B5 included in this data stream. When the frame period is adjusted, it becomes the same as the normal frame period. Therefore, the normal-speed frame C1 can be generated by adding the pixel values of the coordinates corresponding to each frame.

  Specifically, the pixel px9 of the ninth search frame A9, the pixel px10 of the tenth search frame A10, the pixel px4 of the fourth recording frame B4, the pixel px11 of the eleventh search frame A11, and the twelfth search frame The pixel values of the pixel px12 of A12 and the pixel px5 of the fifth recording frame B5 are added up. The sum value is the pixel value of the pixel px1 of the interpolation frame C1. It should be noted that other interpolation frame generation methods may be used, such as adding together after performing a predetermined weighting on each pixel value, instead of simply adding up. When such an interpolation frame is generated, a plurality of images having different frame rates are held in the same section of the data stream, and can be used in various modes on the playback side. For example, when playback is performed at normal speed on the playback side, an interpolation frame generated by the above-described method can be used during a period during which high-speed imaging is performed. As a result, the image quality can be improved as compared with the case of using a frame taken at a high speed.

  As described above, the encoding apparatus according to the second embodiment has the following effects in addition to the effects of the first embodiment. By continuously inserting the shortest search frames between the recording frames, it is possible to detect a very fast movement of the subject. In this regard, there are cases where high-speed movement of the subject cannot be detected at the normal frame rate, and automatic adjustment of the frame rate may not be activated.

  Further, by changing the search frame insertion interval, it is possible to avoid a situation in which the movement of the subject cannot be detected due to the periodic movement of the subject. That is, if the subject is moving periodically in synchronization with the current frame rate, the system determines that the subject is stationary or has little movement. The same applies to the case where the search frame insertion interval is constant and the subject moves periodically in synchronization with the interval. In this regard, such a situation can be avoided by changing the search frame insertion interval.

  In addition, it is possible to further optimize the automatic adjustment of the frame rate by referring to the amount of change in the motion vector amount between the search frames straddling the recording frames having different frame periods and estimating the optimal frame rate. . In other words, it is possible to change to a frame rate far away from the vicinity of the current frame rate, and a wide range of frame rates can be used effectively. In addition, it is possible to optimize the time for imaging at a high frame rate, and to reduce useless data.

  The present invention has been described based on some embodiments. It should be understood by those skilled in the art that these embodiments are exemplifications, and that various modifications can be made to combinations of the respective components and processing processes, and such modifications are within the scope of the present invention. is there.

  For example, in the first and second embodiments, it indicates how many times the image is captured on a display unit such as a finder (not shown) of the imaging apparatus 500 during the period during which high-speed imaging is performed by the adaptive frame rate changing process described above. A message or the like may be displayed. Thereby, the adaptive change of the frame rate can be notified to the user in real time.

  In addition, a data stream in which a section composed of frames to which the search frame described in the second embodiment is added and a section composed of frames to which the search frame is not added may be generated. In a section composed of frames to which no search frame is added, the frame rate changing process described in the first embodiment may be performed. Further, the frame rate may be adaptively controlled with reference to the amount of change in the motion vector amount between adjacent recording frames. At that time, the method described in the second embodiment can be applied. Further, the frame rate change process may be stopped in a section composed of frames to which no search frame is added, and the frame rate change process may be performed in a section composed of frames added with the next search frame.

It is a block diagram of the imaging device carrying the encoding device which concerns on Embodiment 1. FIG. It is a figure explaining the structure of the frame rate control part which concerns on Embodiment 1. FIG. It is a figure which shows three types of frame rate periods t which concern on Embodiment 1. FIG. It is a figure which shows the relationship between the change of the motion vector amount which concerns on Embodiment 1, and the change of a frame rate. FIGS. 5A to 5B are diagrams showing the arrangement of frame data on the time axis when imaging and reproducing. 6 is a flowchart illustrating an example of a frame rate change process according to the first embodiment. FIGS. 7A to 7C are diagrams illustrating examples of search frame insertion patterns according to the second embodiment. It is a figure which shows an example of the data stream which inserted the frame for a search. 10 is a flowchart illustrating an example of a frame rate change process according to the second embodiment. It is a figure which shows another example of the data stream which inserted the frame for a search which concerns on Embodiment 2. FIG. FIG. 10 is an image diagram illustrating a process of generating an interpolation frame from a search frame according to the second embodiment.

Explanation of symbols

  10 block generation units, 12 differentiators, 14 adders, 20 DCT units, 30 quantization units, 40 inverse quantization units, 50 inverse DCT units, 60 motion compensation units, 70 frame rate control units, 72 feature quantity acquisition units, 74 feature quantity analysis unit, 76 condition setting unit, 78 frame rate setting unit, 80 frame memory, 90 variable length encoding unit, 100 encoding device, 400 imaging unit, 500 imaging device.

Claims (5)

An image encoding device that encodes a captured moving image,
An image coding apparatus characterized by adaptively controlling a frame rate of the recording picture with reference to an additional picture taken separately from the recording picture.   When a change in feature amount between additional pictures taken at least two images with a frame period shorter than the frame period of the record pictures is referred to between the record pictures, and the change exceeds a set change amount 2. The image encoding apparatus according to claim 1, wherein the frame rate is changed to a frame rate faster than a frame rate of the recording picture.   The image coding apparatus according to claim 1, wherein an imaging interval of the additional picture that is captured in synchronization with a frame period of the recording picture is changed. When determining a frame rate to be transitioned by referring to a plurality of sets of additional pictures each having a different imaging interval,
A feature amount change amount between additional pictures is obtained for each set of additional pictures, a set closest to the set change amount is specified, and a frame rate corresponding to an imaging interval of the set is determined. The image encoding device according to claim 1. An imaging unit that captures a moving image;
A control unit that adaptively controls a frame rate with reference to a feature amount of a picture imaged by the imaging unit,
The control unit causes the imaging unit to capture an additional picture used for determining a frame rate, separately from the recording picture.

Images