JP2012244501A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an encoding technique that achieves high-compression-rate encoding and smooth bidirectional reproduction of high picture quality of an image comprising a plurality of frames having continuity.SOLUTION: The plurality of frames are divided into GOPs, and an I frame in a GOP is subjected to intra-frame encoding. Then P, B frames are subjected to inter-frame predictive encoding for forward reproduction and inter-frame predictive encoding for backward reproduction. At this time, the P frame is subjected to the encoding such that the same decoding result is obtained in both the directions. The same encoding result is therefore obtained in both the directions with respect to the B frame, so encoded data is generated only in the forward direction. Then a GOP is composed of encoded data of the I, P, and B frames for forward reproduction and encoded data of the P frame for backward reproduction.

Description

  The present invention relates to an image processing apparatus and an image processing method for encoding an image composed of a plurality of frames having continuity.

  In recent years, digital archives that digitize tangible cultural assets as still images or moving images for storage and release have been actively performed. Especially for cultural assets that can be expressed as plane objects such as photographs, paintings, drawings, etc., it is possible to create very high-precision digital data by using digital cameras and digital scanners.

  On the other hand, when digital archiving is performed on objects that change their appearance depending on the viewing direction or the light source direction even if they are flat surfaces such as thin objects and thin film interference and structural colors, it is necessary to acquire a larger amount of data. is there. For this purpose, it is common to perform multi-viewpoint photography in which a subject is photographed from a plurality of viewpoint positions surrounding the subject. Multi-viewpoint shooting methods such as using multiple cameras, mounting the camera on an arm or rail and shooting while moving, setting the subject in a turntable and rotating continuously, etc. It has been known. In addition, the positional relationship between the camera and the subject is fixed, and continuous shooting is performed while changing the position, intensity, and color of the light source.

  The image data continuously photographed in this way is mainly reproduced and displayed on a computer. For example, in the case of data obtained by multi-viewpoint shooting, images are switched as if the subject in the screen is rotated in accordance with a user's instruction, and viewing is performed at a free angle.

  When such reproduction display is performed, it is necessary to capture a large amount of image data on a computer, and therefore it is desirable that the image data is compressed and encoded. There is a high possibility that images taken continuously for the same subject have a strong correlation with the preceding and following images. Therefore, a high compression rate can be realized by using the inter-frame predictive coding technique used for coding moving images such as H.264.

  Here, a general inter-frame prediction encoding technique will be described. Interframe predictive coding is a method of generating a predicted image based on a frame (hereinafter referred to as a reference frame) different from a frame of interest mainly in a moving image and coding a difference between the input image and the predicted image. In general, in the inter-frame predictive encoding technique, each frame to be encoded is classified into three types of I frame, P frame, and B frame according to the encoding method. The I frame is a frame that is encoded without using inter-frame prediction (hereinafter referred to as intra-frame encoding), and can be decoded without depending on other frames. The P frame uses the previous frame in time as a reference frame. The B frame uses the previous frame in time, the subsequent frame, or both as reference frames. In the P frame and the B frame, since only the difference between the input image and the predicted image needs to be encoded, it is possible to achieve a higher compression rate than the I frame. However, when decoding a P frame and a B frame, the reference frame needs to be already decoded. Therefore, it is necessary to start decoding of an image from any I frame. A frame from an I frame to a frame before the next I frame is called a GOP (Group Of Pictures) and is a processing unit in encoding or decoding.

  FIG. 17 is a diagram illustrating a configuration example of a moving image encoded by the inter-frame predictive encoding technique, and a rectangle represents each frame in the moving image. The alphabet in each frame indicates the encoding method, and the numbers indicate the reproduction order. An arrow indicates a frame reference relationship. For example, I0 indicates an I frame that is first decoded and displayed. P2 is decoded using I0 as a reference frame, and B1 is decoded using I0 and P2 as a reference frame. A frame from I0 to B15 constitutes one GOP, and I16 and each frame before the next I frame (not shown) constitute the next GOP.

  As a technique closely related to inter-frame prediction, there is a technique called motion compensation. This is a technique of using a frame image obtained by dividing a reference frame into several blocks and moving each by a certain amount when generating a predicted image in inter-frame prediction. The amount of movement of each block is called a motion vector, and this motion vector is also encoded at the time of encoding.

  The inter-frame prediction technique and motion compensation technique used for encoding a general moving image are specialized for reproduction at a constant speed in the time-series positive direction in order to achieve a high compression rate. On the other hand, when a multi-viewpoint image is reproduced and displayed on a computer, it is desirable that the user can continuously display the subject image in an arbitrary direction at an arbitrary speed. Such display corresponds to special reproduction such as reverse reproduction, slow reproduction, and high-speed reproduction in a moving image.

  As a method of realizing reverse playback from moving image data compression-encoded by general inter-frame prediction, all the frames in the GOP to be played back are decoded and held in the memory, and the desired playback order The method of reproducing according to is known. In addition, there has been proposed a method of generating a bit stream corresponding to a purpose from original moving image data by compression encoding when a user gives an instruction for high-speed playback or reverse playback (see, for example, Patent Document 1). In addition, a method has been proposed in which only the I frame is used during reverse playback to reduce the load during reverse playback (see, for example, Patent Document 2).

Japanese Patent Laid-Open No. 10-032787 JP 2004-289629 A

  However, with the conventional compression encoding method, it has been difficult to perform continuous image reproduction in both the forward direction and the reverse direction while realizing a high compression rate, high image quality, and a reduction in load on the reproduction device. .

  For example, in the method of decoding all the frames in the GOP to be played at once, a memory area for holding all the frames in the GOP in a decoded state, and all the frames in the GOP while maintaining an appropriate playback speed It is necessary to have a calculation speed that is sufficient to decode all of the above. Also, with the technique described in Patent Document 1, generation of a reverse playback bitstream is newly started when the user instructs reverse playback, so it is difficult to reverse the playback direction freely during playback. is there. In the technique described in Patent Document 2, the user can freely switch the playback direction during playback. However, since only the I frame is decoded during backward playback, the image quality is greatly degraded.

  The present invention has been made in view of the above-described problems, and provides an encoding technique that enables high-compression encoding and high-quality and smooth bidirectional reproduction of images composed of a plurality of frames having continuity. The purpose is to do.

  As a means for achieving the above object, an image processing apparatus of the present invention comprises the following arrangement.

  That is, an image processing apparatus that creates an encoded image by encoding an image made up of a plurality of frames shot while continuously changing the positional relationship between the subject and the imaging device, wherein the plurality of frames are subjected to intra-frame encoding. A first frame to be performed, a frame classification means for classifying the second frame to be subjected to inter-frame predictive coding, an intra-frame coding means to perform intra-frame coding for the first frame, and the second frame First inter-frame predictive encoding means for performing inter-frame predictive encoding in a frame sequence when the encoded image is reproduced in the forward direction for the frame, and the encoded image for the second frame. Second inter-frame predictive encoding means for performing inter-frame predictive encoding in a frame sequence when reproducing in a direction opposite to the direction, A first frame encoded by the encoding means, a second frame encoded by the first inter-frame predictive encoding means, and encoded by the second inter-frame predictive encoding means And a coded image creating means for creating the coded image with the second frame as a coding unit.

  According to the present invention, it is possible to provide an encoding technique that enables high-compression encoding and high-quality and smooth bidirectional reproduction of an image including a plurality of frames having continuity.

FIG. 2 is a block diagram showing a configuration example of an image processing apparatus according to the first embodiment. A block diagram showing a logical configuration example of an image encoding application in the first embodiment, A block diagram showing a logical configuration example of an image encoding unit in the first embodiment, FIG. 3 is a schematic diagram showing a method for capturing an original image in the first embodiment; The flowchart which shows the encoding process in 1st Embodiment, The figure which shows an example of the classification | category of each flame | frame in 1st Embodiment, and a reference relationship, The flowchart which shows the inter-frame prediction encoding process in 1st Embodiment, FIG. 2 is a block diagram showing a logical configuration example of an image reproduction application in the first embodiment. The figure which shows the GUI example for the image reproduction in 1st Embodiment, The flowchart which shows the image reproduction process in 1st Embodiment, A block diagram showing a logical configuration example of an image encoding application in the second embodiment, The block diagram which shows the logic structural example of the information addition part for direction change in 2nd Embodiment, The figure which shows an example of the classification | category of each flame | frame in 2nd Embodiment, and a reference relationship, The flowchart which shows the encoding process in 2nd Embodiment, A block diagram showing a logical configuration example of an image reproduction application in the second embodiment, The flowchart which shows the image reproduction process in 2nd Embodiment, It is a figure for demonstrating the encoding method of the moving image using the general inter-frame prediction technique.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The following embodiments do not limit the scope of the claims of the present invention, and all combinations of features described in the following embodiments are indispensable for constituting the present invention. Is not limited.

<First Embodiment>
Image Processing Apparatus First, the configuration of an image processing apparatus that performs encoding and decoding of an image composed of a plurality of continuous frames in the present embodiment will be described with reference to FIGS.

  FIG. 1 is a block diagram illustrating a hardware configuration example of an image processing apparatus according to the present embodiment. In the figure, reference numerals 102 to 107 are components of the image processing apparatus 101. 102 is a main bus, 103 is a UI unit, 104 is a CPU, 105 is a main memory, and 106 is a data storage unit such as an HDD or an SSD. Reference numeral 107 denotes a communication unit such as USB or Wi-Fi that can be connected to the external device 108 or a network.

  The encoding process in the present embodiment is realized as an application that operates on a general-purpose OS in the above configuration. Note that the general-purpose OS and application in the present embodiment can be realized by a software program created by a known technique. Therefore, the details of how the application operates on the general-purpose OS will be omitted.

  Hereinafter, the operation of the image processing apparatus according to the present embodiment will be described. First, an application stored in the data storage unit 106 is activated by a command from the CPU 104 and loaded in the main memory 105. Subsequently, the data of each frame of the original image stored in the data storage unit 106 is transferred to the main memory 105 via the bus 102 according to the processing in the application. Thereafter, the data of each frame encoded by the application is generated in the main memory 105 and stored in the data storage unit 106 via the bus 102. The user acquires information from the application at any time via the UI unit 103 and gives an instruction to the application.

  The original image data to be encoded can be stored in advance in the data storage unit 106 via the communication unit 107 from the external device 108 or the like. In addition, encoded image data generated by an application and stored in the data storage unit 106 can be extracted to the external device 108 via the communication unit 107.

  FIG. 2 is a block diagram illustrating a logical configuration of the image processing apparatus according to the present embodiment. In the figure, reference numeral 201 denotes an image encoding application that performs image encoding processing in the present embodiment. The image encoding application 201 includes an image encoding unit 202.

  FIG. 3 is a block diagram illustrating details of a logical configuration in the image encoding unit 202. In the figure, reference numeral 301 denotes an original image acquisition unit that acquires unencoded original image data from the data storage unit 106. Reference numeral 302 denotes a GOP determination unit that divides the acquired original image data into a plurality of GOPs as encoding units. This GOP is composed of a plurality of predetermined frames as in the conventional inter-frame predictive coding technique. 303 is a frame classification unit that classifies each frame in the GOP according to the encoding method, and each frame is classified as an internal prediction frame (I), a forward prediction frame (P), or a bidirectional prediction frame (B). Categorize. Reference numeral 304 denotes an encoding unit that controls each unit of 305 to 308 to be described later and encodes each frame in the GOP. Reference numeral 305 denotes an intraframe encoding unit that predictively encodes an I frame within a frame. Reference numeral 306 denotes an inter-frame predictive encoding unit that performs inter-frame predictive encoding on P frames and B frames according to a certain reproduction order. That is, the P frame is encoded by forward prediction using the correlation with the reference frame reproduced before the frame. The B frame is encoded by bi-directional prediction using the correlation with reference frames reproduced before and after the frame. Reference numeral 308 denotes a motion vector calculation unit that calculates a motion vector used in inter-frame prediction encoding. A frame decoding unit 309 decodes the frame data encoded by the encoding unit 304. Reference numeral 310 denotes an encoded image output unit that stores image data in which all frames are encoded in the data storage unit 106.

  Here, as the original image data in the present embodiment, it is assumed that an image of all 360 frames taken from the lateral direction with respect to the rotation axis while rotating the subject that is a still object one by one is used. FIG. 4 is a schematic diagram illustrating a method for capturing original image data according to the present embodiment. In the figure, 401 is a subject, 402 is a turntable on which the subject is placed, and 403 is a digital camera. Reference numeral 404 denotes a control device that controls the rotary base 402 and the digital camera 403 to automatically acquire and store 360-frame images continuously photographed for the subject 401. That is, a plurality of frames obtained by photographing while continuously changing the positional relationship between the subject and the imaging device are acquired. Each frame of the original image data continuously photographed in this way is assigned a serial number from 0 to 359 clockwise from the angle at which the image was first photographed. Since the subject rotates once in 360 frame shooting, frames 359 and 0 also have continuity.

  The original image data stored in the control device 404 can be stored in the data storage unit 106 by connecting the control device 404 to the image processing apparatus 101 as the external device 108. Alternatively, the turntable 402 and the digital camera 403 may be connected to the image processing apparatus 101, and an application for controlling the turntable 402 and the digital camera 403 may be driven on the image processing apparatus 101. In this case, the image processing apparatus 101 replaces the control apparatus 404, and the captured image is directly stored in the data storage unit.

Image Encoding Process Hereinafter, the image encoding process performed by the image encoding application 201 will be described in detail with reference to the flowchart of FIG.

  First, in S501, the original image acquisition unit 301 acquires necessary information such as the image size and storage format in addition to the total number of frames as information of the original image data to be encoded. Next, in S502, the GOP determination unit 302 divides the original image data into a plurality of encoding units, that is, GOPs. In the present embodiment, it is assumed that original image data of 360 frames is divided into 24 GOPs. Each GOP consists of 16 frames, and the frames at both ends of the GOP are shared with adjacent GOPs. That is, frames 0 to 15 constitute the first GOP, and frames 15 to 30 constitute the next GOP. The last GOP is composed of frames 345-359 and 0.

  Next, in S503, loop control for performing the following processing of S504 to S507 is performed on each GOP divided in S502.

  In S504, the frame classification unit 303 classifies each frame in the GOP into one of the I / P / B frames, thereby determining an encoding method for each frame. Specifically, first, frames at both ends of the GOP are set as I frames. Subsequently, the remaining frames are set to one of an I frame, a P frame, and a B frame according to factors such as image quality and data compression rate. Here, FIG. 6 shows an example of the classification of each frame constituting the first GOP of this embodiment and the reference relationship between frames at the time of encoding. The rectangle in the figure represents each frame in the image, the alphabet (I / P / B) frame type in each frame, that is, the encoding method, and the number represents the frame number. An arrow indicates a frame reference relationship. In this embodiment, as shown in FIG. 6, frames at both ends of the GOP are set as I frames, and among the remaining frames, even-numbered frames are set as P frames and odd-numbered frames are set as B frames. The upper arrow indicates the reference relationship between the P frame and the B frame that are encoded clockwise in S506, which will be described later. Further, the lower arrow in FIG. 6 indicates the reference relationship of the P frame that is encoded counterclockwise in S507 described later.

  Next, in S505, the intraframe encoding unit 305 encodes each I frame in the GOP using a known intraframe encoding algorithm. In constructing the present invention, the intraframe encoding algorithm to be used is not particularly limited, and may be irreversible compression or reversible compression. Alternatively, compression may not be performed. After that, in S506, the inter-frame predictive encoding unit 306 performs P frame and B frame encoding processing (first inter-frame predictive encoding) for reproducing the image in the clockwise direction (forward direction). In S507, the inter-frame prediction encoding unit 306 performs P-frame encoding processing (second inter-frame prediction encoding) for reproducing the image in the counterclockwise direction (reverse direction).

  In the present embodiment, when a plurality of types of encoding using different reference frames are performed on a certain P frame, the decoding results are the same or encoding such that the error is small enough to be ignored. Use the algorithm. For example, if the algorithm is such that the difference between the input image and the predicted image is losslessly encoded, the decoding results are the same for the same P frame regardless of the playback direction. In this case, for example, the decoding result of the P2 frame encoded using the I0 frame as the reference frame and the decoding result of the P2 frame encoded using the P4 frame as the reference frame can be regarded as the same. Therefore, since the decoding result of the I frame and the P frame, which are reference frames when encoding each B frame, is constant regardless of the playback direction, the encoding result of the B frame is also constant regardless of the playback direction. It becomes. Therefore, in the present embodiment, it is assumed that only the P frame is encoded in S507, and the I frame and B frame are set to use the clockwise encoded data generated in S506 during the counterclockwise reproduction. Details of the inter-frame predictive encoding process in S506 and S507 will be described later.

  The encoded data of the GOP is completed by combining the encoded data of each frame created by the intra-frame encoding process in S505 and the inter-frame predictive encoding process in S506 and S507, respectively. That is, it is necessary to configure the GOP by merging the encoded data for clockwise playback and the encoded data for counterclockwise playback of the P frame. For example, the encoded data for counterclockwise reproduction of the P frame may be merged so as to be continuous with the encoded data for clockwise rotation.

  Note that the GOP is not limited to the above configuration as long as the encoded data of the counterclockwise playback P frame created in S507 is associated with the clockwise playback P frame created in S506. For example, a GOP is composed only of encoded data for clockwise playback, and encoded data for counterclockwise playback may be stored as a separate data file associated with encoded data for clockwise playback. good.

  When the encoding process in S504 to S507 is performed on all GOPs, the loop control in S503 is terminated. In step S508, the encoded image output unit 310 accumulates the generated encoded data of each GOP as an encoded image and stores it in the data storage unit 106.

Interframe Predictive Encoding Hereinafter, the interframe predictive encoding process in S506 and S507 will be described in detail with reference to the flowchart of FIG. FIG. 7 shows a predictive encoding process of an intra-GOP frame for a certain reproduction direction. That is, in S506, the process shown in the flowchart of FIG. 7 is executed in the clockwise direction (forward direction), and then in S507, the process is executed in the counterclockwise direction (reverse direction). Therefore, in a certain GOP, first, encoded data including I, B, and P frames is generated in the forward direction, and then encoded data including only P frames is generated in the reverse direction.

  First, in S701, loop control for performing the following processes in S702 to S708 is performed on each frame in the GOP. At this time, the processing order of frames is not particularly limited, but from the viewpoint of calculation load, it is desirable to process the frames in the clockwise order in S506 and the counterclockwise order in S507. Alternatively, the P frame may be encoded first, and then the B frame may be encoded.

  In S702, it is determined whether or not the processing target frame has already been encoded. If the processing target frame has been encoded, the process returns to S701 to process the next frame. If it has not been encoded yet, the process proceeds to S703. Here, as an example in which the processing target frame is already encoded, the processing target frame is an I frame, which may be already encoded in S505 or another GOP. Further, when other frames are encoded, they may be encoded in S705, which will be described later, as the reference image. Furthermore, when this flowchart is the process in S507, it is determined that the B frame has been encoded. This is because, as described above, in the present embodiment, the clockwise B encoded B frame data generated in S506 is used during counterclockwise reproduction.

  Next, in S703, loop control for performing the processing of S704 to S706 is performed on all reference frames used for encoding the processing target frame. It is assumed that the reference frame setting (frame reference relationship) corresponding to each frame is held in advance in inter-frame prediction encoding section 306.

  In S704, it is determined whether or not the reference frame has already been encoded. If the reference frame is not encoded, the process proceeds to S705 to encode the reference frame. Here, the processing from S703 to S708 is recursively executed with the reference frame as the processing target frame. On the other hand, if the reference frame has been encoded in S704, the process proceeds to S706, and the interframe predictive encoding unit 306 drives the frame decoding unit 309 to decode the encoded data of the reference frame. Since the decoding process used in the conventional inter-frame predictive coding technique can be applied to this decoding process, detailed description is omitted here.

  When the decoding process in S704 to S706 is performed on all reference frames, the loop control in S703 ends. In step S707, the inter-frame prediction encoding unit 306 drives the motion vector calculation unit 308 to generate a motion vector. Since a known motion compensation technique can be applied as the motion vector generation process, detailed description thereof is omitted here.

  Next, in S708, encoded data of the processing target frame is generated. Specifically, predicted frame image data is generated from the decoded data of the reference frame generated in S706 and the motion vector generated in S707, and the difference between the original image data and the predicted frame image data of the processing target frame is encoded. A set of the encoded difference data and motion vector data becomes encoded data of a processing target frame (B, P frame).

  In the present embodiment, as described above, inter-frame predictive coding is performed in consideration of the image reproduction direction. Note that encoded data of an image generated by the processing described above also constitutes the present invention.

Image Reproducing Device Hereinafter, a process for reproducing image data encoded by the above-described interframe predictive encoding process will be described.

  The image reproduction process in the present embodiment is realized as an image reproduction application that operates on a general-purpose OS in the configuration of the image processing apparatus shown in FIG. 1 described above.

FIG. 8 is a block diagram showing a logical configuration example of the image playback application 1001 in the present embodiment. 1002 is a display device capable of sRGB display, and is connected to the hardware as an external device 108.
Reference numerals 1003 to 1005 are components of the image reproduction application 1001. Reference numeral 1003 denotes a frame data acquisition unit that acquires frame data of an encoded image from the data storage unit 106. Reference numeral 1004 denotes a frame decoding unit that decodes the frame data acquired by the frame data acquisition unit 1003, and realizes substantially the same function as the frame decoding unit 309 shown in FIG. Reference numeral 1005 denotes an image display unit which displays an application GUI and a decoded frame on the external monitor 1002 in accordance with an instruction from the UI unit 103. In addition, it instructs the frame decoding unit 1004 to perform decoding processing.

  Assume that the image playback application 1001 can secure a memory area on the main memory 105 that can store at least three frames of decoded frames. Note that if the format of the encoded data is different from that of the present embodiment, the necessary memory area may increase or decrease. For example, when the encoded data is composed only of I frames and P frames, operation is possible if a memory area for two frames is secured. When frames have a more complicated reference relationship, for example, a memory area of 4 frames or more may be required.

● Image playback UI
Here, the UI of the image reproduction application 1001 in the present embodiment will be described with reference to FIG. In the drawing, reference numeral 1101 denotes a drawing area of the image reproduction application 1001 displayed on the external monitor 1002. An image display area 1102 displays one of the decoded frames. 1103 is a clockwise playback button for instructing to play back an image in the clockwise direction, and 1104 is a counterclockwise playback button for instructing to play back the image in the counterclockwise direction. Reference numeral 1105 denotes a playback stop button for instructing to stop the image at the frame being displayed. Hereinafter, 1103 to 1105 are collectively referred to as playback instruction buttons. Reference numeral 1106 denotes an end button for instructing the end of image reproduction.

Image Reproduction Processing Hereinafter, image reproduction processing performed by the image reproduction application 1001 of this embodiment will be described with reference to the flowchart of FIG.

  First, in S1201, the image display unit 1005 displays any I frame in the image as an initial frame in the image display area 1102. In the present embodiment, it is assumed that frame 0 is displayed as the initial frame. Specifically, encoded data of the 0th frame is acquired by the frame data acquisition unit 1003, and the acquired encoded data is decoded by the frame decoding unit 1004 and stored in the main memory 105. The image display unit 1005 displays the decoded frame data stored in the main memory 105 in the image display area 1102.

  In step S1202, the image display unit 1005 determines a frame (playback frame) to be displayed next in the image display area 1102 in accordance with the playback instruction button that is pressed last. Specifically, when the playback instruction button that was pressed last is the playback stop button 1105, the currently displayed frame is determined as the playback frame. In the case of the clockwise playback button 1103, the frame that is adjacent to the currently displayed frame in the clockwise direction is set as the playback frame, and in the case of the counterclockwise playback button 1104, it is adjacent to the currently displayed frame in the counterclockwise direction. Let the frame be a playback frame.

  Next, in S1203, the frame decoding unit 1004 determines whether or not the playback frame determined in S1202 has been decoded. That is, if the decoded frame data of the playback frame is already stored in the main memory 105, the process proceeds to S1206 as having been decoded, but otherwise the process proceeds to S1204.

  In S1204, the frame decoding unit 1004 discards the decoded frame data stored in the main memory 105 that is not necessary for decoding the playback frame, and is necessary for storing the decoded frame data of the playback frame and its reference frame. Secure areas.

  For example, assume that I0, B1, and P2 are reproduced in order in the frame sequence shown in FIG. 6 and decoded frame data for three frames is stored in the main memory 105. At this time, when decoding B3 as a playback frame, an area for storing the decoded frame data for three frames of B3 and its reference frames P2 and P4 is required. On the other hand, the decoded frame data of I0 and B1 are unnecessary. Therefore, the frame decoding unit 1004 deletes the data of I0 and B1 from the main memory 105, and secures an area for storing the decoded frame data of B3 and P4.

  If more storage area of decoded frame data can be secured on the main memory 105, it is not necessary to immediately discard the decoded frame data of a frame that is no longer needed. By holding more decoded frame data after reproduction on the main memory 105, the decoded frame data can be reused when the reproduction direction changes, and the calculation load is reduced.

  In step S1205, the frame decoding unit 1004 performs playback frame decoding processing. For this decoding process, a decoding process used in the conventional inter-frame prediction encoding technique is applied. At this time, if the reference frame of the reproduction frame is not decoded, the reference frame is first decoded by the same means, and then the reproduction frame is decoded. As for the reference frame setting (frame reference relationship) corresponding to each frame, the same content as that referenced in the inter-frame prediction encoding unit 306 at the time of encoding is held in advance in the frame decoding unit 1004. Shall. The decoded frame data of the reproduction frame and the reference frame is stored in the main memory 105.

  In S1206, the image display unit 1005 displays the decoded frame data of the playback frame in the image display area 1102. In step S1207, the image display unit 1005 determines the end of the reproduction process. That is, if the end button 1106 is pressed during the processes of S1202 to S1207, the series of processes is terminated, and if not, the process returns to S1202 to determine the next playback frame.

  As described above, by repeating the processing of S1202 to S1207 at an appropriate speed, it is possible to reproduce an image while arbitrarily switching the reproduction direction. That is, the playback frame determined in S1202 is sequentially updated at a predetermined timing according to the playback direction, and playback frame update stops when the playback stop button 1105 is pressed.

  As described above, according to the present embodiment, two encoded data corresponding to the reproduction direction of an image are generated for one P frame. As a result, it is possible to continuously reproduce the image of the subject in any reproduction direction on a reproduction device (for example, the external monitor 1002). At that time, since a well-known inter-frame predictive coding technique can be applied to the encoding and decoding of each frame, an image quality equivalent to the well-known moving picture coding technique should be realized regardless of the reproduction direction. Can do. According to the encoding method of the present embodiment, the data amount of the P frame is doubled as compared with the conventional moving image encoding method, but the data amount of the I frame and the B frame is the same as the conventional method. . Therefore, in this embodiment, it is possible to realize a data compression rate close to that of the conventional moving image encoding method.

  Further, when the encoded data in the present embodiment is reproduced, the number of frames to be decoded at the time of reproduction of a certain frame is two at the maximum, so that the conventional moving image encoding technique is also applied to the calculation load for the decoding process. It is possible to suppress to the same level. Further, in the playback device, it is sufficient to prepare a memory area for at least 3 frames in order to store the decoded frame data, so that the load on the playback device is also suppressed.

  In the present embodiment, as shown in FIG. 4, an example of encoding original image data obtained by continuously shooting the subject 401 in the clockwise direction, that is, in the horizontal direction has been described. However, the subject 401 is continuously shot in the vertical direction or the oblique direction. It is also possible to do. Also, the locus of continuous shooting is not necessarily a straight line, and the digital camera 403 can also perform continuous shooting while drawing a curve. Therefore, not only the clockwise / counterclockwise direction as the playback direction for a certain frame. Further, there may be a case where a plurality of directions exist. When continuous shooting is performed in a plurality of directions on the same subject, an encoded image may be created so that bidirectional playback in the forward direction and the reverse direction can be performed in one continuous shooting unit. . At the time of playback, for example, it is necessary to increase the direction that can be specified by the playback instruction button in the image playback UI shown in FIG. 9, but setting and decoding the playback frame according to the specified playback direction is the same as in this embodiment. It is realizable by this method.

Second Embodiment
Hereinafter, a second embodiment according to the present invention will be described. In the first embodiment described above, when a plurality of types of encoding using different reference frames are performed on a certain P frame, the image is encoded under such a condition that the decoding results can be regarded as the same. The example which performs a process and a decoding process was shown. However, when pursuing a higher compression ratio and a reduction in calculation load of encoding and decoding processing, it is conceivable to adopt another encoding algorithm. Therefore, in the second embodiment, when a plurality of types of encoding using different reference frames are performed on a certain P frame, an example of using an encoding algorithm that causes a significant difference between the decoding results is used. Indicates.

Image Processing Device Hereinafter, a configuration of an image processing device that performs encoding and decoding of an image including a plurality of continuous frames in the second embodiment will be described. The hardware configuration of the image processing apparatus in the second embodiment is the same as that of the image processing apparatus 101 shown in FIG.

  FIG. 11 is a block diagram illustrating a logical configuration of the image processing apparatus according to the second embodiment. In the figure, reference numeral 1301 denotes an image encoding application for performing the image encoding process in the second embodiment. The image encoding application 1301 includes an image encoding unit 202 and a direction changing information adding unit 1302 that provides a characteristic function in the second embodiment. Note that the logical configuration of the image encoding unit 202 is the same as the configuration shown in FIG. 3 in the first embodiment described above, and a description thereof will be omitted.

  FIG. 12 is a block diagram showing details of a logical configuration in the direction changing information adding unit 1302. In the figure, reference numeral 1401 denotes an encoded image acquisition unit that acquires encoded image data from the data storage unit 106. 1402 is a direction change information generation that generates difference information between the playback frames for the same frame in the encoded data for clockwise reproduction and the encoded data for counterclockwise reproduction as information for direction change used at the time of direction change. Part. The frame decoding unit 309 decodes the encoded frame data in the same manner as the frame decoding units 309 and 1004 in the first embodiment described above.

  Note that the original image data to be encoded in the second embodiment is an image of all 360 frames obtained by rotating a subject that is a still object one by one, as in the first embodiment described above. To do.

Image Encoding Process The image encoding process performed by the image encoding application 1301 of the second embodiment will be described below.

  Here, FIG. 13 shows an example of the classification and reference relationship of each frame constituting the first GOP of the second embodiment. As shown in the figure, also in the second embodiment, inter-frame prediction encoding using I, P, B frames is performed as in the first embodiment. However, since the decoding result of the P frame in the second embodiment differs depending on the reproduction direction, the B frame is encoded also in the counterclockwise reproduction data. Therefore, in order to distinguish the difference in the decoding result depending on the reproduction direction, in FIG. 13, “−R” is attached to the frame reproduced clockwise, and “−L” is attached to the frame reproduced counterclockwise.

  For the motion vector of the B frame, common data may be used during clockwise playback and counterclockwise playback. In the second embodiment, although there are differences in the decoding result of the reference frame depending on the reproduction direction, they are generated from the same original frame, and the position of the subject in the frame does not change. Therefore, it can be considered that the motion vector of the B frame is constant regardless of the reproduction direction, so that common data can be used regardless of the reproduction direction.

  Also, the dotted arrows in the middle of FIG. 13 indicate the operational relationship based on the difference information (direction change information) between the decoded image during clockwise playback and the decoded image during counterclockwise playback of each P frame. In the second embodiment, by adding this difference information to the decoded image during clockwise rotation of each P frame, a decoded image during counterclockwise reproduction can be generated. Further, by subtracting the difference information from the decoded image at the time of counterclockwise reproduction, a decoded image at the time of clockwise rotation can be generated.

  Hereinafter, the image encoding process in the second embodiment will be described in detail with reference to the flowchart of FIG.

  First, in S1601, the image encoding unit 202 inputs original image data and generates an encoded image. The details of the encoded image generation process are the same as the process shown in the flowchart of FIG. 5 in the first embodiment described above, except that the B frame encoding process is also performed on the counterclockwise reproduction data. Therefore, description of the details of the encoded image processing in S1601 is omitted.

  Next, in S1602, the direction changing information adding unit 1302 performs loop control for performing the processes of S1603 to S1606 on all frames of the encoded image.

  In S1603, the direction change information generation unit 1402 determines whether the encoded frame to be processed acquired by the encoded image acquisition unit 1401 from the data storage unit 106 is an I frame. Since there is no difference in the decoding result depending on the reproduction direction in the case of an I frame, there is no need to generate difference information for direction change. Therefore, if the processing target frame is an I frame, the process returns to S1602 to start processing the next frame, but if it is not an I frame, the process proceeds to S1604. In S1604, it is determined whether or not the processing target frame is a reference frame of another frame. For example, if it is not a reference frame of another frame such as a B frame, the difference information need not be generated, and the process returns to S1602. On the other hand, if the frame is a reference frame of another frame such as a P frame, the process proceeds to S1605.

  In S1605, the direction change information generation unit 1402 drives the frame decoding unit 309 to decode the processing frame in both the clockwise and counterclockwise playback directions. In S1606, the direction change information generation unit 1402 subtracts the decoded data at the clockwise playback from the decoded data at the counterclockwise playback, and uses the obtained difference as the information for the direction change. The direction change information is compressed using a known encoding technique and stored in the data storage unit 106. The direction change information may be stored as a separate data file associated with the encoded image data, or may be stored as a new format image data file obtained by merging the encoded image data and the direction change information. May be.

Image Reproduction Processing Hereinafter, image data reproduction processing encoded by the above-described image encoding method according to the second embodiment will be described. The image reproduction process in the second embodiment is realized as an image reproduction application that operates on a general-purpose OS in the same configuration as the image processing apparatus 101 shown in FIG. 1 in the first embodiment.

  FIG. 15 is a block diagram illustrating a logical configuration example of the image playback application 1701 in the second embodiment. Reference numeral 1702 denotes a direction change operation unit that decodes the direction change information and adds or subtracts the decoded frame data on the main memory 150. Since the other configuration of the image playback application 1701 is the same as that of the image playback application 1001 shown in FIG. 8 of the first embodiment, the description thereof is omitted. Also, the GUI of the image playback application 1701 in the second embodiment is the same as that shown in FIG. 9 in the first embodiment, and thus the description thereof is omitted.

  Hereinafter, the image reproduction processing in the second embodiment will be described with reference to the flowchart of FIG. The image reproduction process in the second embodiment is substantially the same as the process shown in the flowchart of FIG. 10 in the first embodiment, but the details of the reproduction frame decoding process in S1205 are different. As a decoding process in S1205, in the first embodiment, an example in which the frame decoding unit 1004 performs a decoding process corresponding to a conventional interframe predictive coding technique has been shown. In the second embodiment, in order to perform the process of applying the direction change information to the reference frame when the reproduction direction is changed, the decoding process as shown in FIG. 16 below is performed.

  FIG. 16 is a flowchart showing decoding processing of a playback frame in the second embodiment. This processing is mainly performed in the frame decoding unit 1004.

  First, in S1801, it is determined whether or not the playback frame is an I frame. If the reproduction frame is an I frame, the process proceeds to S1808. If the reproduction frame is not an I frame, the process proceeds to S1802. In S1802, the frame reference relationship of the reproduction frame held in advance in the frame decoding unit 1004 is acquired, and a reference frame necessary for decoding is determined.

  Next, in S1803, loop control for performing the processing of S1804 to S1807 is performed on all reference frames necessary for decoding. In S1804, it is determined whether or not the reference frame has been decoded. If it has been decoded, the process proceeds to S1806. If it has not been decoded, the process proceeds to S1805 to decode the reference frame. The decoding process used in the conventional inter-frame prediction encoding technique is applied to the decoding here.

  In S1806, for the reference frame that has been decoded, it is determined whether or not the playback direction at the time of decoding is the same as the current playback direction. If they are the same, the process returns to S1803 to start the process of the next reference frame. However, if the playback direction is different, that is, changes from the time of decoding, the process moves to S1807. In S1807, the direction changing operation unit 1702 applies the direction change difference information to the decoded frame data of the reference frame stored in the main memory 105. That is, when the current reproduction direction is counterclockwise (that is, the reproduction direction at the time of decoding the reference frame is clockwise), the direction change information is added to the decoded frame data. On the other hand, when the current reproduction direction is clockwise (that is, the reproduction direction at the time of decoding the reference frame is counterclockwise), the direction change information is subtracted from the decoded frame data.

  In step S1808, the playback frame is decoded. The decoding process used in the conventional inter-frame prediction encoding technique is applied to the decoding here.

  As described above, according to the second embodiment, even when an encoding algorithm is used in which the decoding results for P frames are not the same depending on the playback direction, the subject image is continuously displayed in an arbitrary playback direction. It can be played back. Therefore, it is possible to perform encoding using an encoding algorithm that realizes high compression ratio and reduction of reproduction addition without being restricted by the condition that the decoding results in the reproduction direction are the same.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (10)

An image processing device that creates an encoded image by encoding an image composed of a plurality of frames taken while continuously changing a positional relationship between a subject and an imaging device,
Frame classification means for classifying the plurality of frames into a first frame that performs intra-frame coding and a second frame that performs inter-frame predictive coding;
Intra-frame coding means for performing intra-frame coding for the first frame;
First inter-frame predictive encoding means for performing inter-frame predictive encoding in a frame sequence when reproducing the encoded image in the forward direction for the second frame;
Second inter-frame predictive encoding means for performing inter-frame predictive encoding in a frame sequence when reproducing the encoded image in the direction opposite to the forward direction for the second frame;
A first frame encoded by the intra-frame encoding means, a second frame encoded by the first inter-frame predictive encoding means, and a code by the second inter-frame predictive encoding means A coded image creating means for creating the coded image with the second frame converted into a coding unit;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the first and second inter-frame predictive encoding units generate encoded data such that the decoding results of the same frame are the same. The frame classification means reproduces the second frame further in a forward prediction frame encoded by prediction using a correlation with a reference frame reproduced before the frame, and before and after the frame. Classify into bi-directional prediction frames encoded by prediction using correlation with reference frames,
The first inter-frame prediction encoding means creates encoded data obtained by encoding the forward prediction frame and encoded data obtained by encoding the bidirectional prediction frame,
The image processing apparatus according to claim 2, wherein the second inter-frame prediction encoding unit creates only encoded data obtained by encoding the forward prediction frame. Furthermore, it has difference information generating means for generating difference information between reproduced frames obtained by reproducing encoded data of the same frame by the first and second inter-frame predictive encoding means,
The image processing apparatus according to claim 1, wherein the coded image creating unit creates the coded image by adding the difference information to the coding unit. The frame classification means reproduces the second frame further in a forward prediction frame encoded by prediction using a correlation with a reference frame reproduced before the frame, and before and after the frame. Classify into bi-directional prediction frames encoded by prediction using correlation with reference frames,
The first and second inter-frame prediction encoding means create encoded data obtained by encoding the forward prediction frame and encoded data obtained by encoding the bidirectional prediction frame,
The image processing apparatus according to claim 4, wherein the difference information generation unit generates the difference information for the forward prediction frame. Input means for inputting the encoded image encoded by the image processing apparatus according to any one of claims 1 to 3,
Reproduction direction designating means for designating either the forward direction or the reverse direction as the reproduction direction of the encoded image;
Playback frame determining means for determining a playback frame in the encoded image according to the playback direction;
The determined playback frame is decoded using the encoded data by the first inter-frame predictive encoding means when the playback direction is the forward direction, and when the playback direction is the reverse direction. Decoding means for decoding using the encoded data by the second inter-frame predictive encoding means;
Display means for displaying the decoded playback frame;
An image processing apparatus comprising: Input means for inputting the encoded image encoded by the image processing device according to claim 4 or 5,
Reproduction direction designating means for designating either the forward direction or the reverse direction as the reproduction direction of the encoded image;
A reproduction frame determining means for determining a reproduction frame in the encoded image according to the reproduction direction;
The determined playback frame is decoded using the encoded data by the first inter-frame predictive encoding means when the playback direction is the forward direction, and when the playback direction is the reverse direction. Decoding means for decoding using the encoded data by the second inter-frame predictive encoding means;
When the reference frame referred to when the decoding unit decodes the reproduction frame is decoded in a reproduction direction different from the reproduction direction designated by the reproduction direction designation unit, the reproduction frame is decoded. Arithmetic means for adding or subtracting the difference information to the result;
Display means for displaying the playback frame decoded by the decoding means, or the playback frame for which the difference information has been calculated by the calculation means;
An image processing apparatus comprising: The playback direction designating means further enables designation of playback stop,
The playback frame determining means sequentially updates the playback frame at a predetermined timing according to the playback direction specified by the playback direction specifying means, and when playback stop is specified by the playback direction specifying means, 8. The image processing apparatus according to claim 6, wherein updating of the reproduction frame is stopped. In an image processing apparatus having frame classification means, intra-frame coding means, first and second inter-frame prediction coding means, and coded image creation means, while continuously changing the positional relationship between the subject and the imaging device An image processing method for generating an encoded image by encoding a captured image of a plurality of frames,
The frame classification means classifies the plurality of frames into a first frame that performs intra-frame coding and a second frame that performs inter-frame predictive coding,
The intra-frame encoding means performs intra-frame encoding for the first frame;
The first inter-frame predictive encoding means performs inter-frame predictive encoding in a frame sequence when reproducing the encoded image in the forward direction for the second frame;
The second inter-frame predictive encoding means performs inter-frame predictive encoding in a frame sequence when reproducing the encoded image in the direction opposite to the forward direction for the second frame;
The encoded image creating means includes a first frame encoded by the intra-frame encoding means, a second frame encoded by the first inter-frame predictive encoding means, and the second frame An image processing method characterized in that the encoded image is created using the second frame encoded by the inter-frame predictive encoding means as an encoding unit.   A non-transitory computer-readable storage medium storing a program for causing a computer device to function as each unit of the image processing device according to claim 1 when executed by the computer device.

Images