JP4010270B2 - Image coding and transmission device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image encoding transmission apparatus, and in particular, transmits an encoded bit stream obtained by encoding an arbitrary frame as a reference frame when encoding an image signal through a transmission path using a storage medium or a network. The present invention relates to an image coding and transmission apparatus using an arbitrary reference frame.
[0002]
[Prior art]
In recent years, video distribution systems capable of accessing a moving image to be viewed via a computer network have become widespread. This video distribution system is mutually connected to a plurality of networks, and a plurality of servers for distributing video information are connected to each network. A plurality of video reception / playback terminal devices (hereinafter simply referred to as terminal devices) to which video information is distributed are connected to each network.
[0003]
In the above video distribution system, when the terminal device distributes video information from the server, the terminal device first contacts the server and establishes a line connection state between the terminal device and the server. Thereafter, the terminal device receives desired video information provided from the server and reproduces this video information.
[0004]
In such a video distribution system, generally transmitted video information is often multimedia information created by combining digitized audio, images, and other data. The information amount of such video information (multimedia information) is generally 1 to 2 bytes for the information amount of character information when the data is simply digitized and converted. In the case of voice, the telephone quality is 64 kbps. In the case of a moving image, an information amount of 100 Mbps or more is required at the current terrestrial television broadcast quality. It is not practical to handle such a large amount of digital information as it is in view of the cost of using the current transmission path and recording medium.
[0005]
Therefore, an encoding technique for compressing such an enormous amount of information is currently being actively researched and developed. MPEG (Moving Picture Experts Group) is a representative example of video encoding technology. MPEG is an international standard for moving image data encoding technology. Currently, this MPEG standard makes it possible to cope with various applications such as communication and broadcasting such as MPEG-1 and digital broadcasting mainly for storage media of about 1.5 Mbps such as video CD. MPEG-2, MPEG-4 capable of realizing more advanced multimedia contents by object coding and control thereof, and MPEG-7 for content description are defined.
[0006]
By applying such an encoding technique, video data having an enormous amount of information can be efficiently encoded and the amount of information can be compressed to about 1/20 to 1/40. In addition, there are other encoding methods that introduce new encoding methods such as H.264, JPEG2000 (Joint Photographic Experts Group 2000), which are currently in the process of formulating standards. More effective information encoding can be performed.
[0007]
On the other hand, MPEG is configured so that more effective compression can be achieved by combining a plurality of technologies. FIG. 8 is a block diagram showing an example of a conventional MPEG encoding apparatus. This encoding apparatus is an apparatus that encodes image information by MPEG. By taking the difference between the input code 301 and the image signal obtained by decoding the input code 301 by the motion compensation predictor 305, the time of the input code 301 is obtained. Redundant components included in the axial direction are reduced.
[0008]
MPEG has three prediction modes: intra-frame (Intra) coding, forward (Predictively) prediction coding, and bidirectional (Bidirectional) prediction coding. Among these, in the intra-frame encoding, encoding is performed using only the information in the frame without using the output of the motion compensation predictor 305. A frame encoded in this mode is called an I picture or IVOP (Intra Video Object Plane). The IVOP can be decoded without depending on other frames at the time of decoding.
[0009]
In the forward predictive coding, a motion-compensated predictor 305 compensates for a previously coded frame and predicts the current frame. The difference frame is converted by the orthogonal transformer 303. For the orthogonal transform performed by the orthogonal transformer 303, DCT (Discrete Cosine Transform) is used in MPEG. Orthogonal transformation can also be performed using other transformation bases such as Hadamard bases and wavelet bases. A frame encoded in this mode is called a P picture or PVOP. Since PVOP depends on other frames, it cannot be decoded independently like IVOP, but the compression rate can be higher than that of IVOP.
[0010]
In bi-directional predictive coding, the motion-compensated predictor 305 compensates past and future coded frames and predicts the current frame. The difference frame is converted by the orthogonal transformer 303. A frame encoded in this mode is called a B picture or BVOP. Since BVOP depends on other frames in the past and future, it cannot be decoded independently like IVOP and PVOP, but the compression rate can be further increased than IVOP and PVOP.
[0011]
Each VOP is subjected to prediction processing for each 16 pixels × 16 pixels MB (macroblock). The prediction direction differs depending on IVOP, PVOP, and BVOP. IVOP encodes all MBs independently. PVOP has two modes: a mode in which encoding is performed by prediction from a past frame, and a mode in which the MB is independently encoded without prediction. In addition, BVOP has four modes in which the MB is independently encoded without prediction from the future, prediction from the past, prediction from both, and prediction.
[0012]
The motion estimator 304 performs pattern matching on the motion region of the input code for each MB, and detects a motion vector with 0.5 pixel accuracy. The motion compensation predictor 305 uses the motion vector detected by the motion estimator 304 to make a shift by the amount of motion and performs prediction. The motion vector has a horizontal direction and a vertical direction, and is transmitted as additional information of the MB together with an MC (Motion Compensation) mode indicating where the prediction is from.
[0013]
The difference image signal obtained by subtracting the input code 301 for one frame read from the frame memory 302 and the motion vector from the motion compensation predictor 305 by the subtractor 314 is supplied to the orthogonal transformer 303. Here, orthogonal transformation is performed. This orthogonal transform uses DCT in MPEG. DCT is an orthogonal transformation that discretely transforms an integral transformation based on a cosine function into a finite space. In MPEG, two-dimensional DCT is performed on an 8 × 8 DCT block obtained by dividing MB into four. In general, since a video signal has many low frequency components and few high frequency components, the coefficients can be concentrated on the low frequency components by performing DCT, and the subsequent quantizer 306 can efficiently reduce the amount of information. .
[0014]
The DCT coefficient obtained by DCT in the orthogonal transformer 303 is quantized by the quantizer 306. This quantization is based on a value obtained by multiplying a value obtained by weighting an 8 × 8 two-dimensional frequency called a quantization matrix with a visual characteristic and a value called a quantization scale for multiplying the whole by a scalar, and using a quantized value as a DCT coefficient. Divide by the digitized value. When inverse quantization is performed by the decoder, a value close to the original DCT coefficient is obtained by multiplying by the quantized value.
[0015]
The quantized data is encoded by the entropy encoder 310. In general, the entropy encoder 310 is a variable length code (VLC) device, and performs variable length coding (VLC) on quantized data. Of the quantized values, the direct current (DC) component uses DPCM (Differential Pulse Code Modulation) which is one of predictive coding. Moreover, the alternating current (AC) component among the quantized values is zigzag scanned from the low range to the high range, and the run length of zero and the effective coefficient value are set as one event, and the code length is changed from the one having a high appearance probability. Huffman coding that assigns short codes is performed.
[0016]
The entropy encoder 310 encodes not only the quantized data but also auxiliary information related to the motion vector output from the motion estimator 304 and the motion compensated predictor 305 according to a predetermined condition, and entropy-encodes each. Data is stored in the temporary buffer memory 312 through the multiplexer 311 and output as encoded data (output code) 317 at a predetermined transfer rate. Further, the code amount for each macroblock of the output code 317 is notified to the code amount controller 313. The code amount controller 313 notifies the quantizer 306 of the error code amount between the notified code amount and the target code amount. The quantizer 306 can control the code amount by adjusting the quantization scale based on the notified error code amount information.
[0017]
The quantized data output from the quantizer 306 is inversely quantized by the inverse quantizer 307 and further inversely orthogonally transformed by the inverse orthogonal transformer 308 using a transform base corresponding to the orthogonal transformer 303. The inverse quantizer 307 and the inverse orthogonal transformer 308 constitute a local decoding unit 315. The signal output from the inverse orthogonal transformer 308 is added to the frame from the motion compensated predictor 305 in the adder 316 to form a decoded image, and then temporarily stored in the reference prediction memory 309. The frame temporarily stored in the reference prediction memory 309 is used as a reference frame (reference decoded image) for calculating a difference image in the motion compensation predictor 305.
[0018]
By the way, a group from a certain I picture to a picture before the next I picture is called a GOP (Group Of Picture). A range from one IVOP to the VOP before the next IVOP is referred to as GOV (Group Of Vop). When used in storage media or the like, generally about 15 pictures are used as one GOP and GOV. Here, a structure for configuring the GOV is shown in FIG.
[0019]
As shown in the figure, the GOV start code is 32 bits, the time code indicating the time from the beginning of the sequence is 18 bits, and the closed gov indicating whether the images in the GOV are independent playback from other GOVs One bit, a broken link indicating whether the preceding GOV data is not available for editing is represented by one bit.
[0020]
The encoded bit stream output from the image encoding / transmission apparatus having the configuration shown in FIG. 8 is distributed by transmission through a storage medium or a network and reproduced by the terminal apparatus. In the case of reproduction by the terminal device, the distributed coded bit stream is reproduced after being decoded by the MPEG decoder of the terminal device.
[0021]
FIG. 10 shows a block diagram of an example of a conventional MPEG decoder. In the figure, when an encoded bit stream distributed to a terminal device is input as an input code 401 to an MPEG decoder, it is supplied to a demultiplexer 402. The demultiplexer 402 separates the input code 401 into information such as entropy-encoded motion vector information and texture information, and supplies each information to the entropy decoder 403.
[0022]
The entropy decoder 403 decodes the entropy-encoded texture information, supplies it to the inverse quantizer 404, decodes the entropy-encoded motion vector information, and supplies it to the motion compensation predictor 407. In general, the entropy decoder 403 is an IVLC (Inverse Variable Length Code) device, and performs variable length decoding on the demultiplexed data. The inverse quantizer 404 multiplies the entropy-decoded texture information by a quantized value to calculate a value close to the original orthogonal transform coefficient, and supplies the calculation result to the inverse orthogonal transformer 405. To do.
[0023]
The motion compensated predictor 407 corresponds to the macroblock to be encoded from the reference frame stored in the reference prediction memory 406 with reference to the position of the macroblock to be encoded in units of macroblocks. An area is specified by using motion vector information in reverse. Then, the motion compensated predictor 407 shifts the region on the identified reference frame by the information of the motion vector obtained from the entropy decoder 403, and arranges it at the position of the macroblock to be encoded at present. Create a prediction frame. The inverse orthogonal transformer 405 obtains original frame information or inter-frame difference information by performing inverse orthogonal transformation on the orthogonal transformation coefficient input from the inverse quantizer 404. Inverse orthogonal transform uses IDCT in MPEG.
[0024]
The frame obtained by the IDCT is added to the prediction frame from the motion compensation predictor 407 in the adder 408 and then output as an output code 409 and stored in the reference prediction memory 406. In this way, the input code 401 inputted is decoded by the MPEG decoder of the terminal device and reproduced as video information.
[0025]
[Problems to be solved by the invention]
Conventionally, when video information is to be transmitted through a transmission line, network transmission can be performed using a general MPEG configuration in order to avoid problems such as synchronization on the image receiving / decoding apparatus side. On the image encoding / transmission apparatus side configured as described above, video information is encoded in units of GOP or GOV. In general, an encoded bit stream obtained by encoding video information in units of GOV by an image encoding / transmission apparatus is an image receiving / decoding configured so that it can be transmitted over a network by using a general MPEG system configuration. The data are sequentially transmitted from the head of the GOV to the apparatus via the transmission path.
[0026]
FIG. 11 is a block diagram of an example of a conventional image encoding / transmission apparatus, and FIG. 12 is a block diagram of an example of a conventional image receiving / decoding apparatus. As shown in FIG. 11, a conventional image encoding / transmission apparatus includes an encoding unit 501 and an encoding / transmission unit 502. In general, the encoding / transmission unit 502 efficiently uses a transmission path during transmission. In order to transmit data, the GOV is divided into smaller packets and transmitted.
[0027]
The encoding unit 501 includes a buffer buffer 504 that temporarily stores an image signal from the input unit 503, a multi-frame storage memory 506 that stores at least one frame of the input image signal, a GOV management unit 505, and a multi-frame storage unit. An encoding unit 510 that encodes an image signal output from the memory 506 by the MPEG method, a buffer buffer 509 that stores encoded frames, a decoding unit 511 that decodes encoded frames, and a reference frame that is generated and encoded The frame reference prediction memory 512 supplied to the unit 510.
[0028]
In order to transmit the GOV as a coding unit, the GOV management unit 505 controls the switches SW21 and SW22 as necessary while monitoring the storage state of the multi-frame storage memory 506 and the GOV storage buffer memory 513. , The encoded output in units of GOV is output to the encoded transmission unit 502.
[0029]
The GOV storage buffer memory 513 of the encoding transmission unit 502 temporarily stores the encoded frame input from the encoding unit 510 in the encoding unit 501 via the buffer buffer 509 and the switch SW22, and then outputs it to the transmission packet sending means 514. To do. The transmission packet sending means 514 divides the packet into at least one packet that is a transmission unit for network transmission, and sends the packet to the network or storage means 515 while monitoring the state of the transmission band of the network.
[0030]
Also, as shown in FIG. 12, the conventional image reception encoding apparatus receives and decodes at least one packet sent to the network or storage means 515 (603) by the encoding transmission unit 502 of FIG. A decoding receiving unit 601 that performs decoding, a decoding unit 602 that decodes the decoded data, and an image signal from the decoding unit 602 that is stored for one frame before being supplied to the display device 615 to display an image. And frame memory 614.
[0031]
The decoding receiving unit 601 receives the input packet by the transmission packet receiving unit 604, analyzes the bit stream by the encoded bit stream analyzing unit 605, and obtains the obtained encoded frame information as a frame storage order control unit. The data is input to the GOV storage buffer memory 608 via the switch SW31 controlled by 606, and the GOV as a decoding unit is reconfigured.
[0032]
The GOV management means 609 in the decoding unit 602 controls the switch SW32 as necessary while monitoring the storage state of the GOV storage buffer memory 608, thereby transmitting the encoded data in units of GOV via the buffer buffer 611. The data is sent to the decoding unit 612. The decoding unit 612 performs decoding in units of GOV using the reference frame from the frame reference prediction memory 613, obtains a reproduced video signal, and supplies it to the display frame memory 614. The reproduced video signal for one frame stored in the display frame memory 614 is displayed as an image by the display device 615.
[0033]
Here, when the encoded bit stream is transmitted via the transmission path from the image encoding / transmission apparatus of FIG. 11 to the image reception / decoding apparatus of FIG. The information in the GOV may not be completely reconstructed due to the omission and may affect the decoding of the GOV.
[0034]
In addition, because the bit rate of the GOV currently being transmitted cannot be satisfied due to a drastic decrease in the transmission rate, it takes time to reconfigure the GOV due to packet delay or retransmission, and as a result, Transmission may be delayed.
[0035]
As described above, when the GOV transmission is not sufficient, the image receiving / decoding apparatus discards an incomplete GOV that is not in time for the reproduction time, thereby causing a problem that reproduction for one GOV is stopped.
[0036]
Also, in the image receiving / decoding apparatus, if the GOV playback time has arrived with respect to an incomplete GOV, decoding is performed to the point where decoding can be performed without discarding the incomplete GOV, and playback display is performed as much as possible. You can continue. However, even in this case, since the VOP existing in the latter half of the GOV that could not be transmitted by the reproduction time cannot be decoded, the reproduction stops until the next GOV.
[0037]
Conventionally, in order to avoid such a missing or delayed GOV, when the transmission rate of the transmission path between the image encoding transmission device and the image reception decoding device is monitored at any time and a sudden decrease in the transmission rate is detected. Requests the image coding and transmission apparatus to lower the coding bit rate. In the image encoding transmission apparatus, the encoding bit rate is switched by lowering the spatial resolution, the frame rate, and the image quality according to the required encoding bit rate, and an encoded bit stream having a lower bit rate is generated. It can be considered that the transmission is performed by the encoded transmission unit 502.
[0038]
However, in order to instruct the change of the encoding bit rate while monitoring the transmission bit rate of such a transmission line, a large GOV reception buffer is prepared in the decoding receiving unit of the image receiving decoding apparatus, and transmission is performed. It is necessary to detect the transmission bit rate of the path. Since the transmission bit rate of the transmission line is detected by monitoring the accumulation state of the GOV reception buffer, a certain time difference is generated until the fluctuation of the transmission bit rate is detected. Due to this time difference, an attempt is made to continue to transmit the encoded bit stream with the high encoding bit rate to the transmission line with the low transmission bit rate, so that the transmission delay increases.
[0039]
Further, in the encoding transmission unit 502 in the image encoding transmission apparatus shown in FIG. 11, the encoded bit stream output from the encoding unit 510 in the encoding unit 501 via the buffer buffer 509 and the switch SW22 is changed to GOV. Consider a case in which the GOV is packetized by the transmission packet sending means 514 after being stored in the storage buffer memory 513 and transmitted to the transmission path as the network or the storage means 515.
[0040]
After the encoded bit stream output from the encoding unit 501 is stored in the GOV storage buffer memory 513, the transmission packet transmission means 514 converts the GOV into a packet and sends it to the transmission line. When the image encoding / transmission apparatus detects that the transmission bit rate has decreased, it is assumed that the encoding bit rate of the GOV currently being sent is being changed from the middle.
[0041]
In a normal image encoding / transmission apparatus, if the GOV is stored in the GOV storage buffer memory 513 of the encoding / transmission unit 502, the encoding bit rate cannot be corrected for the stored GOV. A time difference will occur in the control of the coding bit rate.
[0042]
In the case of a more advanced image encoding transmission apparatus, a GOV that is not yet sent to the transmission path is analyzed and a VOP that has not been sent to the transmission path is specified. Thereafter, the GOV currently being transmitted is deleted from the GOV storage buffer memory 513, and a re-encoding request is made from the specified VOP to the encoding unit 501 again. If VOPs corresponding to the number of remaining frames can be acquired from the specified VOP, the transmission delay of the GOV is small, and the reduction in reproduction goods on the image reception decoding apparatus side can be suppressed.
[0043]
However, in the configuration of such an image encoding / transmission apparatus and image receiving / decoding apparatus, when live video or the like is distributed in real time, re-encoding processing can be performed on the image encoding / transmission apparatus side. Therefore, a large amount of frame storage buffer memory is required. There is also a problem that complicated control of the encoding unit 501 is required.
[0044]
The present invention has been made in view of the above points. When an encoded bit stream is transmitted from an image encoding / transmission apparatus to an image reception / decoding apparatus via a transmission path, the transmission rate varies greatly. Even if it is difficult to completely reconstruct the information in the GOV due to packet loss in the conventional method, by considering the encoding method and transmission order of the encoded frame in the GOV, the image reception decoding device side Using the GOV information transmitted up to the decoding time, the frame rate on the decoding side automatically changes according to the number of frames in the GOV accumulated up to the decoding time, so that frames can be reproduced and displayed as equally as possible. Thus, an object of the present invention is to provide an image encoding / transmission apparatus that can reduce the influence on GOV decoding and can realize higher-quality image transmission than before. That.
[0045]
[Means for Solving the Problems]
In order to achieve the above object, the present invention encodes an input image signal using a reference frame, and transmits the obtained encoded frame in accordance with a predetermined frame transmission order determined in advance. A first frame storage memory for storing at least one frame of an input image signal, a second frame storage memory for storing a predetermined number of reference frames used for encoding, and a first frame storage The first switch means provided on the output side of the memory, the second switch means provided on the input side and the output side of the second frame storage memory, and the first frame means according to a predetermined frame sending order. And reference frame control for controlling the transfer of the input image signal and the reference frame stored in the first and second frame storage memories by controlling the second switch means. And an input image signal having a frame number selected by the first switch means from the input image signals stored in the first frame storage memory is stored in the second frame storage memory. Using one reference frame selected by the second switch means from among the reference frames, the encoding unit obtains an encoded frame by encoding, the encoded frame output from the encoding unit is decoded, and the second A reference frame is newly generated from the reference frames stored in the frame storage memory using one reference frame used at the time of encoding by the encoding unit, and the new reference frame is converted into the second reference frame. The local decoding unit newly stored in the second frame storage memory via the switch means and the accumulation status of the first frame storage memory are monitored. I / O frame management means for managing the storage state and monitoring the encoded frames output from the encoding unit, managing the encoded output, and notifying the reference frame control means of these states, A third frame storage memory for storing a group of encoded frames generated by the encoding unit, and each frame included in the group of encoded frames stored in the third frame storage memory are packetized in advance. And an output means for transmitting in accordance with a predetermined frame transmission order.
[0046]
In the present invention, the input image signal having the frame number selected by the first switch means from the input image signals stored in the first frame storage memory is stored in the second frame storage memory. The local decoding unit decodes the encoded frame output from the encoding unit that obtains the encoded frame by performing encoding using one reference frame selected by the second switch means from among the reference frames Then, a reference frame is newly generated from one of the reference frames stored in the second frame storage memory using the one reference frame used for encoding by the encoding unit, and the new reference frame is Since the second frame is newly stored in the second frame storage memory via the second switch means, the encoding unit encodes the second The frames stored in memory for frame storage can be used as arbitrary reference frame.
[0047]
Here, the output means includes an encoded bit stream analyzing means for specifying each frame included in a group of encoded frames stored in the third frame storage memory, and a third frame storage memory. Frame transmission control means for controlling the order in which frames included in the group of encoded frames stored in the frame are transmitted to the transmission line according to a predetermined frame transmission order.
[0048]
The encoded bit stream analyzing means specifies each encoded frame included in the GOV stored in the third frame storage memory. The frame transmission control means uses the position information of each encoded frame specified by the encoded bitstream analyzing means to change the order of the encoded frames in the GOV stored in the third frame storage memory. The GOV can be changed in accordance with a predetermined frame transmission order when it is transmitted to the transmission line.
[0049]
In order to achieve the above object, according to the present invention, the first and second switches are configured such that the reference frame control means uses the sending frame number determined by the computing means as the predetermined frame sending order in the order determined. And the arithmetic means sets the number of encoded frames included in the encoded frame group as M, and the frame number of the input image signal stored in the first frame storage memory ranges from 0 to M−. If it is a natural number up to 1, the frame number 0 is the first transmission frame number, the initial value of the variable A is initialized to M, the initial value of the count value C is initialized to 1, and the initialization unit initializes the initial value. After conversion,
B = [(A + 1) / 2] ([] is a Gaussian symbol)
The value of B is calculated by using the first calculation means for setting the variable n to 0, the value of B obtained by the first calculation means and the value of the variable n, and
D = B + 2 × B × n (n is a positive integer)
D <M by the second calculating means for calculating D by the first calculating means, the first determining means for determining whether or not the value of D calculated by the second calculating means is less than M, and the first determining means. Is determined as the transmission frame number, and the variable n and the count value C are incremented by 1, respectively, and the transmission frame number is determined by the second calculation means to calculate D again. When the D and the first determination means determine that D is M or more, the second determination means and the second determination means determine whether or not the count value C is greater than M. When the count value C is determined to be less than or equal to M, the first calculation means sets the A value to the current B value, and then the first calculation means calculates B. By the determination means of 2, the count value C is M And performing the above-described operation until it is determined to be large.
[0050]
In the present invention, the predetermined frame transmission order is set to the determined order of the transmission frame numbers determined by the calculation means, and this is used by the reference frame control means and the output means, so that the image reception decoding apparatus side Even when decoding is performed in the order of the encoded frames received in step 1, the encoded transmission can be performed so that the frame rate is improved every time the encoded frame is decoded within the GOV unit.
[0051]
In the present invention, at least one of the frame positions which are already transmitted before the frame to be encoded based on a predetermined transmission frame order in the reference frame control means is provided. Managing input / output of frames from the first frame storage memory and the second frame storage memory to the encoding unit and the local decoding unit so that encoding is performed using one reference frame. . With this management, even when decoding is performed in the order of the encoded frames received on the image receiving / decoding apparatus side, the encoded output is controlled so that the frame rate is improved every time the encoded frame is decoded within the GOV unit. can do.
[0052]
Further, according to the present invention, in the above reference frame, in response to a reference frame switching request from the encoding unit, the frame is already transmitted before the frame to be encoded based on the transmission frame order obtained by the arithmetic means. Among the at least one reference frame of at least one frame position that is to be encoded, when encoding the frame that is currently to be encoded, it is not yet used as a reference frame or a combination of reference frames. Input frames from the first frame storage memory and the second frame storage memory into the encoding unit and the local decoding unit so that encoding is performed again by switching to another reference frame or a combination of a plurality of reference frames. It is characterized by managing output.
[0053]
The present invention further includes means for monitoring an encoding bit rate at the time of encoding in the encoding unit, and for performing re-encoding to the reference frame control means in accordance with a predetermined encoding output condition, A means for making a switching request is provided. Thereby, a predetermined encoding output condition is satisfied.
[0054]
Further, according to the present invention, the predetermined encoding output condition is already transmitted prior to the frame to be encoded based on the transmission frame order obtained by the calculation means of claim 2. The encoded frame is output when the minimum encoded bit rate is reached from at least one reference frame at at least one frame position. According to the present invention, an optimum encoded output can be obtained when encoding is performed on each frame in the GOV based on the order of transmission frames.
[0055]
Further, according to the present invention, the content of the third frame storage memory capable of storing the group of at least one encoded frame generated by the encoding unit is the encoding unit of the next at least one encoded frame. And is discarded when the time for transmission is reached, and a group of at least one encoded frame generated by the encoding unit is quickly stored. In the present invention, the encoding bit rate is controlled without constantly monitoring the transmission bit rate of the transmission line, and the transmission delay is increased due to a certain time difference required to detect the fluctuation of the transmission bit rate, and is given to the GOV decoding. The impact can be reduced.
[0056]
DETAILED DESCRIPTION OF THE INVENTION
Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows a block diagram of an embodiment of an image encoding and transmitting apparatus using an arbitrary reference frame according to the present invention. The image coding / transmission apparatus according to this embodiment mainly includes an encoding unit 101 that can encode an input image using at least one reference frame, and a group of at least one coding frame. An encoding transmission unit 102 that can efficiently transmit a GOV to a transmission path, an input unit 103 that enables the encoding unit 101 to acquire an image to be encoded and transmitted, and an encoded frame as an image It comprises a network or storage means 122 that can transmit to the receiving and decoding device.
[0057]
Further, in the present embodiment, the encoding unit 101 includes a buffer buffer 104, switches SW1 to SW5, a first memory for storing a plurality of frames 106, an input / output frame management unit 107, a buffer buffer 109, a reference frame control unit 110, and an encoding unit. 111, a second memory for storing a plurality of frames 113, and a local decoding unit 116. Here, the encoding unit 101 has a configuration in which the frame reference prediction memory 512 of the encoding unit 501 shown in FIG. 11 is expanded to the second multi-frame storage memory 113. Accordingly, the GOV of FIG. In FIG. 1, the management unit 505 is extended to the reference frame control unit 110 and the input / output frame management unit 107.
[0058]
1 corresponds to the local decoding unit 315 in FIG. 8, the second multi-frame storage memory 113 in FIG. 1 corresponds to the reference prediction memory 309 in FIG. 8, and the encoding unit 111 , Corresponding to the remaining part of FIG.
[0059]
The description regarding each component of the encoding part 101 is given below. The buffer 104 has a buffer area sufficient to temporarily store a plurality of image frames input from the input means 103, and stores frames for GOV in the first plurality of frames storage memory 106. This is a buffer for making adjustments.
[0060]
The switch SW1 is for managing the connection between the buffer buffer 104 and the first multiple-frame storage memory 106. The switch SW1 is connected, disconnected, and switched by the input / output frame management means 107. The first multi-frame storage memory 106 needs to be able to store at least the number of input frames necessary for configuring the GOV so that the encoding unit 111 can perform encoding in units of GOV.
[0061]
For example, when the frame rate of the input image is 30 [fps], 1 GOV is 0.5 seconds, and the number of encoded frames stored in 1 GOV is 15, the first multi-frame storage memory 106 is at least 15 A memory area capable of storing input images for frames is secured.
[0062]
The input / output frame management unit 107 stores the input frame from the buffer buffer 104 to the first plural-frame storage memory 106 by connecting / disconnecting / switching the switch SW1 in order to control encoding in units of GOV. to manage. Further, the input / output frame management means 107 manages the output signal from the buffer buffer 109 by connecting, disconnecting, and switching the switch SW5. Further, the input / output frame management unit 107 notifies the reference frame control unit 110 of a request to start encoding in GOV units, thereby managing the timing for starting encoding in GOV units.
[0063]
That is, the input / output frame management means 107 monitors whether or not an input frame necessary for encoding the GOV is stored in the first plural-frame storage memory 106. By confirming that sufficient input frames are stored in the first multi-frame storage memory 106 in units of GOV, encoding is performed in an arbitrary frame order included in the GOV in units of GOV. Will be able to.
[0064]
Further, by monitoring the encoded frame output from the encoding unit 111, it is possible to control the storage state of the third multi-frame storage memory 117. Further, by notifying the reference frame control means 110 of the storage status of the first multiple frame storage memory 106 and the third multiple frame storage memory 117, the reference frame control means 110 is able to It becomes possible to control at least one reference frame.
[0065]
The switch SW5 is for managing the output timing of the GOV temporarily stored in the buffer buffer 109. The timing of connection / disconnection / switching of the switch SW5 is managed by the input / output frame management means 107. The buffer buffer 109 is a buffer buffer for temporarily storing the GOV encoded frame output from the encoding unit 111 in order to output it to the encoding transmission unit in GOV units.
[0066]
The switch SW2 is a switch for managing the connection between the first multiple-frame storage memory 106 and the encoding unit 111, and is controlled to be connected, disconnected, and switched by the reference frame control unit 110. The second multi-frame storage memory 113 stores at least the number of frames necessary for configuring the GOV so that the reference frame generated by the local decoding unit 116 can be stored at the corresponding frame position in the GOV. There is a need to have a frame memory that can. For example, when the frame rate of the input image is 30 [fps], 1 GOV is 0.5 seconds, and the number of encoded frames stored in 1 GOV is 15, the second multi-frame storage memory 113 is at least 15 A memory capacity capable of storing reference frames for frames is secured.
[0067]
The switch SW3 is a switch for managing the connection between the local decoding unit 116 and the second multi-frame storage memory 113, and is controlled to be connected, disconnected, and switched by the reference frame control means 110. The switch SW4 is a switch for managing the connection between the second multi-frame storage memory 113 and the encoding unit 111, and the second multi-frame storage memory 113 and the local decoding unit 116, and is connected by the reference frame control unit 110. And cutting and switching control.
[0068]
The reference frame control unit 110 controls connection, disconnection, and switching of the switch SW2 to identify an input frame to be input to the encoding unit 111 based on a predetermined frame transmission order, and to input a correct input frame to the encoding unit 111. I do. Further, the reference frame control means 110 manages the connection, disconnection, and switching of the switch SW3, thereby generating the reference frame generated by the local decoding unit 116 in the corresponding reference frame position in the second multiple-frame storage memory 113. Control so that it can be stored correctly.
[0069]
Further, the reference frame control means 110 manages the connection, disconnection, and switching of the switch SW4, so that the predetermined frame sending order is applied to the position of the input frame that is currently being encoded. In the frame transmission order, control is performed so that reference frames that have already been transmitted can be used for encoding prediction.
[0070]
The local decoding unit 116 has a configuration like the local decoding unit 115 included in an MPEG encoding apparatus as shown in FIG. 8, for example. That is, the encoded frame output from the encoding unit 111 to the local decoding unit 116 corresponds to the encoded frame after quantization by the quantizer on the input side of the entropy encoder in FIG. The unit 116 includes at least an inverse quantizer and an inverse orthogonal transformer.
[0071]
In addition, when the local decoding unit 116 obtains the encoded frame output from the encoding unit 111, the local decoding unit 116 performs inverse quantization by an inverse quantizer, and then performs inverse orthogonal transformation to perform current encoding. A decoded frame of the encoded frame is obtained. The local decoding unit 116 outputs this decoded frame as a reference frame to the second multi-frame storage memory 113 via the switch SW3, and prepares for the encoding of the next input frame.
[0072]
For example, the encoding unit 111 encodes an IVOP for an input frame that can be acquired from the first multiple-frame storage memory 106 when encoding an IVOP, such as an MPEG encoding device as shown in FIG. A predetermined orthogonal transformation is performed. Here, the predetermined orthogonal transform employs DCT in MPEG. However, if the encoding unit 111, the local decoding unit 116, and the decoding unit 216 of the image reception decoding device in the image encoding / transmission device of the present embodiment all use the same orthogonal transformation, a predetermined orthogonal transformation is performed by Hadamard. It should be noted that other orthogonal transformations such as transformation and wavelet transformation may be used and are not particularly limited.
[0073]
Thereafter, the amount of information is reduced by the quantizer in the encoding unit 111 having the configuration shown in FIG. In the case of PVOP or BVOP, an input frame that can be acquired from the first multi-frame storage memory 106 and a reference frame that can be acquired from the second multi-frame storage memory 113 are used as reference frames. On the other hand, a predetermined orthogonal transformation is performed on the difference frame between the frame subjected to motion compensation and the input frame. Thereafter, the information amount is reduced by the quantizer in the encoding unit 111.
[0074]
The encoding unit 111 outputs the encoded data at this point to the local decoding unit 116. Furthermore, entropy encoding is performed on the encoded data after quantization. Thereafter, additional information such as entropy-encoded vector information is multiplexed by a multiplexer in the encoding unit 111. The encoding unit 111 outputs the encoded frame encoded in this way to the buffer buffer 109.
[0075]
Further, when it is necessary to minimize the code amount at the time of encoding, the code amount controller monitors the buffer memory in the encoding unit 111 and performs at least one encoding using at least one reference frame. It is necessary to monitor the code amount of the encoding result that is output each time encoding is performed, and to use the encoding result when the code amount is the smallest as the encoded frame output. Therefore, the code amount controller in the encoding unit 111 can notify the reference frame control unit 110 of a reference frame change request.
[0076]
When the reference frame control means 110 obtains this reference frame change request, another reference frame is selected from the reference frames already encoded and output before the input frame currently encoded based on the predetermined frame transmission order. Switch to. Thereby, the encoding unit 111 can acquire a new reference frame based on a predetermined frame transmission order.
[0077]
By repeating this operation, it is possible to compare the code amounts when encoding with a plurality of reference frames, and the encoding unit 111 identifies the reference frame having the minimum code amount, and encodes with the minimum code amount. The frame can be output to the buffer buffer 109.
[0078]
For example, it is assumed that 15 frames are encoded as 1 GOV, encoding is performed based on a predetermined frame transmission order as shown in FIG. 2, and encoding is currently performed up to the third transmission order. Assume that you are moving forward. That is, in the frame number, the frames up to the 0th, 8th, 4th, and 12th frames are already encoded and stored as the reference frames in the second multi-frame storage memory 113 of FIG.
[0079]
In the case of normal encoding, the reference frame stores only one frame immediately after encoding, and the reference frame that was previously encoded cannot be used. However, in the present embodiment, by controlling the switch SW4, the reference frame generated by the encoding up to the present time, that is, the frame number 0 stored in the second multi-frame storage memory 113 here. , 8, 4, 12 can be used as reference frames.
[0080]
Therefore, the reference frame to be used when encoding the next encoding target frame number 2 (4 in the transmission order) can be selected from four frame numbers 0, 8, 4, and 12. it can. Therefore, encoding is performed in each of these four frames, the output code amount at that time is obtained, the output code amounts are compared, and the reference frame that outputs the minimum code amount can be specified.
[0081]
Next, the encoded transmission unit 102 will be described. In the present embodiment, the encoded transmission unit 102 includes a third memory for storing a plurality of frames 117, an encoded bitstream analysis unit 118, a frame transmission order control unit 119, a switch SW6, and a transmission packet transmission unit 121. The description regarding each component of the encoding transmission part 102 is given below.
[0082]
The third multi-frame storage memory 117 is a memory that can acquire and store the GOV via the switch SW5 from the buffer buffer 109 that temporarily stores the encoded frames generated by the encoding unit 111 for the GOV. It has. The memory amount of the third plural-frame storage memory 117 is secured by calculating in advance from the maximum bit rate of the encoding unit 101 and the maximum bit rate of the network or storage means 122 serving as a transmission path.
[0083]
The encoded bitstream analysis means 118 analyzes the GOV GOV header stored in the third multi-frame storage memory 117 and the header information of each encoded frame, and transmits each encoded frame in a predetermined frame transmission order. Information for identifying the position of the encoded frame in the GOV that is necessary for transmission through the path is collected.
[0084]
Also, the encoded bitstream analyzing unit 118 transmits the GOV stored in the third plural-frame storage memory 117 through the transmission path by making a request for starting the encoded transmission to the frame transmission order control unit 119. Manage timing. Also, the information for identifying the position of the encoded frame within the collected GOV is notified to the frame transmission order control means 119. Furthermore, a means is provided for acquiring from the frame transmission order control means 119 that transmission of the encoded frame in the GOV has been completed.
[0085]
In addition, the encoded bitstream analyzing unit 118 acquires an encoded frame from the third multi-frame storage memory 117 via the switch SW6 and outputs the encoded frame to the transmission packet transmitting unit 121. At that time, if correction is necessary with information such as time code (time_code) in GOV header information, modulo time base (modulo_time_base), and vop time increment (vop_time_increment) in the header information of the encoded frame The encoded bitstream analyzing means 118 preferably includes means capable of correcting these header information.
[0086]
The frame transmission order control means 119 connects and disconnects the switch SW6 based on information for specifying the position of the encoded frame in the GOV notified by the encoded bitstream analysis means 118 and the predetermined frame transmission order. By switching, the transmission order of the encoded frames in the GOV stored in the third multi-frame storage memory 117 is managed. Here, the encoded frames generated by the encoding unit 111 are input to the third multiple-frame storage memory 117 in the order of generation, and the encoded frames are output from the third multiple-frame storage memory 117 in the input order. Is controlled to switch the switch SW6.
[0087]
In addition, when the frame transmission order control unit 119 detects that all the encoded frames in the GOV have been transmitted when the switch SW6 is switched, the frame transmission order control unit 119 Notify that transmission of the encoded frame in the GOV has been completed.
[0088]
The switch SW6 is a switch for managing the connection between the third plural-frame storage memory 117 and the encoded bitstream analyzing unit 118, and is controlled to be connected, disconnected, and switched by the frame transmission order control unit 119. The transmission packet sending unit 121 acquires the encoded frame from the encoded bitstream analysis unit 118, disassembles the acquired encoded frame into at least one packet, and transmits it to the network or storage unit 122 which is a transmission path. To do.
[0089]
By providing the transmission packet sending means 121 for packetizing and sending the coded frame to be sent out to the transmission line, the coded frame to be currently transmitted in the GOV specified by the frame sending order control means 119 is displayed. The packet can be transmitted in a packet size that can be efficiently transmitted on the transmission path. By providing the above-described configuration and the function of each component, an embodiment of the image coding and transmitting apparatus of the present invention can be realized.
[0090]
Next, in the operation of the image encoding / transmission apparatus and the image receiving / decoding apparatus according to the present invention, since the predetermined frame transmission order is an important factor, the predetermined frame transmission order and the reference frame at the time of encoding The selection method will be described below.
[0091]
First, a method for determining a predetermined frame transmission order will be described below with reference to FIGS. FIG. 2 is a diagram illustrating the order in which the encoded frames included in the GOV are transmitted and which reference frame is used to encode each encoded frame included in the GOV. FIG. 3 is a flowchart showing a method for obtaining a predetermined frame transmission order in the present invention. Note that the processing of FIG. 3 is performed externally in advance, and information regarding the transmission frame order as a result thereof is stored in the reference frame control unit 110 in advance so that the information can be used. This is performed at the stage of initialization processing performed when the entire coding transmission apparatus is activated.
[0092]
First, a set of encoded frames, that is, the number of encoded frames included in the GOV in this embodiment is M. The stored frame number can be expressed as an integer from 0 to M-1. Here, frame number 0 is the first transmission frame number. In the following description, it is assumed that the GOV is composed of PVOP except that M = 15 and the first frame in the GOV is IVOP.
[0093]
Initially, A = M and C = 1 are initialized (step S101), where A is a variable used during the calculation. C is a counter variable for counting how many frame transmission orders are obtained. It is assumed that the frame number to be transmitted first is the 0th frame, and the frame transmission order already obtained is 1, so such initialization is performed. Since M = 15 now, initialization is performed with A = 15 and C = 1.
[0094]
Next, the following formula
B = [(A + 1) / 2]
To calculate the variable B (step S102). However, in the above formula, [] indicates a Gaussian symbol. Subsequently, after initializing the variable n to 0 (step S103),
D = B + 2 × B × n
To calculate the variable D (step S104).
[0095]
D obtained in this step S104 is considered to be a sequence generated by a geometric sequence having an initial term of B and a common ratio of 2 × B, where n is a positive integer. Therefore, the frame number corresponding to the next frame transmission order is obtained from the position of frame 0 at the interval B.
[0096]
Next, it is determined whether D <M (step S105). If it is determined in step S105 that D <M, it means that the frame position in the frame transmission order obtained in step S104 is within the frame position existing in the GOV. If D <M, the process proceeds to step S106, where D obtained in step S104 is adopted as the next frame transmission order, and this value is stored as the next frame transmission number.
[0097]
Then, it progresses to step S107 and C = C + 1 and n = n + 1 are calculated | required. In step S107, the counter C is incremented by one because the frame transmission order is newly obtained. In addition, n is increased by one to obtain the next number sequence. When the process of step S107 is completed, the process proceeds to step S104, and the value of the next number sequence is obtained. On the other hand, if it is determined in step S105 that D <M, it means that the frame position in the frame transmission order obtained in step S104 is not within the frame position existing in the GOV.
[0098]
Here, it is confirmed that the process has been actually performed in the process so far. First, the first frame transmission order starts from frame 0. Since the next frame transmission order is the first term B, that is, B = 8, the frame 8 is adopted as the next frame transmission order. Thereafter, since it increases at a common ratio of 2 × B, when D is calculated again, D = 8 + 2 × 8 × 1 = 24. Since this value exceeds M = 15 and cannot exist in the GOV, it is determined whether C> M (step S108).
[0099]
If C> M, this means that all frames included in the GOV have been adopted as the frame transmission order, and thus the process for obtaining the frame transmission order is terminated. On the other hand, if it is determined in step S108 that C> M is not satisfied, the process proceeds to step S109, where A = B, that is, the value of A is set to the value of B. Then, it returns to step S102 and continues processing.
[0100]
By performing the processing as described above, the frame transmission order of the present invention can be obtained. For reference, the actual processing from step S102 will be described a little more continuously. Currently, in step S108, the obtained frame transmission order is the order of frame 0 and frame 8, and the process is continued from the place where A = 15, B = 8, D = 24, C = 2, and n = 1. To do.
[0101]
Since C> M is not satisfied at this time, the process proceeds to step S109. In step S109, since the value of A is updated to the value “8” of B, the value is updated to A = 8. In step S102, B = [(A + 1) / 2] is obtained again. Since A is updated to 8, B = [4.5] = 4 is obtained. After that, n = 0 is reset in step S103, and the frame transmission order is obtained using new B = 4 in step S104. Then, the value of D changes like 4 (when n = 0), 12 (when n = 1), and 20 (when n = 2) (repetition of steps S104 to S107). Accordingly, since D <M is not satisfied at the time point of n = 2, the above process is repeated after updating the value of A to the value of B (4 at this time) again in step S109. Therefore, until now, the frame transmission order has been obtained from frame 0 → frame 8 → frame 4 → frame 12. As for the subsequent frame transmission order, an order as shown in “transmission order” in FIG.
[0102]
2C and 2D, “transmission order” and “reference frame number” are described on four horizontal lines, but this is shown for convenience of illustration. In terms of time, it changes from left to right and from top to bottom. That is, what is shown in FIG. 2C is the order of transmission, not the frame number, but the frame number 0 shown in FIG. Next, a frame of frame number 8 shown in FIG. 2B is transmitted at the position indicated by 1 in FIG. 1C, and then the frame shown in FIG. 2B at the position indicated by 2 in FIG. The frame with the number 4 is transmitted, and then the frame with the frame number 12 shown in FIG. 5B at the position indicated by 3 in the figure (C) is transmitted. Each frame is transmitted in the order of frame numbers 10,..., And finally, a frame with frame number 13 shown in FIG. 2 (B) at the position shown in FIG.
[0103]
Next, a reference frame selection method at the time of encoding based on the frame transmission order of the present invention will be described. First, the frame type is schematically shown in FIG. 2A by surrounding it with a square. The head of the encoded frame in the GOV is IVOP as indicated by I, and the second and subsequent encoded frames are P. As shown in the figure, it is constituted by PVOP, the GOV is 0.5 seconds, and the encoded frames included in the GOV are 15 frames. In addition, it is assumed that the order in which the encoded frames are transmitted is the predetermined frame transmission order in the present embodiment as shown in FIG.
[0104]
The frame encoded first in the GOV is frame 0, which is the first frame of the GOV. This frame 0 is encoded as IVOP. Next, as shown by “transmission order” in FIG. 2D, it can be seen that the frame of frame number 8 (frame 8) is the frame to be encoded next.
[0105]
Conventionally, when this frame 8 is to be encoded, one of the input frames input at a certain sample interval is this frame 8. Since this frame 8 needs to be encoded as PVOP, a normal reference frame uses a reference frame of a frame encoded before this frame 8.
[0106]
In other words, in the conventional method, when the frame to be encoded is the frame 8 and the previous encoded frame is the frame 0, the sampling interval when performing the encoding is 8. . This means that when the subsequent frames are encoded, the sampling interval must be 8 and the frame rate in this case is 2 [fps].
[0107]
On the other hand, in the present embodiment, since it is possible to use a reference frame of a frame encoded before the frame to be currently encoded, as shown in FIG. Frame 8 is encoded with reference to frame 0. Subsequently, the frame 4 to be transmitted third can refer to the frame 0 and the frame 8, but here, as illustrated in FIG. 2D, the frame 0 is referred to. Further, the frame 12 transmitted fourth can refer to the frames 0, 8, and 4. Here, as shown in FIG. 2D, the frame 8 closest to the frame 12 is referred to. ing.
[0108]
As described above, in the present embodiment, as shown in FIG. 2D, according to the transmission order, and closest to the reference frame at the already encoded frame position in the GOV. Reference frames can be used.
[0109]
Furthermore, since the present invention performs encoding according to a predetermined frame transmission order, the frame interval at the time of encoding does not occur as described in the conventional method, and the frame interval is gradually shortened. Encoding proceeds. Thus, the frame rate on the decoding side can be increased by the number of frames that can be decoded by the predetermined frame transmission order and the reference frame selection method of the present invention.
[0110]
By adopting the method for determining the frame transmission order as described above, it is possible to transmit the encoded frame via the transmission path in such a frame transmission order and decode it in the order transmitted by the image reception decoding apparatus. When encoding is performed by the image encoding transmission apparatus as possible, a new effect is produced.
[0111]
In the example of the GOV configuration in FIG. 2, initially, playback frames are decoded at a time interval of 8 frames, which is about half of the number of frames M = 15 included in the GOV. Thereafter, the interval between frames to be decoded is further reduced to half of 8/2 = 4 frames, and finally the interval between frames to be decoded becomes one frame, which is the original frame rate.
[0112]
Here, it is assumed that the transmission bit rate of the transmission path is drastically reduced and only half of the encoded frames included in the GOV are transmitted. As described above, 1 GOV is for 0.5 seconds, the number of frames M included in the GOV is M = 15, and the decoding side can receive only 8 frames out of 15 GOV frames. To do.
[0113]
In the conventional method, since the frame rate is 30 [fps] as it is, the images for 8 frames that can be received, that is, the images for about 0.25 seconds, are displayed smoothly, but the remaining images after that are displayed. Since there is no frame to display for about 0.25 seconds, the moving image display is stopped for about 0.25 seconds. This means that the content that occurred in the moving image display in the latter half of 0.25 seconds is completely lost. If such a phenomenon occurs, assuming that video distribution is performed in sequence using VOD (Video On Demand) or surveillance cameras, the possibility of missing important information in the video increases, which is important for video quality. Will have a negative impact.
[0114]
On the other hand, in this embodiment, even if only the first 8 frames in the GOV can be received, the frame rate is automatically set at the time of decoding by the encoding based on the frame transmission order of this embodiment. The encoded frame in the GOV can be configured so that it can be reproduced and displayed at equal intervals as much as possible in units of GOV. In this embodiment, the smoothness of the video display is reduced as the frame rate is reduced, and there is a possibility that the video content that occurred between frames may be lost, but the video content is completely lost. Can be avoided.
[0115]
This effect is even more pronounced when the encoded frame is transmitted in an environment in which the transmission bit rate of the transmission path fluctuates frequently. In the conventional method, when the transmission rate fluctuates frequently, video information is reproduced in small pieces, and it becomes extremely difficult to grasp the contents of the entire video. However, as in the present embodiment, the frame rate dynamically changes according to the received frame, so that the content of the entire video is reproduced without much concern, and the frame rate is extremely high. Since fluctuations can also be avoided, it can be seen that this is a very effective technique for an environment where the transmission rate fluctuates frequently.
[0116]
In addition, a new effect other than the above effects also occurs. Consider a case where the encoded frame transmitted by the transmission path is a bit stream encoded in advance. In the case of a bit stream encoded by the conventional method, the frame rate at the time of decoding is determined at the time of encoding. Therefore, if the frame rate is set as a control specification as necessary, some transcoding means is required.
[0117]
On the other hand, according to the present embodiment, the GOV is reproduced and displayed at a frame rate corresponding to the number of frames decoded so far by decoding the bitstream once generated in units of GOV and stopping the decoding at a certain position. Is possible. Furthermore, in the present embodiment, even if the decoding capability on the image receiving / decoding device side is low, for the same reason, playback and display can be performed at an appropriate frame rate according to the number of frames that can be decoded. It becomes.
[0118]
Next, with regard to the operation of the image coding / transmission apparatus according to the present invention, refer to the block diagram of one embodiment of the image coding / transmission apparatus according to the present invention of FIG. 1 and the flowcharts of FIGS. This will be described below. FIG. 4 is a flowchart showing the operation of the encoder 101 in FIG.
[0119]
Hereinafter, the operation of the encoder 101 in FIG. 1 will be described. When image input is started by the input unit 103, the encoder 101 stores the input frame acquired from the input unit 103 in the buffer buffer 104 (step S201 in FIG. 4). At this point, it is assumed that the number of frames necessary for configuring the GOV has already been stored in the buffer buffer 104.
[0120]
In step S <b> 202, the input / output frame management unit 107 determines whether the buffer buffer 104 stores a sufficient number of frames necessary for configuring the GOV. If enough frames are stored in the buffer 104, the process proceeds to step S203. If there are not enough frames stored in the buffer buffer 104, the process proceeds to the terminal (3) and the processing of the encoding unit 101 is terminated.
[0121]
In step S203, the input / output frame management means 107 connects the switch SW1. When the switch SW1 is connected by the input / output frame management means 107, the buffer buffer 104 can store the number of frames necessary for configuring the GOV in the first multiple-frame storage memory 106. The first multiple-frame storage memory 106 stores the input frames input from the buffer buffer 104 for the number of frames necessary for configuring the GOV (step S204 in FIG. 4).
[0122]
When the number of frames necessary for configuring the GOV is sufficiently stored in the first plural-frame storage memory 106, the input / output frame management means 107 disconnects the switches SW1 and SW5 (step S205 in FIG. 4). Subsequently, the input / output frame management unit 107 notifies the reference frame control unit 110 of an encoding start request (step S206 in FIG. 4). Thereafter, the reference frame control means 110 connects the switches SW2 and SW3 to a position where the first frame of the GOV can be acquired (step S207 in FIG. 4), and further disconnects the switch SW4 (step S208 in FIG. 4). .
[0123]
Subsequently, the reference frame control unit 110 notifies the encoding unit 111 of an encoding start request (step S209 in FIG. 4). Then, the encoding unit 111 acquires an input frame from the first multi-frame storage memory 106 via the switch SW2 (step S210 in FIG. 4), and encodes the input frame (step S211 in FIG. 4). . Then, the encoding unit 111 outputs the encoded frame to the buffer buffer 109 (step S212 in FIG. 4). Further, the local decoding unit 116 acquires an encoded frame from the encoding unit 111 (step S213 in FIG. 4).
[0124]
The local decoding unit 116 decodes the acquired encoded frame to generate a reference frame (step S214 in FIG. 4), and stores the reference frame in the second multiple-frame storage memory 113 via the switch SW3 (FIG. 4). 4 step S215). Next, the reference frame control unit 110 determines whether or not all the encoded frames in the GOV have been prepared from the current encoded frame position based on a predetermined frame transmission order (step S216 in FIG. 4). If all the encoded frames are ready, the process proceeds to step S226 corresponding to terminal (1). If all the encoded frames are not complete, the process proceeds to step S217.
[0125]
When all the encoded frames are not complete, the reference frame control unit 110 switches the switch SW2, the switch SW3, and the switch SW4 to positions where the next frame of the GOV can be acquired based on a predetermined frame transmission unit. Thereafter, the reference frame control unit 110 notifies the encoding unit 111 of an encoding start request (step S218 in FIG. 4).
[0126]
Based on the above encoding start request, the encoding unit 111 acquires an input frame from the first multiple-frame storage memory 106 via the switch SW2, and refers to the second multiple-frame storage memory 113 via the switch SW4. A frame is acquired (step S219 in FIG. 4). Thereafter, the encoding unit 111 performs encoding using the acquired input frame and reference frame (step S220 in FIG. 4), and outputs the obtained encoded frame to the buffer buffer 109 (step S221 in FIG. 4).
[0127]
Next, the local decoding unit 116 acquires an encoded frame from the encoding unit 111 and also acquires a reference frame from the second multiple-frame storage memory 113 via the switch SW4 (step S222 in FIG. 4). Subsequently, the local decoding unit 116 generates a reference frame by decoding the acquired encoded frame and reference frame (step S223 in FIG. 4), and stores the reference frame in the second plurality of frames via the switch SW3. (Step S224 in FIG. 4).
[0128]
Subsequently, the reference frame control means 110 determines whether or not all the encoded frames in the GOV are prepared from the current encoded frame position based on a predetermined frame transmission order (step S225 in FIG. 4). If all the encoded frames are ready, the process proceeds to step S226. If all the encoded frames are not complete, the process proceeds to step S217, and the processes of steps S217 to S224 described above are performed again.
[0129]
If the reference frame control unit 110 determines in step S216 or step S226 that all the encoded frames in the GOV have been prepared, the reference frame control unit 110 notifies the input / output frame management unit 107 and the encoding unit 111 of a GOV encoding stop request. Then (step S226 in FIG. 4), the switch SW2 is disconnected (step S227 in FIG. 4). On the other hand, the input / output frame management means 107 connects the switch SW5 (step S228 in FIG. 4). As a result, all the encoded frames in the GOV are stored from the buffer buffer 109 via the switch SW5 into the third plural-frame storage memory 117 in the encoded transmission unit 102 (step S229 in FIG. 4). Thereafter, the process proceeds to step S202 corresponding to terminal (2).
[0130]
By performing the processing as described above, the operation of the image coding / transmission apparatus of the present embodiment is performed, and various problems in the conventional method can be solved.
[0131]
Next, the operation of the coding transmission unit 102 in FIG. 1 will be described with reference to the flowcharts in FIGS. First, it is determined whether or not the GOV can be acquired from the buffer buffer 109 in the encoding unit 101 (step S301 in FIG. 5). If the GOV can be acquired, the third multi-frame storage memory 117 receives the GOV from the buffer buffer 109. Is acquired (step S302 in FIG. 5). Subsequently, the encoded bitstream analyzing unit 118 acquires the header information in the GOV from the third multi-frame storage memory 117 (step S303 in FIG. 5), and analyzes the header information (step S304 in FIG. 5). ), An encoding transmission start request is sent to the frame transmission order control means 119 (step S305 in FIG. 5).
[0132]
As a result, the frame transmission order control means 119 switches the switch SW6 based on the predetermined frame transmission order (step S306 in FIG. 5), and the encoded frames stored in the third multi-frame storage memory 117 are changed to the predetermined frame transmission order. The encoded bit stream analyzing means 118 is supplied in the order of frame transmission (step S307 in FIG. 5). The encoded bitstream analyzing unit 118 adds / corrects necessary header information to the input encoded frame (step S308 in FIG. 5), and outputs the encoded frame together with the header information to the transmission packet transmitting unit 121. (Step S309 in FIG. 5). The operations in steps S305 to S309 are repeated until all the encoded frames in the GOV are processed (step S310 in FIG. 5).
[0133]
Next, the transmission packet sending unit 121 determines whether or not the encoded frame can be acquired from the encoded bitstream analyzing unit 118 (step S401 in FIG. 6), and acquires the encoded frame if it can be acquired (FIG. 6). Step S402). Then, the transmission packet sending unit 121 packetizes the acquired encoded frame (step S403 in FIG. 6), and outputs the packetized data to the network or storage unit 122 which is the transmission path (step S404 in FIG. 6). ). Finally, the transmission packet sending unit 121 ends the process when all packetized data is output to the network or storage unit 122 (steps S405 and S404 in FIG. 6).
[0134]
Next, an example of an image receiving / decoding device that receives and decodes an encoded bitstream transmitted from the image encoding / transmission device of the present invention via a transmission path will be described. FIG. 7 is a block diagram showing an example of the image receiving / decoding apparatus. As shown in the figure, the image receiving / decoding apparatus receives at least one packet from the image encoding / transmission apparatus shown in FIG. 1 via the network or storage means 203 (122), and encodes the code from at least one packet. A decoding reception unit 201 for reconstructing a GOV and a decoding unit 202 for decoding the reconstructed GOV and decoding using at least one reference frame And a display frame memory 219 for storing the image signal from the decoding unit 202 for one frame, and a display device 220 for displaying the image signal from the display frame memory 219.
[0135]
In FIG. 7, the decoding receiving unit 201, the switch SW12, and the buffer buffer 211 are components that extend the demultiplexer 402 shown in FIG. 10, and the second multi-frame storage memory 213 is a reference shown in FIG. A configuration in which the prediction memory 406 is expanded, and a configuration in which the other configuration units in FIG. 10 are expanded is a decoding unit 216, reference frame control means 212, switches SW13 to SW15, and input / output frame management means 209.
[0136]
Next, the configuration and operation of the decryption receiving unit 201 will be described. The decoding unit receiving unit 201 includes a transmission packet receiving unit 204, an encoded bitstream analyzing unit 205, a frame storage order control unit 206, a switch SW11, and a first multiple frame storage memory 208. First, at least one packet input from the network or storage unit 203 is received by the transmission packet receiving unit 204 and the encoded frame is reconstructed, and then supplied to the encoded bitstream analyzing unit 205.
[0137]
The encoded bitstream analyzing unit 205 analyzes the input header information and requests the frame storage order control unit 206 to perform switching control of the switch SW11 in order to reconstruct the GOV. Thereby, the frame storage order control means 206 switches the switch SW11 based on a predetermined frame storage order, and supplies the encoded frame from the encoded bitstream analysis means 205 to the first plural-frame storage memory 208 for storage. At the same time, the header information is also stored in the first plural-frame storage memory 208.
[0138]
The first multi-frame storage memory 208 is a memory for storing the encoded frames reconstructed by the transmission packet receiving unit 204 and the encoded bit stream analyzing unit 205, and stores encoded frames for at least 1 GOV. It has a memory capacity that can This memory capacity is secured by calculating in advance from the maximum bit rate of the encoding unit and the maximum bit rate of the transmission path.
[0139]
Next, the configuration and operation of the decoding unit 202 will be described. The decoding unit 202 includes an input / output frame management unit 209, switches SW12 to SW15, a buffer buffer 211, a reference frame control unit 212, a second multi-frame storage memory 213, a decoding unit 216, and a buffer buffer 218. The switches SW13 and SW15 are the same in that the reference frame control means 212 selects one frame image signal from the plurality of frame image signals stored in the second plurality of frame storage memory 213. There is a difference in that the switch SW13 outputs the image signal of the selected frame to the decoding unit 216, and the switch SW15 outputs the image signal of the selected frame to the buffer buffer 218.
[0140]
Further, the input / output frame management means 209 controls the storage of the encoded frame from the first plural-frame storage memory 208 to the buffer buffer 211 in order to control the decoding in units of GOV, Manage by switching. Further, the timing for starting encoding in GOV units is managed by notifying the reference frame control unit 212 of a request to start encoding in GOV units.
[0141]
As a result, when the encoded frames in all GOVs are stored in the first multiple-frame storage memory 208, the switch SW12 in the decoding unit 202 is connected under the control of the input / output frame management means 209, and the first multiple-frame storage memory 208 is connected. The encoded frames in the GOV stored in the frame storage memory 208 are temporarily stored in the buffer buffer 211 sequentially in GOV units.
[0142]
Subsequently, the input / output frame management unit 209 notifies the reference frame control unit 212 of a decoding start request. Then, the reference frame control means 212 connects the switch SW13 and the switch SW14 to a position where the first decoded frame of the GOV can be acquired, and disconnects the switch SW12.
[0143]
Subsequently, the reference frame control unit 212 notifies the decoding unit 216 of a decoding start request. Here, for example, the decoding unit 216 includes at least an entropy decoder 403, an inverse quantizer 404, an inverse orthogonal transformer 405, an adder 408, and a motion compensation predictor 407 of the MPEG decoding apparatus illustrated in FIG. Must be configured. The reference prediction memory 406 shown in FIG. 10 includes the second plural-frame storage memory 213 in FIG.
[0144]
When the decoding unit 216 obtains the encoded frame from the buffer buffer 211, the decoding unit 216 performs entropy decoding by the internal entropy decoder, then performs inverse quantization by the inverse quantizer, and then results of performing inverse orthogonal transform. From the output from the motion compensated predictor that receives the image signal of the reference frame from the second multi-frame storage memory 213 via the switch SW13, a decoded frame of the currently encoded frame is obtained.
[0145]
The reference frame control means 212 switches and controls the switch SW14 so that the reference frame decoded by the decoding unit 216 is correctly stored in the corresponding reference frame position of the second multiple-frame storage memory 213 via the switch SW14. Prepare for decoding of the next encoded frame.
[0146]
The second multi-frame storage memory 213 stores at least the number of frames necessary for configuring the GOV so that the reference frame generated by the decoding unit 216 can be stored at the corresponding frame position in the GOV. It has a frame memory. For example, when the frame rate of the input image is 30 [fps], 1 GOV is 0.5 seconds, and the number of encoded frames stored in 1 GOV is 15, the second multi-frame storage memory 213 has at least 15 frames. The memory capacity that can store the reference frames of minutes is secured.
[0147]
Next, the reference frame control means 212 determines whether all the decoded frames in the GOV are aligned in the second multi-frame storage memory 213 from the current decoded frame position based on a predetermined frame transmission order, If all the decoded frames are not complete, the switch SW14 is switched to a position where the next decoded frame of the GOV can be acquired based on a predetermined frame sending means.
[0148]
When the reference frame control means 212 determines that all the encoded frames in the GOV have been decoded by the decoding unit 216 and aligned in the second multi-frame storage memory 213 by repeating the above operation, The GOV decoding stop request is notified to the switch SW14. On the other hand, the input / output frame management means 209 connects the switch SW15.
[0149]
Thus, after all the image signals (decoded reference frames) in the GOV are supplied from the second plural-frame storage memory 213 to the buffer buffer 218 via the switch SW15 and temporarily stored, the display frame memory 219 is then stored. The image is displayed on the display device 220 after being supplied to and temporarily stored.
[0150]
Note that the present invention is not limited to the above embodiment. For example, in FIG. 1, when the encoded frame generated by the encoding unit 111 is stored in the buffer buffer 109 for 1 GOV, the frame numbers 0, 1 ,..., 14, the encoded frames are stored in the third plural-frame storage memory 117 in the original frame number order, and the frame transmission order control unit 119 encodes the encoding unit 111 in the order (FIG. 2). The switch SW6 may be controlled so that the encoded frame having the corresponding frame number is read from the third multi-frame storage memory 117 in the order of transmission shown in (C). However, in this case, transmission frame order determination means for determining the transmission frame order according to the flowchart shown in FIG. 3 is provided in advance in, for example, the encoding unit 101, and the transmission frame order determined by the transmission frame order determination means. The reference frame control means 110 and the frame sending order control means 119 need to be able to refer to each other.
[0151]
The first plurality of frames storing memory 106 may have a memory capacity capable of storing one image frame.
[0152]
In addition, even if the transmission bit rate of the transmission path is reduced by considering the encoding method of the encoded frame in the GOV and the transmission order, the generated GOV cannot be transmitted in a certain time. By discarding the remaining GOVs that could not be transmitted, the number of encoded frames to be transmitted is limited, and by considering the encoding method and transmission order of the encoded frames in the GOV, the image reception decoding device side The frame rate automatically changes according to the number of encoded frames included in the received GOV, and the transmission bit rate of the transmission path is constantly monitored by enabling the frames to be reproduced and displayed at equal intervals as much as possible. Alternatively, the encoding bit rate may be controlled.
[0153]
Furthermore, considering the encoding method and transmission order of the encoded frames in the GOV, even if the transmission bit rate of the transmission path is reduced, the generated GOV cannot be transmitted in a certain time. By discarding the remaining GOVs that could not be transmitted, the number of encoded frames to be transmitted is limited, and by considering the encoding method and transmission order of the encoded frames in the GOV, the image reception decoding device side The frame rate may be automatically changed according to the number of encoded frames included in the received GOV so that the frames can be reproduced and displayed at equal intervals as much as possible. It increases the transmission delay due to a certain time difference required to constantly monitor and detect the change in the transmission bit rate, and reduces the influence on the GOV decoding. It can be carried out.
[0154]
【The invention's effect】
As described above, according to the present invention, at the time of encoding by the encoding unit, the frame stored in the second frame storage memory is encoded from the frames encoded before the current frame to be encoded. Since an arbitrary frame is used as a reference frame, the reference frame that is closest in time can be used. Therefore, information in the GOV is completely reconstructed due to packet loss due to a change in transmission rate. It is possible to send out the encoded bit stream in a predetermined sending frame order even for a transmission line in which the transmission rate fluctuates drastically.
[0155]
Further, according to the present invention, the frame rate on the decoding side is automatically changed according to the number of frames in the GOV accumulated up to the decoding time using the information of the GOV transmitted up to the decoding time, and as much as possible. When an encoded bit stream is transmitted to a transmission path where the transmission rate fluctuates significantly because the encoded bit stream can be transmitted to the image reception encoding apparatus in a predetermined transmission frame order so that frames can be reproduced and displayed at equal intervals. However, the content of the image is not completely lost, and transmission that allows the image receiving and coding apparatus to grasp the content of the entire image can be performed.
[0156]
In addition, according to the present invention, since encoded frames are transmitted in a predetermined frame transmission order so that frames can be reproduced and displayed at equal intervals as much as possible, encoding is performed without always monitoring the transmission bit rate of the transmission path. The bit rate can be controlled. Further, according to the present invention, the transmission bit rate of the transmission line is constantly monitored to detect a change in the transmission bit rate. Can be reduced. As described above, according to the present invention, it is possible to realize image transmission with higher quality than before.
[Brief description of the drawings]
FIG. 1 is a block diagram of an embodiment of an image encoding and transmitting apparatus of the present invention.
FIG. 2 is a diagram for explaining an example of a frame transmission order and a method of using a reference frame at the time of encoding by the image encoding / transmission apparatus of the present invention.
FIG. 3 is a flowchart showing an embodiment of a process for obtaining a frame transmission order according to the present invention.
FIG. 4 is a flowchart for explaining the operation of an encoding unit in the image encoding transmission apparatus of the present invention.
FIG. 5 is a flowchart (No. 1) for explaining the operation of the encoding transmission unit in the image encoding transmission apparatus of the present invention;
FIG. 6 is a flowchart (part 2) for explaining the operation of the encoding transmission unit in the image encoding transmission device of the present invention.
FIG. 7 is a block diagram of an example of an image receiving / decoding apparatus that decodes an encoded frame output by an image encoding / transmission apparatus of the present invention.
FIG. 8 is a block diagram of an example of a conventional encoding apparatus based on MPEG technology.
FIG. 9 is a diagram representing a GOV header in MPEG-4.
FIG. 10 is a block diagram of an example of a conventional decoding device based on MPEG technology.
FIG. 11 is a block diagram of an example of a conventional image encoding / transmission apparatus.
FIG. 12 is a block diagram of an example of a conventional image receiving / decoding apparatus.
[Explanation of symbols]
101 Coding unit
102 Encoded transmission unit
103 Input means (input image)
104, 109, 211, 218 Buffer buffer
106, 208 First multiple frame storage memory
107, 209 Input / output frame management means
110, 212 Reference frame control means
111 Encoding part
113, 213 Second memory for storing a plurality of frames
116 Local decoding part
117 Third multi-frame storage memory
118, 205 Coding bitstream analysis means
119 Frame transmission order control means
121 Transmission packet sending means
122, 203 Network or storage means
201 Decoding receiver
202 Decryption unit
204 Transmission packet receiving means
206 Frame storage order control means
209 I / O frame management means
216 Decoding part
219 Display frame memory
220 Display device
SW1 to SW6, SW11 to SW14 switch

Claims (2)

An image encoding transmission apparatus that encodes an input image signal using a reference frame and transmits the obtained encoded frame according to a predetermined frame transmission order,
A first frame storage memory for storing the input image signal for at least one frame;
A second frame storage memory for storing a predetermined number of reference frames used at the time of encoding;
First switch means provided on the output side of the first frame storage memory;
Second switch means provided on the input side and output side of the second frame storage memory;
In accordance with the predetermined frame transmission order, the first and second switch means are controlled to control the transfer of the input image signal and the reference frame stored in the first and second frame storage memories. Reference frame control means,
An input image signal having a frame number selected by the first switch means from the input image signals stored in the first frame storage memory is stored in the second frame storage memory. An encoding unit that encodes and obtains an encoded frame by using one reference frame selected by the second switch means from among the reference frames;
The encoded frame output from the encoding unit is decoded, and one reference frame used at the time of encoding by the encoding unit is used from among the reference frames stored in the second frame storage memory. A local decoding unit for newly generating a reference frame and storing the new reference frame in the second frame storage memory via the second switch means;
The storage state of the first frame storage memory is monitored to manage the storage state, the encoded frame output from the encoding unit is monitored, the encoded output is managed, and these states are managed. I / O frame management means for notifying the reference frame control means,
A third frame storage memory for storing a group of the encoded frames generated by the encoding unit;
And an output means for packetizing each frame included in the group of encoded frames stored in the third frame storage memory and transmitting the packets in accordance with the predetermined frame transmission order. Transmission equipment. The reference frame control means is means for controlling the first and second switch means with the transmission frame number determined by the calculation means as the predetermined frame transmission order in the determined order. Is
Assuming that the number of encoded frames included in the group of encoded frames is M, and the frame number of the input image signal stored in the first frame storage memory is a natural number from 0 to M−1. Initializing means for initializing the initial value of the variable A to M and the initial value of the count value C to 1 with the frame number 0 as the first transmission frame number;
After initialization by the initialization means, the following formula B = [(A + 1) / 2] ([] is a Gaussian symbol)
And calculating a value of B, and a first calculating means for setting the variable n to 0,
Using the value of B and the value of variable n obtained by the first calculating means, the following formula D = B + 2 × B × n (n is a positive integer)
Second calculating means for calculating D by:
First determination means for determining whether or not the value of D calculated by the second calculation means is less than M;
When it is determined by the first determination means that D <M, the value of D at that time is determined as a transmission frame number, and the variable n and the count value C are incremented by 1, respectively. Sending frame number determining means for calculating D again by the calculating means of 2;
Second determination means for determining whether or not the count value C is greater than M when the first determination means determines that D is equal to or greater than M;
When the second determination means determines that the count value C is less than or equal to M, the value of A is set to the value of B at that time, and then the first calculation means determines the B 2. The setting means for calculating the value, wherein the calculation means performs calculation until the second determination means determines that the count value C is greater than the M. Image encoding transmission apparatus.

Images