JP5530408B2 - Image encoding method, image encoding apparatus, and recording medium - Google Patents

Description

  The present invention relates to an image encoding method for efficiently compressing a moving image signal using correlation between screens, an image decoding method for correctly decoding the image signal, a program for implementing the same in software, and the like.

  In recent years, with the era of multimedia that handles sound, images, and other pixel values in an integrated manner, conventional information media, that is, means for transmitting information such as newspapers, magazines, televisions, radios, telephones, etc., to people have become multimedia. It has come to be taken up as a target. In general, multimedia refers to not only characters but also figures, sounds, especially images, etc. that are associated with each other at the same time. It is an essential condition.

  However, when the information amount of each information medium is estimated as a digital information amount, the amount of information per character is 1 to 2 bytes in the case of characters, whereas 64 kbits per second (phone quality) in the case of speech. In addition, for a moving image, an information amount of 100 Mbits (current television reception quality) or more per second is required, and it is not realistic to handle the enormous amount of information in the digital format as it is with the information media. For example, a video phone has already been put into practical use by an integrated services digital network (ISDN) having a transmission rate of 64 kbps to 1.5 Mbps, but it is not possible to send video from a TV camera as it is through ISDN. Is possible.

  Therefore, what is required is information compression technology. For example, in the case of a videophone, H.264 has been internationally standardized by ITU-T (International Telecommunication Union Telecommunication Standardization Sector). 261 and H.264. H.263 standard video compression technology is used. In addition, according to the information compression technology of the MPEG-1 standard, it is possible to put image information together with audio information on a normal music CD (compact disc).

  Here, MPEG (Moving Picture Experts Group) is an international standard for digital compression of moving image signals, and MPEG-1 has moving image signals up to 1.5 Mbps, that is, information of television signals is about 1/100. It is a standard that compresses up to. In addition, since the transmission speed for the MPEG-1 standard is mainly limited to about 1.5 Mbps, MPEG-2 standardized to meet the demand for higher image quality has 2 to 2 video signals. Compressed to 15 Mbps.

  Furthermore, MPEG-4 with a higher compression rate has been standardized by a working group (ISO / IEC JTC1 / SC29 / WG11) that has been standardizing with MPEG-1 and MPEG-2. MPEG-4 initially introduced not only high-efficiency coding at a low bit rate, but also a powerful error resilience technology that can reduce subjective image quality degradation even if a transmission path error occurs. . Also, standardization activities of JVT (Joint Video Team) are advancing as a next-generation screen coding system jointly by ISO / IEC and ITU, and what is called Joint Model 2 (JM2) is the latest.

  In JVT, unlike conventional video coding, an arbitrary image (picture) can be selected as a reference image from a plurality of images (pictures) as a forward reference image. Here, a picture represents a frame or a field.

  FIG. 35A is an explanatory diagram of image coding in which coding is performed with reference to an image selected from a plurality of reference images stored in a memory. FIG. 35B is a configuration diagram illustrating a configuration of a memory in which an image is stored.

  As shown in FIG. 35B, the memory includes a short-time storage memory and a long-time storage memory. The short-time storage memory stores several pictures decoded immediately before, and refers to so-called MPEG-1 and MPEG-2 P pictures (forward prediction coded pictures) and B pictures (bidirectional prediction coded pictures). Corresponds to an image. The long-time storage memory is used for storing an image signal for a longer time than the short-time storage memory.

  Normally, the short-time storage memory is a FIFO (first-in first-out) memory. When an image exceeding the upper limit of the memory is stored in the short-time storage memory, the image at the oldest time in the short-time storage memory is removed. And a new image is stored in the area. Therefore, when it is usually desired to refer to a reference image that has been removed from the memory by the FIFO mechanism, the reference image is previously moved from the short-time storage memory to the long-time storage memory and stored in the long-time storage memory. Long-term reference is possible. The long-time memory is a method of clearly indicating the area to be saved, and the picture saved in the area can be referred to unless the same area is designated and overwritten.

  FIG. 35A shows a prediction situation at the time of image coding. An image with picture number 2 refers to an image with picture number 0, and an image with picture number 1 is an image with picture number 0 or picture number 2. refer. Similarly, an image with picture number 4 refers to an image with picture numbers 0 and 2, and an image with picture number 6 refers to an image with picture number 0. Furthermore, in the picture with picture number 5, the pictures with picture numbers 0, 2, 4, and 6 can be referred to.

  In FIG. 35 (a), the images with picture numbers 0, 6, and 12 are referred to after a relatively long time, whereas the images with picture numbers 2, 4, and 8 are obtained from the image after a short time. Only referenced.

  Therefore, the memory area for storing images is divided into a short-time storage memory and a long-time storage memory as shown in FIG. 35B, and picture (frame) numbers 0, 6, and 12 are stored in the memory that needs to be stored for a long time. Images can be saved.

  Now, in order to use the memory as shown in FIG. 35 (a) efficiently, advanced memory management is required, and a mechanism for controlling the memory is introduced in JVT.

  The commands that control the memory are:

1. A command for selecting a referenceable image. 2. A command for releasing a memory area in which a picture that is no longer necessary as a reference image for predictive coding is stored in a short-time storage memory Command to move the contents of the short-term storage memory to the long-term storage memory

  In image encoding / decoding, an image with a small prediction error in units of blocks is selected as a reference image from among referenceable images, and thus a signal that indicates the reference image in units of blocks is required. By selecting images that can be referred to in advance, the number of reference image candidates can be narrowed down to an appropriate value, and the number of bits of the reference image instruction signal required for each block can be saved.

  In addition, when moving from the short-time storage memory to the long-time storage memory, it is only useless even if the same contents are stored in both the short-time storage memory and the long-time storage memory. Remove the image.

  FIGS. 36A and 36B are flowcharts showing a conventional image encoding method and image decoding method.

  FIG. 36A shows the operation of the image coding apparatus when a memory area storing a picture that is no longer necessary as a reference image for predictive coding is released. In FIG. 36A, first, the image encoding device encodes an input image to be input (Step 100). After encoding, an unnecessary area (an image that will not be referred to in future encoding) is examined in the memory (Step 101), and it is determined whether there is an unnecessary memory area (Step 102). When it is determined that there is an unnecessary memory area (Yes in Step 102), a command for releasing the unnecessary memory area is encoded as memory management information (Step 103), and the unnecessary memory area is released (an image in the memory is removed). (Step 104), and the process is terminated. On the other hand, if the image coding apparatus determines that there is no unnecessary memory area (No in Step 102), the operation ends in Step 103 and Step 104 without performing the operations.

  Next, an operation performed by the image decoding apparatus when a memory area storing a picture that is no longer necessary as a reference image for predictive encoding is released will be described with reference to the flowchart of FIG. First, the image decoding apparatus decodes the memory management information (Step 110), and decodes the image signal from the encoded signal (Step 111). The image decoding apparatus determines whether there is a memory release command as a result of the investigation (Step 112). If there is a memory release command (Yes in Step 112), there is an image to be removed by the command, or the memory has already been released. It is determined whether the image has been removed (Step 113). If it is determined that it has been released (Step 113: Yes), an error (ERROR) is assumed. This is because in JVT, it is prohibited to send a command for removing the same image again after removing the image from the memory. Therefore, an error occurs when the released memory is released again. On the other hand, if it is determined that the image decoding apparatus has not been released (No in Step 113), the image decoding apparatus releases the memory (Step 114) and ends the process. If it is determined that there is no memory release command (No at Step 112), Step 113 and Step 114 are not performed, and the process ends. Note that Step 110 and Step 111 are out of order and may be switched.

  FIGS. 37A and 37B are flowcharts showing another conventional image encoding method and image decoding method.

  FIG. 37A shows an operation performed by the image coding apparatus when moving an image from the short-time storage memory to the long-time storage memory.

  In FIG. 37A, first, the image encoding device encodes an input image (Step 120). After encoding, it is investigated whether there is an image to be moved to the long-time storage memory (Step 121), and it is determined whether there is an image to be moved (Step 122). If there is an image to be moved (Yes in Step 122), a command indicating how to move to the long-time storage memory is encoded as memory management information (Step 123), and the image is moved to the long-time storage memory according to the command. Then (Step 124), the process is terminated. On the other hand, when the image coding apparatus determines that there is no image to be moved to the long-time storage memory (No in Step 122), the operation of Step 123 and Step 124 is not performed, and the process ends.

  Next, an operation performed by the image decoding apparatus when moving an image from the short-time storage memory to the long-time storage memory will be described with reference to the flowchart of FIG. First, the image decoding apparatus decodes the memory management information (Step 130), and then decodes the image signal from the encoded signal (Step 131). Then, the image decoding apparatus determines whether or not there is a command to move to the long-term storage memory in the decoded memory management information (Step 132). If it is determined (Yes in Step 132), then the command is used. It is determined whether there is an image to be moved or whether it has already been moved (no image exists because it has been removed after movement) (Step 133). In JVT, it is prohibited to send a command to move the same image to the long-time storage memory again after moving to the long-time storage memory. In case it is supposed to be an error. Therefore, if it is determined that the image decoding apparatus has been moved to the long-time storage memory (Yes in Step 133), an error (ERROR) is determined. If it is determined that it has not been moved, the image decoding apparatus moves to the long-time storage memory ( (Step 134) The processing is terminated.

  On the other hand, when it is determined that there is no command to move to the long-time storage memory (No in Step 132), the image decoding apparatus does not perform Step 133 and Step 134 and ends the process. Note that Step 130 and Step 131 are out of order and may be switched.

  38A and 38B are flowcharts showing still another conventional image encoding method and image decoding method.

  First, an operation performed by the image encoding device when selecting a referenceable image will be described with reference to the flowchart of FIG.

  First, the image encoding device selects a reference image (usually a reference image close in time) that is expected to have a high correlation with the encoded image as a reference image candidate (Step 200). Next, the instruction information (a kind of memory management information) indicating the selected reference image candidate is encoded (Step 201), and is encoded with reference to an appropriate reference image in block units from the selected reference image candidates ( (Step 202), the process is terminated. Note that Step 201 and Step 202 are out of order and may be interchanged.

  Next, an operation performed by the image decoding apparatus when selecting a referenceable image will be described with reference to the flowchart of FIG.

  First, the image decoding apparatus decodes instruction information, which is a kind of memory management information (Step 210), and as a result, selects a reference image candidate from the memory (Step 211), and selects from the selected reference image candidates. An appropriate reference image is selected in block units and decoded while being referenced (Step 212), and the process is terminated.

"Working Draft Number 2, Revision 0 (WD-2)", Joint Video Team (JVT) of ISO / IEC and ITU-T VCEG, JVT-B118, 13 March, 2002

  In such a conventional image encoding method and image decoding method, a command for removing an unnecessary image from a memory and a command for moving an image from a short-time storage memory to a long-time storage memory are transmitted by an image encoding device. Encoded and output, transmitted to the image decoding device and decoded, but since the number of transmissions is limited to only one picture, if the picture with the command is lost due to a transmission error etc. Since the image arrangement in the memory cannot be restored correctly, the image cannot be decoded.

  In addition, when selecting a reference image in encoding and decoding of an image, if only an image that is temporally close is used as a reference image candidate, the image decoding scalability (the prediction structure of FIG. In the example, an I picture or a P picture can be decoded without decoding a B picture, or other P pictures can be decoded without decoding a P picture with picture numbers 4, 10, 16). The optimum coding in consideration cannot be performed. That is, an image temporally close to an image with picture number 6 is an image with picture numbers 4 and 2, but since only an image with picture number 0 can actually be referred to, an image with picture numbers 4 and 2 that cannot be referred to is referred to as a reference image. The coding efficiency is not so good.

  Further, in the conventional image encoding method, a command for removing unnecessary images in the memory and a command for moving an image from the short-time storage memory to the long-time storage memory are transmitted along with the image not stored in the memory. This prohibits the command transmission of flexible memory management information. There are the following reasons for prohibiting the transmission of the command accompanying an image not stored in the memory. In other words, images that are not stored in memory are the least important, and it is highly possible that they will not be decoded due to scalability. Therefore, the image layout in the memory is correctly restored without decoding the commands associated with images that are not stored in memory. This is to avoid what cannot be done.

  Accordingly, in order to solve the above-described problems, the present invention provides an image encoding method and an image decoding method that can be correctly restored even if some memory management information is lost due to a transmission path error, and a reference image that can be referred to. It is an object of the present invention to provide an image encoding method, an image decoding method, and the like that improve the encoding efficiency by more appropriately selecting the candidates.

  In order to solve this problem, an image encoding method according to an aspect of the present invention is an image encoding method in which encoding is performed with reference to a reference picture selected from a plurality of reference pictures stored in a memory. Then, referring to the selected reference picture, the encoding target picture is encoded, and the first memory management information for managing the reference picture stored in the memory is stored in the encoded encoding target picture. The first memory management information is encoded again as second memory management information with a picture different from the encoding target picture, and the memory management information is stored in the memory. It is information that designates a memory area that becomes unnecessary and is released.

  As a result, since the memory management information is encoded and output a plurality of times, even if a transmission path error occurs when the memory management information is transmitted to the decoding device, one of the memory management information transmitted a plurality of times is transmitted. Since it is assumed that the picture is decoded, there is a high possibility that the picture can be correctly restored.

  Further, the memory management information may be information for designating a memory area that becomes unnecessary and is released in the memory.

  As a result, even if a transmission path error occurs when the data is transmitted to the decoding apparatus, it is possible to free a memory area that does not require memory.

  Further, the memory includes a short-time storage memory having a short reference picture storage time and a long-time storage memory having a reference picture storage time longer than the short-time storage memory, and the memory management information includes the short-time storage memory. It may be information specifying a reference picture to be moved from the storage memory to the long-time storage memory.

  As a result, even when a transmission path error occurs during transmission to the decoding apparatus, it is possible to correctly specify the reference picture to be moved from the short-time storage memory to the long-time storage memory.

  The recording medium of the present invention is a recording medium on which a data stream encoded by referring to a reference picture selected from a plurality of reference pictures stored in a memory is recorded, and refers to the selected reference picture Then, the encoded data obtained by encoding the encoding target picture and the first memory management information for managing the reference picture stored in the memory are encoded in association with the encoding target picture. Information encoded data, and management information re-encoded data obtained by encoding the first memory management information again as second memory management information in association with a picture different from the encoding target picture, The second memory management information is accompanied by information for specifying the encoding target picture.

  The image encoding device of the present invention is an image encoding device that performs encoding by referring to a reference picture selected from a plurality of reference pictures stored in a memory, and refers to the selected reference picture. Then, the first memory management information for encoding the encoding target picture and managing the reference picture stored in the memory is encoded in association with the encoded encoding target picture, and the first Management information encoding means for encoding again the second memory management information as a second memory management information in association with a picture different from the picture to be encoded, wherein the management information encoding means The memory management information is associated with information for specifying the encoding target picture.

As a result, since the memory management information is encoded and output a plurality of times, even if a transmission path error occurs when the memory management information is transmitted to the decoding device, one of the memory management information transmitted a plurality of times is transmitted. Since it is assumed that the picture is decoded, the possibility that the picture can be correctly restored increases.
An image encoding method according to the present invention is an image encoding method for encoding with reference to a reference picture selected from a plurality of reference pictures stored in a memory, and refers to the selected reference picture, The first memory is encoded by encoding the first memory management information for encoding the encoding target picture and managing the reference picture stored in the memory in association with the encoded encoding target picture. The management information is again encoded as second memory management information with a picture different from the picture to be encoded, and the information specifying the picture to be encoded is attached to the second memory management information. The image management method may be characterized in that the memory management information is information that designates a memory area that becomes unnecessary and is released in the memory.
The memory includes a short-time storage memory having a short reference picture storage time and a long-time storage memory having a reference picture storage time longer than the short-time storage memory, and the memory management information further includes the short-time storage memory. Information for designating a reference picture to be moved from the storage memory to the long-time storage memory may be included.
An image encoding apparatus according to the present invention is an image encoding apparatus that performs encoding by referring to a reference picture selected from a plurality of reference pictures stored in a memory, and refers to the selected reference picture, First encoding means for encoding the encoding target picture and first memory management information for managing the reference picture stored in the memory are attached to the encoded encoding target picture. A second encoding means for encoding, the first memory management information is encoded again as second memory management information in association with a picture different from the encoding target picture, and the second memory management information is encoded. Management information encoding means for associating the memory management information with information for specifying the encoding target picture, and the memory management information designates a memory area that is unnecessary and freed in the memory. The information may be an image coding apparatus characterized by.
A recording medium according to the present invention is a recording medium on which a data stream encoded by referring to a reference picture selected from a plurality of reference pictures stored in a memory is recorded, and refers to the selected reference picture. Then, the management information encoded by encoding the encoded data of the encoding target picture and the first memory management information for managing the reference picture stored in the memory in association with the encoding target picture The encoded data and the first memory management information are encoded again as second memory management information in association with a picture different from the picture to be encoded, and the code is added to the second memory management information. Management information re-encoded data associated with information for specifying a picture to be encoded, and the memory management information becomes unnecessary and freed in the memory Is information specifying, may be a recording medium, characterized in that.

  Note that the present invention can be realized not only as an apparatus but also as a method using steps as processing units constituting the apparatus, as a program for causing a computer to execute the steps, or as a computer read recording the program. It can also be realized as a possible recording medium such as a CD-ROM, or as information, data or a signal indicating the program. These programs, information, data, and signals may be distributed via a recording medium such as a CD-ROM or a communication network such as the Internet.

  As described above, according to the image encoding method and the image decoding method according to the present invention, an image encoding method and an image decoding method that can be correctly restored even if some memory management information is lost due to a transmission path error. Thus, it is possible to realize an image encoding method and an image decoding method that can more appropriately select reference image candidates that can be referred to and increase the encoding efficiency, and its practical value is high.

It is a block diagram which shows the structure of the image coding apparatus of this invention. It is a flowchart which shows the image coding method in Embodiment 1 of this invention. It is a block diagram which shows the structure of the image decoding apparatus of this invention. It is a flowchart which shows the image decoding method in Embodiment 2 of this invention. It is a flowchart which shows the image coding method in Embodiment 3 of this invention. It is a flowchart which shows the image decoding method in Embodiment 4 of this invention. It is a flowchart which shows the image coding method in Embodiment 5 of this invention. It is a flowchart which shows the image coding method in Embodiment 6 of this invention. It is a flowchart which shows the image coding method in Embodiment 7 of this invention. (A) is explanatory drawing which shows the relationship between the picture number of an image, a preservation | save picture number, and transmission order, (b) shows the relationship between the picture number to decode, the preserved picture number, and the deleted picture number. FIG. 4C is a relationship diagram illustrating another relationship between a picture number to be decoded, a stored picture number, and a deleted picture number. It is a correspondence figure which shows the command of the memory management information in this invention. It is a flowchart which shows the command execution procedure in Embodiment 8 of this invention. It is a schematic diagram which shows the relationship between the header information and frame data in the encoded signal of each picture. It is a schematic diagram which shows the command of the memory management information in the header information of an encoding signal. It is explanatory drawing which shows the relationship between the picture number of each image, a preservation | save picture number, and transmission order. It is a flowchart which shows the method of encoding an initialization command. It is a flowchart which shows the method of decoding the encoded initialization command. It is a response | compatibility figure which shows the command of the memory management information used for Embodiment 8 of this invention. It is a flowchart which shows the image coding method using the initialization resending command in this invention. 6 is a flowchart illustrating a method of decoding an encoded initialization retransmission command according to the present invention. It is explanatory drawing which shows the other relationship between the picture number of each image, a preservation | save picture number, and transmission order. It is a response | compatibility figure which shows the command of the memory management information used in Embodiment 9 of this invention. It is a flowchart which shows the image coding method in Embodiment 9 of this invention. It is a flowchart which shows the image decoding method in Embodiment 9 of this invention. (A) is a correspondence diagram showing command contents and additional information, and (b) is a correspondence diagram showing command execution timing. It is a schematic diagram which shows the command of the memory management information in the header information of an encoding signal. It is a schematic diagram which shows the command of the memory management information in the header information of another encoded signal. It is a schematic diagram which shows the data stream structure encoded by the slice unit. (A) (b) is a schematic diagram which shows the data stream structure encoded in slice unit. (A) (b) (c) is a storage medium for storing a program for realizing the image encoding method and the image decoding method of Embodiments 1 to 10 of the present invention by a computer system. It is explanatory drawing about. It is a block diagram which shows the whole structure of the content supply system with which the image coding method and image decoding method which concern on this invention are used. It is an external view which shows an example of the mobile telephone in which the image coding method and image decoding method which concern on this invention are used. It is a block diagram which shows the structure of the mobile phone. It is a block diagram which shows the structure of the system for digital broadcasting in which the image coding method and image decoding method which concern on this invention are used. (A) is explanatory drawing of the image encoding encoded with reference to the image selected from the some reference image preserve | saved at memory, (b) is a block diagram which shows the structure of the memory where an image is preserve | saved. is there. (A) is a flowchart which shows the conventional image coding method, (b) is a flowchart which shows the conventional image decoding method. (A) is another flowchart which shows the conventional image coding method, (b) is another flowchart which shows the conventional image decoding method. (A) is another flowchart which shows the conventional image coding method, (b) is another flowchart which shows the conventional image decoding method.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
First, the first embodiment will be described.

  FIG. 1 is a block diagram showing a configuration of an image encoding device for realizing an image encoding method according to the present embodiment.

  The image encoding apparatus 100 includes a memory information control unit 101, a short-time storage memory management unit 102, a long-time storage memory management unit 103, a non-storage memory management information unit 104, a management information encoding unit 105, a reference An image selection unit 106, a storage area designation unit 107, a reference area designation unit 108, an image memory 109, an image decoding unit 111, an image coding unit 110, a variable length coding unit 112, a counter 113, , Counter 114 and the like.

  The reference image selection unit 106 selects a reference image candidate from the importance instruction signal Pri and the picture type information PicType input from the outside, and notifies the memory information control unit 101 of them.

  The memory information control unit 101 determines whether forward or backward images (both images) can be referred to based on the picture type information PicType, and instructs the reference area designating unit 108 to respond from the image memory 109. The reference image is output to the image encoding unit 110.

  The image encoding unit 110 encodes the input image signal Vin with reference to the reference image output from the image memory 109, and the variable length encoding unit 112 further performs variable length encoding and outputs an image encoded stream VideoStr. The output of the image encoding unit 110 is also decoded by the image decoding unit 111 to become a decoded image, which is stored in the image memory 109 as a reference image.

  At this time, the memory location where the decoded image can be stored in the image memory 109 is designated as follows. The memory information control unit 101 makes an inquiry to the short-time storage memory management unit 102, specifies the memory location from which the image has been removed in the short-time memory, and the storage region designation unit 107 records the decoded image at the memory location. An instruction is issued to the image memory 109.

  The short-time storage memory management unit 102 notifies the memory information control unit 101 of a command to detect and remove (release the memory) unnecessary (not referenced) images in the short-time storage memory. Further, the long-time storage memory management unit 103 notifies the memory information control unit 101 of an instruction to move an image in the short-time storage memory to the long-time storage memory. The unnecessary image removal (memory release) command and the command to move the image in the short-time storage memory to the long-time storage memory are encoded by the management information encoding unit 105 and are stored in the memory management information stream CtlStr. Become.

  On the other hand, in order to prevent the memory management information from being lost due to a part of the memory management information stream CtlStr being lost due to a transmission path error, the short-time storage memory counter 113 and the long-time storage memory counter 114 are prevented. Thus, the number of times of encoding an unnecessary image removal command or a command for moving an image in the short-time storage memory to the long-time storage memory is measured, and the command can be transmitted a plurality of times as necessary.

  In addition, the non-storage memory management information unit 104 is accompanied by an instruction to remove unnecessary images and an instruction to move an image in the short-time storage memory to the long-time storage memory along with an image having low importance and difficult to decode. The memory information control unit so that if the command is encoded along with an image having a low importance level, the command is encoded again along with an image having a higher importance level. 101 is instructed.

  Next, the image coding method in Embodiment 1 of this invention is demonstrated. FIG. 2 is a flowchart showing an image encoding method according to Embodiment 1, and shows operations performed by the image encoding apparatus 100 shown in FIG. 2 that have the same operations as those in FIG. 36A are denoted by the same symbols.

  The feature of the image coding method shown in FIG. 2 is that when an unnecessary image (picture) exists as a reference image for predictive coding in the memory, the memory area in which the image is stored is released (the image is saved). (Removing) It is to repeatedly encode the command of the memory management information. Thus, by repeatedly encoding the memory management information command, even if one memory management information command disappears due to a transmission path error, it is stored in the memory from the other memory management information command. Since the image management information can be restored, there is a high possibility that the image can be correctly restored even if there is a transmission path error.

  In FIG. 2, first, an input image is encoded (Step 100). After encoding, an unnecessary area (an image that will not be referred to in future encoding) is examined in the memory (Step 101), and it is determined whether there is an unnecessary memory area (Step 102). If there is an unnecessary memory area (Yes in Step 102), the management information encoding unit 105 encodes a command for releasing the unnecessary memory area as memory management information (Step 103). Then, the unnecessary memory area is released (Step 104). If there is no unnecessary memory area (No in Step 102), the operations in Step 103 and Step 104 are not performed.

  Next, a command for releasing an unnecessary memory area is encoded as memory management information in association with encoding of an image (an image before an encoding target) encoded immediately before by the memory information control unit 101. If the command is not encoded (No at Step 105), the process is terminated. If the command is encoded (Yes at Step 105), the management information encoding unit 105 The command for releasing the unnecessary memory area is encoded again as memory management information (Step 106), and the process is terminated.

  In this way, when a command for releasing unnecessary memory (in the memory management information) is encoded by encoding the previous image, the command in the memory management information is encoded again. The memory management information encoded in association with the encoding of the immediately preceding image and the memory management information encoded again are output from the image encoding device, transmitted to the image decoding device, and decoded.

  In addition, when a command for releasing an unnecessary memory area is encoded along with the encoded signal of the image encoded immediately before in Step 105, the command is encoded again. The command may be accompanied by an image several images before, or may be repeatedly encoded as memory management information and transmitted along with a plurality of images.

  In addition, it is only necessary that the command for releasing the unnecessary memory area can be transmitted as memory management information a plurality of times. When the command is encoded and transmitted again, it is not necessarily transmitted along with the encoded signal of the image. do not have to.

  In addition, when retransmitting a memory management information command, the retransmitted command may not be in the same stream as the encoded picture, but may be transmitted as a separate stream, for example. May be recorded in this area.

  As described above, by transmitting a command for releasing unnecessary memory areas (in memory management information) multiple times, even if a transmission path error occurs, one of the commands transmitted multiple times is transmitted. Therefore, there is a high possibility that the image can be correctly restored.

(Embodiment 2)
Next, a second embodiment of the present invention will be described.

  FIG. 3 is a block diagram of an image decoding apparatus for realizing the image decoding method according to the second embodiment.

  The image decoding apparatus 200 includes a memory information control unit 201, a short-time storage memory management unit 202, a long-time storage memory management unit 203, a management information decoding unit 205, a storage region specification unit 207, and a reference region specification. Unit 208, image memory 209, image decoding unit 210, variable length decoding unit 212, and the like.

  Based on the picture type information PicType, the memory information control unit 201 determines whether the front and rear images or both of the images can be referred to the encoding target, and instructs the reference region designating unit 208 to From 209, the corresponding reference image is output to the image decoding unit 210.

  The variable length decoding unit 212 decodes the encoded stream VideoStr, and the image decoding unit 210 further decodes it, outputs it as a decoded image signal Vout, and stores it as a reference image in the image memory 209.

  At this time, the memory location where the decoded image can be stored in the image memory 209 is designated as follows. The memory information control unit 201 makes an inquiry to the short-time storage memory management unit 202 to identify the memory location from which the image has been removed, and the image memory 209 so that the storage region designation unit 207 records the decoded image at the memory location. To give instructions.

  The management information decoding unit 205 decodes the memory management information stream CtlStr and notifies the short-time storage memory management unit 202 of information on unnecessary (unreferenced) images in the short-time storage memory through the memory information control unit 201. The long-time storage memory management unit 203 is notified of an instruction to move the image in the short-time storage memory to the long-time storage memory.

  Next, the image decoding method in Embodiment 2 of this invention is demonstrated. FIG. 4 is a flowchart showing an image decoding method according to the second embodiment, and shows operations performed by the image decoding apparatus 200 shown in FIG. In FIG. 4, the same symbols are attached to the same operations as those in FIG. 36 (b).

  If the image encoding device transmits a command to release an unnecessary memory area multiple times, the image decoding device releases the same image area in the memory multiple times unless the command is lost due to a transmission path error. The command to be received will be received. Therefore, it is necessary to realize an image decoding method for judging that the memory can be correctly received without processing it as an error even when the image decoding apparatus receives a command to reopen the memory area that has already been released. . In the present embodiment, such an image decoding method is realized.

  In FIG. 4, first, the management information decoding unit 205 decodes the memory management information (Step 110). Next, the image signal is decoded from the encoded signal (Step 111). Then, the memory information control unit 201 determines whether or not there is a command to release the memory in the decrypted memory management information (Step 112). If there is a command to release the memory (Yes in Step 112), it is determined whether there is an image to be removed by the command or whether the image has already been released (removed) (Step 113), and if the memory has been released (Step 113). Yes) The process ends without performing any processing. Otherwise, the memory is released (Step 114), and the processing ends. On the other hand, when there is no memory release command (No in Step 112), the operation is terminated without performing Step 113 and Step 114. Note that Step 110 and Step 111 are out of order and may be switched.

  Even if the image decoding apparatus 200 receives and transmits the command for releasing the same image area in the memory a plurality of times by the image encoding method of the first embodiment by the operation as described above, the command is transmitted a plurality of times. Since it is not processed as an error, an image decoding method capable of correctly decoding can be realized.

  The command for releasing the unnecessary memory area only needs to be transmitted as memory management information a plurality of times. When the command is encoded and transmitted again, it is not necessarily transmitted along with the encoded signal of the image. There is no need to be.

  In addition, when retransmitting a memory management information command, the retransmitted command may not be in the same stream as the encoded picture, but may be transmitted as a separate stream, for example. May be recorded in this area.

(Embodiment 3)
Next, an image encoding method according to Embodiment 3 will be described. FIG. 5 is a flowchart showing an image encoding method according to Embodiment 3, and shows operations performed by the image encoding apparatus 100. In FIG. 5, the same operation as that in FIG.

  A feature of the present embodiment is that when there is an image to be moved from the short-time storage memory to the long-time storage memory in the memory, a command of memory management information for moving the image is repeatedly encoded. By re-encoding the memory management information command, even if one memory management information command disappears due to a transmission path error, the image management information stored in the memory from the other memory management information command Therefore, there is a high possibility that an image can be correctly restored even if there is a transmission path error.

  In FIG. 5, first, an input image is encoded (Step 120). It is investigated whether there is an image to be moved to the long-time storage memory after encoding (Step 121). Then, the memory information control unit 101 determines whether there is an image that moves to the long-time storage memory (Step 122). If there is an image to be moved (Yes in Step 122), the management information encoding unit 105 encodes a command indicating how to move to the long-time storage memory as memory management information (Step 123). Then, the image is moved to the long-time storage memory according to the command (Step 124).

  Next, whether the memory information control unit 101 encodes, as memory management information, a command that moves to the long-term storage memory in association with the encoded signal of the image encoded immediately before (the image prior to the encoding target) If it is not encoded (No in Step 125), the process ends. If it is encoded (Yes in Step 125), the management information encoding unit 105 moves to the long-time storage memory. The command to be encoded is encoded again as memory management information (Step 126), and the process is terminated.

  As described above, when a command for moving to the storage memory for a long time (in the memory management information) is encoded by encoding the previous image, the command in the memory management information is encoded again. The memory management information encoded in association with the encoding of the immediately preceding image and the memory management information encoded again are output from the image encoding device, transmitted to the image decoding device, and decoded.

  It should be noted that the command is encoded again when the command to move to the long-time storage memory is encoded along with the encoded signal of the image encoded immediately before at Step 125. The command may be attached to an image several images before, or may be repeatedly encoded as memory management information and attached to a plurality of images.

  Also, it is only necessary that the command for moving to the long-term storage memory can be transmitted multiple times as memory management information, and the command is not necessarily transmitted when it is encoded again and transmitted. do not have to.

  In addition, when retransmitting a memory management information command, the retransmitted command may not be in the same stream as the encoded picture, but may be transmitted as a separate stream, for example. May be recorded in this area.

  As described above, a command for moving an image to a long-time storage memory is encoded and transmitted multiple times, so that even if a transmission path error occurs, one of the commands transmitted multiple times is transmitted. Since it is considered that the image is decoded, there is a high possibility that the image can be correctly restored.

(Embodiment 4)
Next, an image decoding method according to the fourth embodiment will be described.

  If the image encoding device transmits a command that moves to the long-term storage memory multiple times, the image decoding device will extend the same image area in the short-time storage memory unless the command is lost due to a transmission path error. The command to move to the time storage memory will be received multiple times. Therefore, it is necessary to realize an image decoding method for determining that the image has been correctly received without processing as an error even when the image decoding apparatus receives a command for moving an image that has already been moved. The feature of the image decoding method in the present embodiment is the realization of such an image decoding method.

  FIG. 6 is a flowchart showing an image decoding method according to the fourth embodiment, and shows the operation of the image decoding apparatus 200 shown in FIG. In FIG. 6, components having the same operation as those in FIG.

  In FIG. 6, first, the management information decoding unit 205 decodes the memory management information (Step 130). Then, the image signal is decoded from the encoded signal (Step 131).

  Then, the memory information control unit 201 determines whether or not there is a command for moving the image to the long-time storage memory in the decoded memory management information (Step 132). If there is a command to move to the storage memory for a long time (Yes at Step 132), it is determined whether there is an image to be moved by the command or whether it has already been moved (there is no image because it has been removed after the movement) ( Step 133) If it has been moved to the long-time storage memory (Yes in Step 133), the process ends without performing any processing. Otherwise, it moves to the long-time storage memory (Step 134) and ends the process.

  On the other hand, when there is no command to move to the long-term storage memory (No in Step 132), the processing is terminated without performing Step 133 and Step 134. Note that Step 130 and Step 131 are out of order and may be switched.

  With the operation as described above, the image decoding method according to the third embodiment can realize an image decoding method that can be correctly decoded even if a command for moving an image to a long-term storage memory is encoded and transmitted a plurality of times. .

  It is only necessary to transmit the command for moving to the long-term storage memory as memory management information a plurality of times, and when the command is encoded again and transmitted, it is transmitted along with the encoded signal of the image. There is no need to be.

  In addition, when retransmitting a memory management information command, the retransmitted command may not be in the same stream as the encoded picture, but may be transmitted as a separate stream, for example. May be recorded in this area.

(Embodiment 5)
Next, an image encoding method in the present embodiment will be described. FIG. 7 is a flowchart showing an image coding method according to the fifth embodiment, and shows the operation of the image coding apparatus 100 shown in FIG. In FIG. 7, the same operation as that in FIG.

  The feature of this embodiment shown in FIG. 7 is that when an unnecessary image exists in the memory, the command of the memory management information for removing the image is repeatedly encoded, and at least once is attached to the important screen stored in the memory. Is transmitted. Even when the memory management information command is repeatedly encoded, if the memory management information command is transmitted along with the non-important image, the memory management information is not decoded when all the non-important images are not decoded. This command cannot be acquired.

  For example, in FIG. 35 (a), the picture number 4 image becomes unnecessary after the picture number 5 image is encoded. Therefore, a memory area in which the picture number 4 image is attached to the picture number 5 image. The command to be released can be encoded.

  However, in addition to encoding the command for releasing the memory area in which the picture with the picture number 4 is attached to the picture with the picture number 5, any of the commands attached to the picture with the picture number 7 is the most. The command is encoded along with a B picture having a low importance level (less image quality deterioration when not decoded). These B pictures may not be decoded, and a command for releasing a memory area with an image of picture number 4 will not be decoded, and management information in the memory cannot be reproduced correctly. Therefore, at least once, it is necessary to encode a command that has a high degree of importance and is necessarily decoded and frees an image area accompanying an image stored in a memory.

  In FIG. 7, first, an input image is encoded (Step 100). After encoding, an unnecessary area (an image that will not be referred to in future encoding) is examined in the memory (Step 101), and it is determined whether there is an unnecessary memory area (Step 102). If there is an unnecessary memory area (Yes in Step 102), the management information encoding unit 105 encodes a command for releasing the unnecessary memory area as memory management information (Step 103). Then, the unnecessary memory area is released (Step 104). If there is no unnecessary memory area (No in Step 102), the operations in Step 103 and Step 104 are not performed.

  Next, the memory information control unit 101 determines whether a command encoded in the past and freeing an unnecessary memory area is encoded accompanying an important image (decoded and stored in the memory). If (Step 140) is encoded along with the important image (Yes at Step 140), the process ends. If it is not encoded along with the important image (No at Step 140), the management information encoding unit 105 Encodes the command for releasing the unnecessary memory area again as memory management information (Step 141), and ends the process.

  As a result, a command for releasing an unnecessary memory area is encoded along with the important image.

  As described above, since the command is attached to the important image that is decoded and stored in the memory, the command is decoded, and there is a possibility that the image can be correctly restored when a transmission path error occurs. Becomes higher.

  It is only necessary to transmit the command for releasing the unnecessary memory area as memory management information a plurality of times. When the command is encoded and transmitted again, it is not necessarily transmitted along with the encoded signal of the image. do not have to.

  In addition, when retransmitting a memory management information command, the retransmitted command may not be in the same stream as the encoded picture, but may be transmitted as a separate stream, for example. May be recorded in this area.

(Embodiment 6)
Next, the image coding method according to the present embodiment will be described. FIG. 8 is a flowchart showing an image encoding method according to the sixth embodiment. FIG. 8 shows the operation of the image coding apparatus 100 shown in FIG. In FIG. 8, those having the same operation as those in FIG.

  The feature of this embodiment shown in FIG. 8 is that the command of the memory management information for moving the image to the long-time storage memory is repeatedly encoded, and is attached to the important screen (decoded and stored in the memory) at least once. Is transmitted. Even when a memory management information command for moving an image to a long-term storage memory is repeatedly encoded, if a memory management information command is transmitted along with a non-important image, an unimportant image is If all of them are not decrypted, the command of the memory management information cannot be acquired.

  In FIG. 8, first, the input image is encoded (Step 120). After encoding, it is investigated whether there is an image to be moved to the long-time storage memory (Step 121), and it is determined whether there is an image to be moved (Step 122).

  If there is an image to be moved (Yes in Step 122), a command indicating how the management information encoding unit 105 moves to the long-time storage memory is encoded as memory management information (Step 123), and the image is transmitted according to the command. Is moved to the long-time storage memory (Step 124).

  Next, the memory information control unit 101 determines whether or not the command for moving to the long-time storage memory encoded in the past is encoded accompanying the important image (decoded and stored in the memory). If it is attached to the important image (Yes in Step 150), the process ends. If not attached to the important image (No in Step 150), the management information encoding unit 105 stores it in the long-time storage memory. The moving command is encoded again as the memory management information (Step 151), and the process ends.

  As a result, the command for moving the image to the long-term storage memory is encoded along with the important image.

  As described above, since the command is attached to the important image that is decoded and stored in the memory, the command is decoded, and there is a possibility that the image can be correctly restored when a transmission line error occurs. Becomes higher.

  It is only necessary to transmit the command for moving to the long-term storage memory as memory management information a plurality of times. When the command is encoded and transmitted again, it is not necessarily transmitted along with the encoded signal of the image. There is no need to be.

  In addition, when retransmitting a memory management information command, the retransmitted command may not be in the same stream as the encoded picture, but may be transmitted as a separate stream, for example. May be recorded in this area.

(Embodiment 7)
An image encoding method according to Embodiment 7 will be described.

  A feature of the present embodiment is an image encoding method in which encoding is performed with reference to a reference image selected according to the importance of the image.

  FIG. 9 is a flowchart showing an image coding method according to Embodiment 7 of the present invention. FIG. 9 shows operations performed by the image coding apparatus 100 shown in FIG.

  In FIG. 9, first, the importance level of each image to be encoded is set (Step 160). For example, the importance of I pictures and P pictures is high, and the importance of B pictures is low. Also, even for the same P picture, the importance of P pictures that are referenced from many images is high, and the importance of P pictures that are not often referenced is low.

  Next, an image having an importance level equal to or higher than that of the encoding target image is selected from the reference images in the memory and set as reference image candidates (Step 161). For example, a B picture can refer to an I picture and a P picture, but a P picture excludes a less important P picture from the reference image candidates.

  Next, the instruction information (a kind of memory management information) indicating the selected reference image candidate is encoded (Step 162), and is encoded with reference to an appropriate reference image in block units from the selected reference image candidates ( Step 163). Note that Step 162 and Step 163 are out of order and may be interchanged.

  In this way, an image having an importance level lower than the importance level of the encoding target image is not included in the reference image candidates.

  As described above, referencing an image that cannot be referred to when a stream that can achieve scalability is generated by not including an image with an importance lower than the importance of the encoding target image in the reference image candidates. Since it can be excluded from image candidates, encoding efficiency is improved.

  Here, the image coding method performed according to the importance of the image set as described above will be specifically described with reference to FIG.

  FIG. 10A shows a number (picture (frame) number) given to each frame, a number when each frame is stored in the memory (stored picture (frame) number), and each frame is transmitted. It is explanatory drawing which shows the relationship of the number (transmission order) which shows an order.

  In FIG. 10A, the I picture with picture number 0 is stored in the memory because it does not refer to other pictures, and the stored picture number is 0. Next, since the P picture with the picture number 2 referring to the I picture with the picture number 0 is saved in the memory, the saved picture number with respect to the P picture with the picture number 2 becomes 1. Next, since the B picture of picture number 1 that refers to the I picture of picture number 0 and the P picture of picture number 2 is stored in the memory, the stored picture number of the B picture of picture number 1 is 2. The order in which each picture is transmitted is the order stored in the memory. The relationship between the picture number, the stored picture number, and the transmission order is determined by the same procedure.

  Next, an example of the relationship between the picture number to be decoded (decoded), the picture number stored in the memory, and the picture number to be deleted will be described with reference to FIG.

  FIG. 10B is a relationship diagram showing the relationship between the picture number to be decoded (frame number), the saved picture number (frame number), and the deleted picture number (frame number). Here, the maximum number of pictures that can be stored in the memory is 5. Pictures are stored in the memory according to the transmission order.

  For example, when a P picture with a picture number of 4 is decoded, the saved picture number of the P picture with a picture number of 4 is 3, so that the pictures with the saved picture numbers of 0, 1, and 2 are saved in the memory. Will be. When the B picture with the picture number 3 to be decoded is decoded, the pictures with the picture numbers 4, 1, 2, and 0 are stored as shown in FIG. Here, as shown in FIG. 10A, the B picture with the picture number 1 is not referenced by any picture after the picture with the picture number 3 is decoded. Is deleted when is decoded.

  Similarly, when a B picture with a picture number of 5 to be decoded is decoded, pictures with a picture number of 6, 3, 4, 2, 0 are stored as shown in FIG. Here, since the B picture with the picture number 3 is not referred to by any picture after the picture with the picture number 5 is decoded, it is deleted when the B picture with the picture number 5 is decoded. .

  Further, when a P picture with a picture number 8 to be decoded is decoded, pictures with picture numbers 5, 6, 4, 2, 0 are stored as shown in FIG. Here, since only a maximum of 5 frames can be stored in the memory, in order to refer to the P picture with the picture number 8 later, any picture with the picture number 5, 6, 4, 2, 0 is deleted and the picture number is deleted. Memory for storing 8 P pictures must be reserved. Therefore, as a reference for selecting a frame to be deleted in FIG. 10B, in decoding of a P picture, that is, decoding of an even-numbered picture number, an oldest picture in time, that is, an I picture having a picture number of 0 in this case is selected. When the P picture with picture number 8 is decoded, it is deleted.

  Similarly, when a B picture with a picture number 7 to be decoded is decoded, pictures with picture numbers 8, 5, 6, 4, 2 are stored as shown in FIG. 10B. Here, since the B picture with picture number 5 is not referred to by any picture after the B picture with picture number 7 is decoded, it is deleted when the B picture with picture number 7 is decoded. .

  Furthermore, when a P picture with a picture number 10 to be decoded is decoded, pictures with picture numbers 7, 8, 6, 4, 2 are stored as shown in FIG. Here, since only a maximum of 5 frames can be stored in the memory, in order to refer to the P picture with the picture number 10 later, one of the pictures with the picture numbers 7, 8, 6, 4, and 2 is deleted. A memory for storing the picture number 10 must be secured. Therefore, as a reference for selecting a frame to be deleted in FIG. 10B, in decoding of a P picture, that is, decoding of a picture having an even-numbered picture number, the picture with the oldest picture number 2 in time is assigned the picture number 10 When the P picture is decoded, it is deleted.

  When a picture is deleted in this way, a command of memory management information for deleting the picture is encoded and attached to the encoded signal of the picture to be decoded and transmitted.

  In the example shown in FIG. 10B, an example in which an unnecessary image (picture) exists in the memory and a command of memory management information for removing the image is sent once has been described. As described above, if the memory management information command to be removed is sent only once, the memory management information command sent along with the B picture may not be executed. This is because a B picture is unlikely to be used as an image that is referred to in encoding / decoding of a P picture, and therefore, when a sufficient storage capacity or transmission capacity cannot be secured, the B picture data is discarded with priority. This is because there is a possibility that the command of the memory management information sent accompanying the B picture cannot be executed.

  In order to solve this problem, an example in which a command of memory management information for removing an image is repeatedly encoded and transmitted will be described. Hereinafter, FIG. 10C will be specifically described.

  FIG. 10C is a relationship diagram showing another relationship between a picture number to be decoded (frame number), a stored picture number (frame number), and a deleted picture number (frame number). FIG. 10C shows that the command for deleting the picture with the picture number to be deleted is attached to the encoded signal of the picture with the picture number to be decoded.

  As shown in FIG. 10C, when the B picture with the picture number 3 is decoded, the pictures with the picture numbers 4, 1, 2, and 0 are stored. Here, as shown in FIG. 10A, the B picture with the picture number 1 is not referred to by any picture after the picture with the picture number 3 is decoded. Therefore, when the picture with the picture number 3 is decoded, the B picture with the picture number 1 is deleted, and a command of memory management information for deletion is attached to the picture with the picture number 3.

  However, since the picture with picture number 3 is a B picture, it is less important in terms of image reproduction than the I picture and P picture as described above, and data is easily discarded at the time of transmission. There is a possibility that the command of the memory management information sent along with the B picture cannot be executed (when a frame is stored as shown in FIG. 21).

  Therefore, it is shown that the picture with the picture number 1 attached to the picture number 3 is deleted from the P picture with the picture number 6, which is more important in terms of image reproduction than the B picture with the picture number 3 to be decoded next. A command of memory management information is attached (see FIG. 10C).

  Similarly, a memory management information command (indicating that a picture with picture number 3 is to be deleted) associated with a B picture with picture number 5 is attached to the P picture with picture number 8, and the P picture with picture number 10 is assigned a picture. A memory management information command (indicating that the picture with picture number 5 is to be deleted) is attached to the B picture with number 7. Since the picture with picture number 8 is a P picture, as shown in FIG. 10C, the B picture with picture number 7 is not accompanied by the command of the memory management information associated with the picture with picture number 8. However, it may be accompanied.

  As described above, as shown in FIG. 10C, the same memory management information command as the memory management information command first attached to the B picture is a picture that is more important in terms of image reproduction than the B picture. Yes, it is repeatedly attached to a picture stored or transmitted after a B picture to which a command of memory management information is first attached. As a result, even if the B picture accompanied with the memory management information command is lost first, the memory management information command can be executed normally.

  Note that, as described with reference to FIG. 10C, even when the memory management information command is attached to the B picture, and the repeated memory management information command is additionally attached to the P picture, the set importance is set. Is used. Note that the importance setting is not limited to that shown in the present embodiment.

  In this embodiment, it is determined whether or not to send each image according to the importance of each image, and the importance in each image is determined according to the memory management information shown in the above embodiment. The encoding is not necessarily accompanied. Therefore, the decoding process of the data encoded in the present embodiment is not different from the conventional method.

(Embodiment 8)
Next, an eighth embodiment will be described.

  A feature of this embodiment is that all images (pictures) in the memory are deleted and a command for initializing the memory area (in memory management information) is encoded and transmitted a plurality of times.

The memory management information shown in the above embodiments is given as code information as shown in FIG.

  FIG. 11 is a correspondence diagram showing commands of the memory management information, and shows a code number (Code), command contents (command), and additional information (additional information).

  For example, a command for releasing an unnecessary memory area of the short-term storage memory (short-term storage memory release) is given as code information Code1, and a picture number (frame number) to be released is added as additional information.

  The code information is given as header information of each frame shown in FIG.

  FIG. 13 is a schematic diagram showing the relationship between header information and frame data in the encoded signal of each picture. In FIG. 13, each encoded signal indicates an encoded signal of frames Frm12, Frm11, and Frm14 described later. Each encoded signal includes a frame header having header information and frame data related to image encoding. For example, the encoded signal of the frame Frm12 includes a frame header Frm12Hdr and frame data including each data MB12a, MB12b, MB12c, MB12d, and the like.

  Details of this encoded signal are shown in the schematic diagram of FIG.

  FIG. 14 is a schematic diagram showing a command of memory management information in the header information of the encoded signal.

  As shown in FIG. 14, the encoded signal of the frame FrmA includes a frame header FrmAHdr having header information, and frame data composed of data MBa, MBb, MBc, Mbd, and the like. The additional information AddA of the code information CodeA follows the code information CodeA of the command, the code information CodeB of the command to be executed next to the command of the code information CodeA, and the additional information AddB of the code information CodeB are the frame header FrmAHdr. To be added. If there is no additional information, only code information is added as in code information CodeC.

  Next, FIG. 12 shows a command execution procedure.

  FIG. 12 is a flowchart showing a command execution procedure.

  In FIG. 12, first, a command is acquired (Step C0), and it is determined whether or not the acquisition of the command is completed (Step C1). If the command acquisition is not completed and the command is acquired (No in Step C1), the acquired command is executed (Step C2), and the process returns to Step C0 to repeat this operation. On the other hand, if the acquisition of the command has been completed and the command has not been acquired (Yes in Step C1), the command execution process ends. This procedure is performed for each frame. Even when command information is sent in units of slices composed of a plurality of macroblocks, the command is executed in the same procedure as above.

  In the first embodiment, the memory management information command for removing an unnecessary image (releasing memory) has been described. Furthermore, in the first embodiment, by repeatedly encoding a memory management information command for removing unnecessary images, even if one memory management information command disappears due to a transmission path error, the other memory management information It was shown that the management information of the image stored in the memory can be restored from this command, and the possibility that the image can be restored correctly is increased.

  Here, consider the initialization command Code5 that removes all the information in the memory from the code information shown in FIG.

  If the initialization command Code5 disappears due to a transmission path error when the initialization command Code5 is sent only once, the processing such as memory management performed after the initialization is affected. Therefore, as in the first embodiment, a case where the initialization command Code5 is repeatedly encoded and transmitted will be described with reference to FIG.

  FIG. 15 shows a number (picture (frame) number) assigned to each frame, a number when each frame is stored in the memory (stored picture (frame) number), and an order in which each frame is transmitted. It is explanatory drawing which shows the relationship of a number (transmission order).

  Hereinafter, FIG. 15 will be specifically described. First, since the I picture with picture number 0 does not refer to other pictures, it is saved in the memory, and the saved picture number becomes 0. Next, since the P picture with the picture number 2 referring to the I picture with the picture number 0 is saved in the memory, the saved picture number with respect to the P picture with the picture number 2 becomes 1. Then, since the B picture with the picture number 1 referring to the I picture with the picture number 0 and the P picture with the picture number 2 is saved in the memory, the saved picture number of the B picture with the picture number 1 becomes 2. The order in which each picture is transmitted is the order stored in the memory. The relationship between the picture number, the stored picture number, and the transmission order is determined by the same procedure.

  Assume that the initialization command Code5 shown in FIG. 11 is sent when the I picture with the picture number 12 shown in FIG. 15 is encoded. Since the saved picture number of the I picture with the picture number 12 is 11, all the pictures with the saved picture number of 10 or less can be removed from the memory by this initialization command Code5.

  Here, a method of encoding the initialization command Code5 will be described with reference to FIG.

  FIG. 16 is a flowchart showing a method of encoding the initialization command code5, and shows the operation performed by the image encoding device 100 shown in FIG.

  First, the input image is encoded (Step A0). It is investigated (initialization investigation) whether all pictures that can be referred to in the memory are unnecessary after coding (whether no picture is referred to in future encoding) (Step A1), and the pictures stored in the memory are It is determined whether it is better to initialize without reference in the future (Step A2).

  If it is better to initialize (Yes in Step A2), an initialization command Code5 for initializing the memory area is encoded as memory management information (Step A3), initialization is performed (Step A4), and the process is terminated. On the other hand, when the initialization is not necessary (No in Step A2), Step A3 and Step A4 are not performed, and the process ends.

  Next, a method for decoding the encoded initialization command Code5 will be described with reference to FIG.

  FIG. 17 is a flowchart showing a method of decoding the encoded initialization command Code5, and shows operations performed by the image decoding apparatus 200 shown in FIG.

  First, the memory management information is decoded (Step A10), and the image signal is decoded from the encoded signal (Step A11). Next, it is determined whether or not there is an initialization command Code5 in the decoded memory management information (Step A12). If there is the initialization command Code5 (Yes in Step A12), all the pictures stored in the memory are removed. Initialization is performed (Step A13), and the process ends. However, at this time, the decoded image (in Step A11) is not removed.

  On the other hand, if there is no initialization command Code5 in the memory management information (No in Step A12), the process is terminated.

  Hereinafter, a method for initializing the memory will be specifically described with reference to FIG. Assume that the same initialization command Code5 as the initialization command Code5 given to the I picture with picture number 12 is given to the B picture with picture number 11 shown in FIG.

  As shown in FIG. 13, the initialization command Code5 is assigned to the frame header Frm12Hdr of the frame Frm12 (picture number 12) and the frame header Frm11Hdr of the frame Frm11 (picture number 11). Since the initialization command Code5 does not have additional information as shown in FIG. 11, all pictures stored in the memory at the time of decoding are removed.

  Therefore, the initialization command Code5 assigned to the I picture with picture number 12 (stored picture number 11) disappears due to a transmission path error, and the initialization command Code5 assigned to the B picture with picture number 11 (stored picture number 12) is executed. Then, all the pictures stored in the memory with pictures decoded before the stored picture number 11 are removed. In other words, even the I picture of picture number 12 (stored picture number 11) that should not be removed is removed.

  Thus, when the same initialization command Code5 as the initialization command Code5 assigned to the I picture with the picture number 12 is assigned to the B picture with the picture number 11, one picture (I picture with the picture number 12) is provided. Will be missing. On the other hand, an initialization command Code5 that is the same as the initialization command Code5 assigned to the I picture with picture number 12 (saved picture number 11) is assigned to the P picture with picture number 14 (saved picture number 13). When the initialization command Code5 assigned to the I picture disappears due to a transmission path error and the initialization command Code5 assigned to the P picture with picture number 14 is executed, two pictures (B picture with picture number 11 and picture number 11) 12 I pictures) will be lost.

  Note that the same problem as described above also occurs when the initialization command Code5 is repeatedly encoded, and the initialization command Code5 sent first and the initialization command Code5 sent subsequently are executed without any transmission path error. This is because it is initialized by the initialization command Code5 sent first, and is initialized again by the initialization command Code5 sent subsequently.

  A method for solving such a problem in memory initialization will be described.

  FIG. 18 shows commands for memory management information used to solve a problem in memory initialization.

  The difference from FIG. 11 is that an initialization retransmission command Code6 is newly added in FIG. The initialization retransmission command Code6 has an initialization picture (frame) number (number of a frame accompanied by the initialization command Code5 for initializing the memory area) as additional information.

  Hereinafter, the flow of image encoding processing using the initialization retransmission command Code 6 will be described with reference to FIG.

  FIG. 19 is a flowchart showing an image encoding method using the initialization retransmission command Code6, and shows operations performed by the image encoding apparatus 100 shown in FIG. 19, the same operations as those in FIG. 16 are denoted by the same reference numerals.

  First, the input image is encoded (Step A0). It is investigated (initialization investigation) whether all pictures in the memory are unnecessary after the encoding (whether no image is referred to in future encoding) (Step A1). The memory information control unit 101 determines whether initialization is necessary (Step A2). If initialization is necessary (Yes in Step A2), the management information encoding unit 105 performs initialization to initialize the memory area. Command Code5 is encoded as memory management information (Step A3), and initialization is performed (Step A4). When initialization is not required (No in Step A2), the operations of Step A3 and Step A4 are not performed.

  Next, the memory information control unit 101 encodes, as memory management information, an initialization command Code5 that initializes the memory area in association with the encoded signal of the image encoded immediately before (the image prior to the encoding target). (Step A30), and if encoded (Yes in Step A30), the management information encoding unit 105 encodes an initialization retransmission command Code6 for initializing the memory area as memory management information. (Step A31), the process ends.

  Further, if the initialization command Code5 for initializing the memory area accompanying the encoded signal of the image encoded immediately before (the image prior to the encoding target) is not encoded as memory management information (No in Step A30) ), The process is terminated.

  In the method shown in FIG. 19, when the initialization command Code5 for initializing the memory area is encoded along with the encoded signal of the image encoded immediately before, the initialization retransmission command Code6 is encoded again. However, if the initialization command Code5 that initializes the memory area is encoded in association with the encoding of the image that was encoded several images before rather than immediately before, the initialization retransmission command Code6 is encoded again. The initialization retransmission command Code6 for initializing the memory area may be repeatedly encoded as the memory management information along with a plurality of images.

  Specifically, as shown in FIG. 15, when the initialization command Code5 is encoded along with the encoding of the I picture with the picture number 12, the initialization retransmission command accompanying the encoding of the B picture with the picture number 11 Code 6 may be encoded, or the initialization retransmission command Code 6 may be encoded accompanying the encoding of the P picture with the picture number 14.

  In the former case, as shown in FIG. 13, the initialization command Code5 is assigned to the frame header Frm12Hdr of the frame Frm12, and further, the initialization retransmission command Code6 is assigned to the frame header Frm11Hdr of the frame Frm11. The initialization command Code5 is assigned to the frame header Frm12Hdr of Frm12, and the initialization retransmission command Code6 is assigned to the frame header Frm14Hdr of the frame Frm14.

  Furthermore, the initialization retransmission command Code6 is encoded along with the encoding of the B picture with the picture number 11, and the initialization retransmission command Code6 is encoded along with the encoding of the P picture with the picture number 14. You may do it. In this case, as shown in FIG. 13, the initialization command Code5 is assigned to the frame header Frm12Hdr of the frame Frm12, and the initialization retransmission command Code6 is assigned to the frame header Frm11Hdr of the frame Frm11 and the frame header Frm14Hdr of the frame Frm14. .

  Next, processing for decoding data encoded with the initialization retransmission command Code 6 will be described with reference to FIG. FIG. 20 is a flowchart showing a method of decoding the encoded initialization retransmission command Code 6, and shows the operation of the image decoding apparatus 200 shown in FIG. In FIG. 20, the same operations as those in FIG. 17 are denoted by the same reference numerals.

  First, the management information decoding unit 205 decodes the memory management information (Step A10). Then, the image signal is decoded from the encoded signal (Step A11).

  It is determined whether there is an initialization command Code5 in the decoded memory management information (Step A12). If there is an initialization command Code5 (Yes in Step A12), all the pictures in the memory are removed and initialized (Step A13). If there is no enable command Code5 (No in Step A12), initialization is not performed.

  Next, the memory information control unit 101 determines whether there is an initialization retransmission command Code6 in the memory management information (Step A40). If there is no initialization retransmission command Code6 (No in Step A40), the process is terminated. If there is an initialization retransmission command Code6 (Yes in Step A40), it is investigated whether initialization has been completed (Step A41). If initialization has been completed (Yes in Step A41), the process ends. If initialization has not been completed (No in Step A41), an initialization frame (memory area is initialized based on the additional information of the initialization retransmission command Code6). The previous storage frame (the frame stored in the reference image memory at the time of encoding the initialization frame) is deleted, and the long-time storage memory size is set to 0 ( Step A42) The processing is terminated. When the long-time storage frame is not used, it is not necessary to set the long-time storage memory size to zero.

  Therefore, when the initialization command Code5 is encoded with the picture number 12 shown in FIG. 15 and the initialization retransmission command Code6 is encoded with the picture number 14, the initialization command Code5 is not lost due to a transmission path error. When the initialization command Code5 is lost due to a transmission path error by the initialization command Code5, all the pictures stored in the memory with the stored picture number of 10 or less are deleted by the initialization retransmission command Code6. Become.

  As described above, when the initialization command code5 is repeatedly encoded and transmitted, the initialization retransmission command Code6 to which the initialization picture number as additional information is added is encoded and transmitted after the second time. Based on the information, the storage frame before the initialization frame (the frame stored in the reference image memory at the time of encoding the initialization frame accompanied by the initialization command Code 5 first) is deleted. Therefore, the above-described problem that a necessary image (picture) is lost can be solved.

  Note that the initialization retransmission command Code6 described above is effective even in the case of assigning the stored picture number different from FIG. 15 as shown in FIG.

  This will be specifically described below.

  FIG. 21 shows a number (picture (frame) number) assigned to each frame, a number when each frame is stored in the memory (stored picture (frame) number), and an order in which each frame is transmitted. It is explanatory drawing which shows the other relationship in a number (transmission order).

  A method for assigning these numbers will be described. First, since the I picture with picture number 0 does not refer to other pictures, it is saved in the memory, and the saved picture number becomes 0. Next, since the P picture with the picture number 2 referring to the I picture with the picture number 0 is saved in the memory, the saved picture number with respect to the P picture with the picture number 2 becomes 1. Then, the B picture with the picture number 1 that refers to the I picture with the picture number 0 and the P picture with the picture number 2 is saved in the memory. However, since this B picture is not referred to by other pictures, it is saved. The picture number is 1 in the same way as the saved picture number of the P picture of picture number 2 saved immediately before. The order in which each picture is transmitted is the order stored in the memory. The relationship between the picture number, the stored picture number, and the transmission order is determined by the same procedure.

  As shown in FIG. 21, it is assumed that an initialization command Code5 shown in FIG. Since the saved picture number of the I picture with the picture number 12 is 6, all the pictures having the saved picture number of 5 or less can be removed from the memory by this initialization command Code5.

  Here, when the initialization command Code5 is repeatedly encoded, specifically, when the same initialization command Code5 as the initialization command Code5 assigned to the I picture with the picture number 12 is given to the P picture with the picture number 14 Will be described.

  Since the initialization command Code5 has no additional information as shown in FIG. 11, all pictures stored in the reference memory at the time of decoding are removed. Therefore, the initialization command Code5 assigned to the I picture with picture number 12 (stored picture number 6) disappears due to a transmission path error, and the initialization command Code5 assigned to the P picture with picture number 14 (stored picture number 7) is executed. Then, all the pictures stored in the memory among the pictures having the stored picture number 6 or less are removed. That is, up to an I picture of picture number 12 (stored picture number 6) that should not be removed is removed.

  However, when the initialization retransmission command Code6 is associated with the P picture with the picture number 14 instead of the initialization command Code5, the initialization command Code5 associated with the I picture with the picture number 12 does not disappear due to a transmission line error. When the initialization command Code5 is lost due to a transmission path error due to the initialization command Code5, the picture with the saved picture number of 5 or less is saved in the memory by the initialization retransmission command Code6 attached to the P picture with the picture number 14. All pictures will be deleted.

  That is, since an initialization frame (in this case, picture number 12) is added as additional information to the initialization retransmission command Code6, a storage frame before the initialization frame (the reference image at the time of storing the initialization frame) A saved frame having a saved picture number of 5 or less stored in the memory) is deleted.

  As described above, the initialization retransmission command Code6 having additional information increases the possibility that the initialization can be normally executed even when the initialization command code5 is lost due to a transmission path error. Note that the initialization retransmission command shown in the present embodiment, in which the additional information is the picture number accompanied by the initialization retransmission command, is substituted as the initialization command, and Code 5 and Code 6 shown in FIG. 18 are used as one command. You may make it implement | achieve. This is because when initialization retransmission is performed for retransmission of initialization information, the number of the frame accompanied by the initialization command is designated, so that a picture number for retransmitting the frame is not used. is there. At this time, the initialization command Code5 may be invalidated.

  In addition, when realizing the initialization retransmission command and the initialization command Code 5 shown in the above embodiment with one command in this way, a special value not used in the initialization retransmission command shown in the above embodiment is added. A command having the same function as the initialization command Code 5 to be sent first may be used with the initialization retransmission command included as information.

  Further, as described in the above embodiments, when memory management information such as a command for releasing a memory area that has become unnecessary or an initialization command is transmitted again, an image as shown in FIGS. 13 and 14 is displayed. The header information included in the memory management information may be transmitted separately from the frame data, instead of being included in the header information added to the frame data related to the encoding of the frame. That is, the above-described command to be retransmitted may be transmitted as a separate stream, for example, instead of being in the same stream as the encoded picture. Further, it may be recorded in another area of the storage medium.

  Furthermore, in this embodiment, when retransmitting the initialization command, the picture number (initialization frame number) of the picture to which the initialization command is first attached is added to the initialization retransmission command as additional information. Also when resending the command of the memory management information such as the command indicating the memory area to be released and the command specifying the target picture to be moved from the short-time storage memory to the long-time storage memory shown in each of the above-described embodiments. Naturally, the picture number of the picture to be encoded (information for specifying the picture) transmitted with the command first may be included and transmitted as a parameter. In this way, it is possible to detect which picture has been transmitted when a transmission error has occurred.

(Embodiment 9)
Next, an image encoding method and an image decoding method in Embodiment 9 will be described.

  The feature of this embodiment is that when the memory management information is transmitted a plurality of times, the timing of processing based on the memory management information transmitted after the second time is changed.

  When data obtained by repeatedly encoding the memory management information shown in the above embodiment is decoded, an image signal accompanied by the memory management information must be decoded before processing the memory management information repeatedly sent. It was. As a specific example, a case where a command for releasing an unnecessary memory area described in the second embodiment is transmitted a plurality of times will be described again with reference to FIG.

  It is assumed that the picture of the picture number 12 shown in FIG. 15 is encoded with the Code 1 command shown in FIG. 18 and that the picture of the picture number 11 is also attached with the Code 1 command. At this time, decoding is performed according to FIG.

  First, Code1 attached to the picture of picture number 12 is decoded (Step 110). Next, the picture with the picture number 12 is decoded (Step 111). Here, if Code1 that should originally be attached to the picture with picture number 12 is lost during transmission (No in Step 112), the processing relating to this frame ends.

  It is the picture with picture number 11 that is decoded next to the picture with picture number 12 in the transmission order.

  First, Code 1 encoded with the picture of picture number 11 is decoded (Step 110). Next, the picture of picture number 11 is decoded (Step 111). When Code 1 is transmitted without being lost during transmission, Code 1 which is a memory release command is present in the decoded memory management information (Yes in Step 112), and the next processing (Step 113) is reached.

  Here, since the memory is not released when the picture with the picture number 12 decoded before the picture with the picture number 11 is decoded (No in Step 113), the memory releasing process is performed (No in Step 113). Step 114).

  As shown in the above specific example, execution of a command to be executed for a picture (picture number 12) that was not originally executed due to transmission of a command for releasing an unnecessary memory area a plurality of times. However, after the decoding process of the image signal of the picture (picture number 11) sent later, the execution of the command is delayed.

  Therefore, in this embodiment, a method for solving the above problem will be described with reference to FIGS. 22, 23, and 24. FIG.

  FIG. 22 is a correspondence diagram showing the relationship between memory management information and commands used in this embodiment.

  In FIG. 22, Code indicates the command number, the command indicates the content of the command, the additional information indicates additional information added to the command, and the processing position indicates the timing of executing the command.

  FIG. 22 differs from FIG. 11 in that commands from CodeA1 to CodeA4 are executed after the image decoding process, while CodeA6 to CodeA9 corresponding to CodeA1 to CodeA4 are executed before the image decoding process. This is a command point.

  When the memory management information is repeatedly sent, the command of the memory management information to be encoded first is a command (CodeA1 to CodeA4) whose processing position is decoded (executed after decoding of the image), and repeatedly (2 A command to be encoded is defined as a command (Code A6 to Code A9) whose processing position is before decoding (executed before decoding of an image).

  Thereby, even when the memory management information sent first is lost, the command to be executed with the memory management information originally sent can be executed at an early stage, thereby making it difficult to cause problems such as delay.

  The processing procedure when using the command of FIG. 22 will be described below with reference to FIGS.

  FIG. 23 is a flowchart showing an image encoding method according to the present embodiment, and shows the operation of the image encoding apparatus 100 shown in FIG.

  In FIG. 23, first, an image is encoded (Step B0). After encoding, an unnecessary area (an image that will not be referred to in future encoding) is investigated in the memory (Step B1), and it is determined whether there is an unnecessary memory area (Step B2). If there is an unnecessary memory area (Yes in Step B2), the post-decoding memory management information is encoded (Step B3), assuming that a command to release the unnecessary memory area is executed after decoding of the image signal, and that is unnecessary. A new memory area is released (Step B4). On the other hand, when there is no unnecessary memory area (No in Step B2), the operations of Step B3 and Step B4 are not performed.

  Next, whether or not the memory information control unit 101 encodes, as memory management information, a command for releasing an unnecessary memory area accompanying the encoding of the image encoded immediately before (the image prior to the encoding target). Determine (Step B30). If not encoded (No in Step B30), the process ends. If encoded (Yes in Step B30), the management information encoding unit 105 transmits a command for releasing the unnecessary memory area to decode the image signal. The pre-decoding memory management information is encoded to be executed before (Step B31), and the process ends.

  In addition, when a command for releasing an unnecessary memory area is encoded along with the encoded signal of the image encoded immediately before in Step B30, the command is encoded again. The above command may be attached to an image several images before, and the above command may be repeatedly encoded as memory management information and transmitted along with a plurality of images.

  Next, a procedure for decoding data encoded according to the procedure of FIG. 23 will be described with reference to FIGS. 24 and 15.

  FIG. 24 is a flowchart showing an image decoding method according to the present embodiment, and shows operations performed by the image decoding apparatus 200 shown in FIG.

  In the following description, in FIG. 15, the picture number 12 is encoded with the code A1 command shown in FIG. 22, and the picture number 11 is encoded with the code A6 command. And In FIG. 13, CodeA1 is assigned to the frame header Frm12Hdr of the frame Frm12 of picture number 12, and CodeA6 is assigned to the frame header Frm11Hdr of the frame Frm11 of picture number 11.

  The image decoding apparatus receives a command for releasing the same image area in the memory a plurality of times unless the command is lost due to a transmission path error. Therefore, in the image decoding method performed by the image decoding apparatus, even when a command for reopening an image that has already been released is received, it must be determined that the image has been correctly received without being processed as an error.

  First, the decoding process for the picture with picture number 12 will be described.

  In FIG. 24, first, the memory management information of the picture with the picture number 12 is decoded (Step B5), and it is investigated whether this memory management information is the pre-decoding memory management information (Step B7). Here, since the memory management information (CodeA1) is the memory management information for after decoding (No in Step B7), the image signal having the picture number 12 is decoded. As described above, since the memory management information (CodeA1) is the post-decoding memory management information (Yes in Step B9), the memory is released (Step B11), and the processing related to the memory management information of the picture with the picture number 12 is completed. To do.

  On the other hand, when Code A1 of the memory management information is missing, it is not determined in Step B7 that it is memory management information for before decoding (No in Step B7), and also in Step B9 that it is memory management information for after decoding. The determination is not made (No in Step B9), only the image signal of picture number 12 is decoded (Step B6), and the process related to the memory management information of picture number 12 ends.

  Next, decoding processing relating to the frame of picture number 11 will be described with reference to FIG.

  First, the memory management information of the picture number 11 is decoded (Step B5), and it is investigated whether this memory management information is the memory management information for before decoding (Step B7). Here, since Code A6 is pre-decryption memory management information (Yes in Step B7), it is investigated whether the memory has been released (Step B8). In the process of picture number 12, if CodeA1 is executed, the memory has been released (step B8: Yes), so the memory release process (step B10) is not performed, and the image signal of picture number 11 is decoded (step B6). . Then, it is determined whether the memory management information is for after decoding (Step B9). Since Code A6 is the memory management information for before decoding (No in Step B9), the processing related to the memory management information of the picture with the picture number 11 is performed. Ends.

  However, if the memory management information of picture number 12 is lost due to packet loss during the transmission process and the memory is not released in the process related to picture number 12, the memory related to picture number 11 has not been released. It is determined (No in Step B8), and the memory is released in the next step (Step B10). After the memory is released, the image signal of picture number 11 is decoded (Step B6). Since CodeA6 is pre-decoding memory management information (No in Step B9), the processing related to the memory management information of the picture with the picture number 11 ends.

  As described above, by executing the command for the re-transmission before the decoding of the image signal, the delay in executing the command can be reduced even if the first transmitted command is lost.

  Although the case where the memory management information is CodeA1 and CodeA6 has been described as a specific example, the same processing can be realized even when CodeA2 and CodeA7 are used, and the same processing is possible even when CodeA3 and CodeA8 and CodeA4 and CodeA9 are used. It is feasible.

  Also, it is possible to use the initialization command CodeA5 shown in FIG. 22 as the post-decoding memory management information and the initialization retransmission command Code6 shown in FIG. 18 as the pre-decoding memory management information.

  In addition, when one post-decoding memory management information and a plurality of pre-decoding memory management information are provided as header information in one frame, the plurality of pre-decoding memory management information is the post-decoding memory management. It is better to process before information.

  That is, it is only necessary to add the pre-decoding memory management information to the head of the header information shown in FIG.

  In addition, the above-described implementation is performed by setting whether the memory management information is pre-decryption management information or post-decryption management information according to a combination of instructions shown in FIGS. 25 (a) and 25 (b). The command shown in the form may be realized.

  FIG. 25A is a correspondence diagram showing command contents and additional information. FIG. 25B is a correspondence diagram showing command execution timing (processing position).

  FIG. 26 is a schematic diagram illustrating a command of memory management information in the header information of the encoded signal.

  In FIG. 26, the encoded signal of the frame FrmB has a frame header FrmBHdr and frame data such as MBa and MBb. The frame header FrmBHdr has code information CodeD and the like as header information.

  At this time, for example, as shown in FIG. 26, the frame header FrmBHdr of the frame FrmB may be the code information CodeD of the command from the top, FlagD indicating the processing position, and additional information AddD indicating the additional information of the command. When there is no additional information, as shown in FIG. 26, CodeE of the command and FlagE indicating the processing position may be added to the frame header FrmBHdr. The processing of Step B7 and Step B9 shown in FIG. 24 can be optimized by setting the flag indicating the processing position instead of “Add” indicating additional information immediately after the code indicating the command.

  In addition, in order to distinguish whether the execution timing of the command is before or after decoding of the image signal, a new command indicating the processing position of the command is used, and the position in the frame header of the command indicating the processing position is used. The command positioned before the decoding may be executed after decoding, and the command positioned after the position in the frame header of the command indicating the processing position may be executed before decoding. As a result, when there are a plurality of commands, the execution timing (processing position) of each command can be indicated by one command, and the information to be sent is reduced compared to sending a flag indicating the processing position for each command. Encoding efficiency is improved.

  A specific example will be described with reference to FIG.

  FIG. 27 is a schematic diagram illustrating a command of memory management information in the header information of another encoded signal.

  In FIG. 27, the encoded signal of the frame FrmC includes a frame header FrmCHdr and frame data such as MBa and MBb. The frame header FrmCHdr includes, as header information, a command CodeF, a command dif, and a command in order from the front. CodeG, additional information AddG, and command CodeH are located.

  Then, it is determined whether or not the command dif indicating the processing position is present in the frame header FrmCHdr. The command CodeF preceding the command dif indicating the processing position is executed after decoding the frame FrmC, and the command CodeG or the command after the command dif is executed. CodeH may be executed before decoding the frame FrmC. In this case, if there is no command dif indicating the processing position, all commands in the frame header FrmCHdr are executed after the decoding process of the frame FrmC.

  As described in the above embodiments, when memory management information such as a command for releasing a memory area that has become unnecessary or an initialization command is transmitted again, the header added to the encoded signal of the image Instead of being included in the information and transmitted, the header information including the memory management information may be transmitted separately from the encoded signal of the image. That is, the above-described command to be retransmitted may be transmitted as a separate stream, for example, instead of being in the same stream as the encoded picture. Further, it may be recorded in another area of the storage medium.

(Embodiment 10)
Next, an embodiment 10 of the invention will be described.

  The present embodiment is different from the above embodiments in the unit of encoding. That is, when a command for releasing an unnecessary memory area is transmitted a plurality of times in the first embodiment, the memory management information stream CtlStr and the image encoded stream VideoStr shown in FIG. 1 corresponding to the command are an image (picture). In this embodiment, one frame may be encoded in units of slices as in the stream structure shown in FIG.

  Encoding in units of slices means that slice 1 of frame 1 in FIG. 28 has header 1-1, ctlStr1, and VideoStr1-1, and slice 2 of frame 1 has header 1-2, ctlStr1, and VideoStr1-2. In addition, a header, a memory management information stream CtlStr, and an image encoded stream VideoStr are encoded for each slice included in each frame. After encoding by the image encoding device, the image encoding device outputs a data stream. Note that a slice is a unit for returning from synchronization, is a band-shaped region composed of one or a plurality of blocks, and a picture is composed of a plurality of slices. A picture is a basic encoding unit corresponding to one image, and a block is a basic unit of encoding / decoding.

  Further, as described above, the contents when the memory management information stream CtlStr is transmitted a plurality of times are the same information in the same frame. By using the same information, addition of the memory management information stream CtlStr in units of slices can be omitted. For example, information indicating whether multiple transmissions are omitted in the slice is added to the header of the slice, and “0” is transmitted when transmission of the command multiple times is omitted in the slice. Is transmitted in the slice (when not omitted), “1” is added. Specifically, an example is shown in FIG. The header and the image encoded stream VideoStr in slice 1 to slice 3 in frame 1 are different from each other. On the other hand, slice 1 and slice 2 have the same memory management information stream CtlStr1, and information “1” indicating that the same memory management information stream CtlStr1 is encoded in a plurality of slices in the same frame is slice 1. And each slice 2 has. Further, the slice 3 has information “0” indicating that the memory management information stream CtlStr1 is omitted. As a result, when multiple transmissions are omitted in the current slice, the memory management information stream CtlStr is referred to by referring to the memory management information stream CtlStr in the slice indicated by “1”, such as the first slice. Can be omitted, and the number of bits can be reduced.

  That is, the information “0” indicating that the above-described memory management information stream CtlStr1 is omitted is a slice (slice 3) that does not have information specifying the picture to be removed, and designates the picture to be removed. This is information indicating that the information specifying the picture to be removed is referred to when the information to be referred to is referred to.

  Such a method of making it possible to omit the addition of the memory management information stream CtlStr is effective because it is unlikely that the memory management information stream CtlStr is lost multiple times during the transmission process.

  Furthermore, when the presence or absence of the memory management information stream CtlStr can be determined without information indicating that the memory management information stream CtlStr is omitted, this may be omitted as shown in FIG. For example, when the head of the memory management information stream CtlStr can be distinguished from the head of the image encoded stream VideoStr, whether there is information indicating whether the memory management information stream CtlStr1 is encoded as shown in FIG. The determination can be made based on whether or not there is predetermined information at a predetermined position from the head in each slice.

  Such a method of making it possible to omit the addition of the memory management information stream CtlStr is effective in reducing the number of times of encoding the memory management information stream CtlStr and reducing the number of bits.

  The encoding has been described above. Similarly, one frame can be decoded in units of slices. When transmitting a command for releasing an unnecessary memory area a plurality of times in the second embodiment, the image decoding apparatus 200 shown in FIG. 3 uses the management information stream CtlStr and the image code shown in FIG. 28 corresponding to the command. The stream structure having the encrypted stream VideoStr is separated and each is input in units of images (pictures), but each may be input in units of slices.

  In addition, in encoding / decoding in other embodiments, one frame may be similarly encoded / decoded in units of slices.

  The encoding method and decoding method shown in the first to tenth embodiments are applied to mobile communication devices such as mobile phones and car navigation systems, and photographing devices such as digital video cameras and digital still cameras. It can be mounted by a semiconductor such as an LSI. In addition to the transmission / reception type terminal having both an encoder and a decoder, there are three possible mounting formats: a transmitting terminal having only an encoder and a receiving terminal having only a decoder.

(Embodiment 11)
Next, an eleventh embodiment of the present invention will be described.

  In the present embodiment, a program for realizing the configuration of the image encoding method or the image decoding method shown in the first to tenth embodiments is further recorded on a storage medium such as a flexible disk. As a result, the processing described in the above embodiment can be easily performed in an independent computer system.

  FIG. 30 is an explanatory diagram in the case of implementing by a computer system using a flexible disk storing the image encoding method or the image decoding method of the first embodiment.

  FIG. 30B shows an appearance, a cross-sectional structure, and a flexible disk as seen from the front of the flexible disk, and FIG. 30A shows an example of a physical format of the flexible disk that is a recording medium body. The flexible disk FD1 is built in the case F, and on the surface of the disk, a plurality of tracks Tr are formed concentrically from the outer periphery toward the inner periphery, and each track is divided into 16 sectors Se in the angular direction. ing. Therefore, in the flexible disk storing the program, an image encoding method as the program is recorded in an area allocated on the flexible disk FD1.

  FIG. 30C shows a configuration for recording and reproducing the program on the flexible disk FD1. When the program is recorded on the flexible disk FD1, the image encoding method or the image decoding method as the program is written from the computer system Cs via the flexible disk drive FDD. When the image encoding method is built in the computer system by the program in the flexible disk FD1, the program is read from the flexible disk FD1 by the flexible disk drive FDD and transferred to the computer system Cs.

  In the above description, a flexible disk is used as the recording medium, but the same can be done using an optical disk. Further, the recording medium is not limited to this, and any recording medium such as an IC card or a ROM cassette capable of recording a program can be similarly implemented.

  In addition, the image encoding method and the image decoding method shown in the above embodiment are applied to a mobile communication device such as a mobile phone or a car navigation system, a photographing device such as a digital video camera or a digital still camera, and a semiconductor such as an LSI. It is possible to implement. In addition to the transmission / reception type terminal having both an encoder and a decoder, there are three possible mounting formats: a transmitting terminal having only an encoder and a receiving terminal having only a decoder.

  Here, application examples of the image coding method and the image decoding method shown in the first to tenth embodiments and a system using the same will be described.

  FIG. 31 is a block diagram showing an overall configuration of a content supply system ex100 that implements a content distribution service. The communication service providing area is divided into desired sizes, and base stations ex107 to ex110, which are fixed wireless stations, are installed in each cell.

  The content supply system ex100 includes, for example, a computer ex111, a personal digital assistant (PDA) ex112, a camera ex113, a mobile phone ex114, a camera via the Internet ex101, the Internet service provider ex102, the telephone network ex104, and the base stations ex107 to ex110. Each device such as the attached mobile phone ex115 is connected.

  However, the content supply system ex100 is not limited to the combination as shown in FIG. 31, and may be connected in any combination. Also, each device may be directly connected to the telephone network ex104 without going through the base stations ex107 to ex110 which are fixed wireless stations.

The camera ex113 is a device capable of shooting a moving image such as a digital video camera. In addition, the mobile phone includes a PDC (Personal Digital Communications) system, a CDMA (Code Division Multiple Access) system, and a W-CDMA (Wideband-Code Division Multiple).
The mobile phone may be an Access (Access) method, a GSM (Global System for Mobile Communications) method, or a PHS (Personal Handyphone System).

  In addition, the streaming server ex103 is connected from the camera ex113 through the base station ex109 and the telephone network ex104, and live distribution or the like based on the encoded data transmitted by the user using the camera ex113 becomes possible. The encoded processing of the captured data may be performed by the camera ex113, or may be performed by a server or the like that performs data transmission processing. The moving image data shot by the camera ex116 may be transmitted to the streaming server ex103 via the computer ex111. The camera ex116 is a device such as a digital camera that can shoot still images and moving images. In this case, the moving image data may be encoded by the camera ex116 or the computer ex111. The encoding process is performed in the LSI ex117 included in the computer ex111 and the camera ex116. Note that image encoding / decoding software may be incorporated into any storage medium (CD-ROM, flexible disk, hard disk, or the like) that is a recording medium readable by the computer ex111 or the like. Furthermore, you may transmit moving image data with the mobile phone ex115 with a camera. The moving image data at this time is data encoded by the LSI included in the mobile phone ex115.

  In this content supply system ex100, the content (for example, video shot of music live) taken by the user with the camera ex113, camera ex116, etc. is encoded and transmitted to the streaming server ex103 as in the above embodiment. On the other hand, the streaming server ex103 stream-distributes the content data to the requested client. Examples of the client include a computer ex111, a PDA ex112, a camera ex113, a mobile phone ex114, and the like that can decode the encoded data. In this way, the content supply system ex100 can receive and play back the encoded data at the client, and also realize personal broadcasting by receiving, decoding and playing back at the client in real time. It is a system that becomes possible.

  The image encoding method or the image decoding method described in the above embodiments may be used for encoding and decoding of each device constituting this system.

  A mobile phone will be described as an example.

  FIG. 32 is a diagram illustrating the mobile phone ex115 using the image coding method and the image decoding method described in the above embodiment. The mobile phone ex115 includes an antenna ex201 for transmitting and receiving radio waves to and from the base station ex110, a video from a CCD camera, a camera unit ex203 capable of taking a still image, a video shot by the camera unit ex203, and an antenna ex201. A display unit ex202 such as a liquid crystal display that displays data obtained by decoding received video and the like, a main body unit composed of a group of operation keys ex204, an audio output unit ex208 such as a speaker for audio output, and audio input To store encoded data or decoded data such as a voice input unit ex205 such as a microphone, captured video or still image data, received mail data, video data or still image data, etc. Recording media ex207 and mobile phone ex115 with recording media ex207 And a slot portion ex206 to ability. The recording medium ex207 stores a flash memory element which is a kind of EEPROM (Electrically Erasable and Programmable Read Only Memory) which is a non-volatile memory that can be electrically rewritten and erased in a plastic case such as an SD card.

  Further, the cellular phone ex115 will be described with reference to FIG. The cellular phone ex115 controls the main control unit ex311 which is configured to control the respective units of the main body unit including the display unit ex202 and the operation key ex204. The power supply circuit unit ex310, the operation input control unit ex304, and the image encoding Unit ex312, camera interface unit ex303, LCD (Liquid Crystal Display) control unit ex302, image decoding unit ex309, demultiplexing unit ex308, recording / reproducing unit ex307, modulation / demodulation circuit unit ex306, and audio processing unit ex305 via a synchronization bus ex313 Are connected to each other.

  When the end of call and the power key are turned on by a user operation, the power supply circuit ex310 starts up the camera-equipped digital mobile phone ex115 in an operable state by supplying power from the battery pack to each unit. .

  The mobile phone ex115 converts the voice signal collected by the voice input unit ex205 in the voice call mode into digital voice data by the voice processing unit ex305 based on the control of the main control unit ex311 including a CPU, a ROM, a RAM, and the like. The modulation / demodulation circuit unit ex306 performs spread spectrum processing, the transmission / reception circuit unit ex301 performs digital-analog conversion processing and frequency conversion processing, and then transmits the result via the antenna ex201. In addition, the cellular phone ex115 amplifies the received data received by the antenna ex201 in the voice call mode, performs frequency conversion processing and analog-digital conversion processing, performs spectrum despreading processing by the modulation / demodulation circuit unit ex306, and analog audio by the voice processing unit ex305. After the data is converted, it is output via the audio output unit ex208.

  Further, when an e-mail is transmitted in the data communication mode, text data of the e-mail input by operating the operation key ex204 of the main body is sent to the main control unit ex311 via the operation input control unit ex304. The main control unit ex311 performs spread spectrum processing on the text data in the modulation / demodulation circuit unit ex306, performs digital analog conversion processing and frequency conversion processing in the transmission / reception circuit unit ex301, and then transmits the text data to the base station ex110 via the antenna ex201.

  When transmitting image data in the data communication mode, the image data captured by the camera unit ex203 is supplied to the image encoding unit ex312 via the camera interface unit ex303. When image data is not transmitted, the image data captured by the camera unit ex203 can be directly displayed on the display unit ex202 via the camera interface unit ex303 and the LCD control unit ex302.

  The image encoding unit ex312 has a configuration including the image encoding device described in the present invention, and an encoding method using the image data supplied from the camera unit ex203 in the image encoding device described in the above embodiment. The image data is converted into encoded image data by compression encoding, and is sent to the demultiplexing unit ex308. At the same time, the cellular phone ex115 sends the sound collected by the audio input unit ex205 during imaging by the camera unit ex203 to the demultiplexing unit ex308 via the audio processing unit ex305 as digital audio data.

  The demultiplexing unit ex308 multiplexes the encoded image data supplied from the image encoding unit ex312 and the audio data supplied from the audio processing unit ex305 by a predetermined method, and the multiplexed data obtained as a result is a modulation / demodulation circuit unit Spread spectrum processing is performed in ex306, digital analog conversion processing and frequency conversion processing are performed in the transmission / reception circuit unit ex301, and then transmitted through the antenna ex201.

  When data of a moving image file linked to a home page or the like is received in the data communication mode, the received data received from the base station ex110 via the antenna ex201 is subjected to spectrum despreading processing by the modulation / demodulation circuit unit ex306, and the resulting multiplexing is obtained. Data is sent to the demultiplexing unit ex308.

  Also, in order to decode the multiplexed data received via the antenna ex201, the demultiplexing unit ex308 separates the multiplexed data into a bit stream of image data and a bit stream of audio data, and synchronizes them. The encoded image data is supplied to the image decoding unit ex309 via the bus ex313, and the audio data is supplied to the audio processing unit ex305.

  Next, the image decoding unit ex309 has a configuration including the image decoding apparatus described in the present invention, and decodes a bit stream of image data by a decoding method corresponding to the encoding method described in the above embodiment. Thus, the reproduction moving image data is generated and supplied to the display unit ex202 via the LCD control unit ex302, and thereby, for example, the moving image data included in the moving image file linked to the home page is displayed. At the same time, the audio processing unit ex305 converts the audio data into analog audio data, and then supplies the analog audio data to the audio output unit ex208. Thus, for example, the audio data included in the moving image file linked to the home page is reproduced. The

  Note that the present invention is not limited to the above-described system, and recently, digital broadcasting using satellites and terrestrial waves has become a hot topic, and as shown in FIG. Any of the decoding devices can be incorporated. Specifically, in the broadcasting station ex409, a bit stream of video information is transmitted to the communication or broadcasting satellite ex410 via radio waves. Receiving this, the broadcasting satellite ex410 transmits a radio wave for broadcasting, and receives the radio wave with a home antenna ex406 having a satellite broadcasting receiving facility, such as a television (receiver) ex401 or a set top box (STB) ex407. The device decodes the bitstream and reproduces it. In addition, the image decoding apparatus described in the above embodiment can also be implemented in a playback apparatus ex403 that reads and decodes a bitstream recorded on a storage medium ex402 such as a CD or DVD that is a recording medium. In this case, the reproduced video signal is displayed on the monitor ex404. Further, a configuration in which an image decoding apparatus is mounted in a set-top box ex407 connected to a cable ex405 for cable television or an antenna ex406 for satellite / terrestrial broadcasting, and this is reproduced on a monitor ex408 of the television is also conceivable. At this time, the image decoding apparatus may be incorporated in the television instead of the set top box. It is also possible to receive a signal from the satellite ex410 or the base station ex107 by the car ex412 having the antenna ex411 and reproduce the moving image on a display device such as the car navigation ex413 that the car ex412 has.

  Further, the image signal can be encoded by the image encoding device shown in the above embodiment and recorded on a recording medium. Specific examples include a recorder ex420 such as a DVD recorder that records image signals on a DVD disk ex421 and a disk recorder that records images on a hard disk. Further, it can be recorded on the SD card ex422. If the recorder ex420 includes the image decoding device described in the above embodiment, the image signal recorded on the DVD disc ex421 or the SD card ex422 can be reproduced and displayed on the monitor ex408.

  The configuration of the car navigation ex 413 is, for example, the configuration shown in FIG. 33 excluding the camera unit ex203, the camera interface unit ex303, and the image encoding unit ex312. The same applies to the computer ex111 and the television (receiver). ) Ex401 can also be considered.

  In addition to the transmission / reception type terminal having both the encoder and the decoder, the terminal such as the mobile phone ex114 has three mounting formats, that is, a transmitting terminal having only an encoder and a receiving terminal having only a decoder. Can be considered.

  As described above, the image encoding method or the image decoding method described in the above embodiment can be used in any of the above-described devices and systems, and thereby, the effects described in the above embodiment can be obtained. Can be obtained.

  The present invention is not limited to the above-described embodiment, and various changes or modifications can be made without departing from the scope of the present invention.

  Each functional block in the block diagrams (FIGS. 1, 3, 33, etc.) is typically realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them.

  For example, the functional blocks other than the memory may be integrated into one chip.

  The name used here is LSI, but it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

  Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied.

  In addition, among the functional blocks, only the means for storing the data to be encoded or decoded may be configured separately instead of being integrated into one chip.

  Although the embodiments of the present invention have been described with reference to the drawings, the present invention is not limited to the illustrated embodiments. Various modifications and variations can be made to the illustrated embodiment within the same range or equivalent range as the present invention.

  The present invention can be applied to a multimedia device including an image encoding device, and particularly applicable to a mobile phone, a digital camera, a video camera, a PDA, a television, a DVD recorder, a BD recorder, an STB, a personal computer, a car navigation device, and the like.

DESCRIPTION OF SYMBOLS 100 Image coding apparatus 101 Memory information control part 102 Short-time storage memory management part 103 Long-time storage memory management part 104 Non-storage memory management information part 105 Management information encoding part 106 Reference image selection part 107 Storage area designation part 108 Reference area Designating unit 109 Image memory 110 Image encoding unit 111 Image decoding unit 112 Variable length encoding unit 113 Counter 114 Counter 200 Image decoding device 201 Memory information control unit 202 Short-time storage memory management unit 203 Long-time storage memory management unit 205 Management information decoding unit 207 Storage area specifying unit 208 Reference area specifying unit 209 Image memory 210 Image decoding unit 212 Variable length decoding unit Cs Computer system FD Flexible disk FDD Flexible disk drive

Claims (4)

An image encoding method for encoding with reference to a reference picture selected from a plurality of reference pictures stored in a memory,
The encoding target picture is encoded with reference to the selected reference picture,
Encoding first memory management information for managing a reference picture stored in a memory in association with the encoded picture to be encoded;
The first memory management information is again encoded as second memory management information by attaching it to a picture different from the picture to be encoded,
The second memory management information is accompanied with information for specifying the encoding target picture,
The memory management information is information that designates a memory area that becomes unnecessary and is released in the memory.
An image encoding method characterized by the above.   The memory includes a short-time storage memory having a short reference picture storage time and a long-time storage memory having a reference picture storage time longer than the short-time storage memory,
  The memory management information further includes information specifying a reference picture to be moved from the short-time storage memory to the long-time storage memory.
  The image encoding method according to claim 1. An image encoding device for encoding with reference to a reference picture selected from a plurality of reference pictures stored in a memory,
First encoding means for encoding the encoding target picture with reference to the selected reference picture;
Second encoding means for encoding first memory management information for managing reference pictures stored in a memory in association with the encoded picture to be encoded;
The first memory management information is encoded again as second memory management information by being attached to a picture different from the encoding target picture, and the encoding target picture is added to the second memory management information. Management information encoding means for attaching information for identifying
The memory management information is information that designates a memory area that becomes unnecessary and is released in the memory.
An image encoding apparatus characterized by that. A recording medium on which a data stream encoded with reference to a reference picture selected from a plurality of reference pictures stored in a memory is recorded,
With reference to the selected reference picture, encoded data obtained by encoding the encoding target picture;
Management information encoded data in which first memory management information for managing reference pictures stored in a memory is encoded in association with the encoding target picture;
The first memory management information is encoded again as second memory management information by being attached to a picture different from the encoding target picture, and the encoding target picture is specified in the second memory management information. Management information re-encoded data accompanied by information to be
The memory management information is information that designates a memory area that becomes unnecessary and is released in the memory.
A recording medium characterized by the above.

Images