JP2018019195A - Moving image formation method and moving image formation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve a coding efficiency in coding processing by using filter processing.SOLUTION: In a moving image formation method and a moving image formation device according to the present disclosures, the moving image formation device generates a coded moving image by coding a coding target image constituting a moving image on the basis of a reference image. The reference image includes a long term reference image as a reference image of which a time distance from the coding target image is longer than those of other reference images. The moving image formation method is for determining whether filter processing is executed for at least one region of the coding target image, and using an image obtained by decoding an image obtained by coding an image having a size of a region where the filter processing is not excited as the coding target image of a predetermined value or more, as the long term reference image.SELECTED DRAWING: Figure 6

Description

  The present disclosure relates to a moving image generation method and a moving image generation apparatus.

  Patent Document 1 determines whether each area is a motion area or a static area based on the magnitude of the motion vector “MV” and the difference value “DIFF”, and determines a moving part such as a person part as a motion area. Then, the smoothing process is skipped, the coding process such as the DCT process and the quantization process is performed, and the non-moving part such as the background part is determined as the still area, and the smoothing filter process is performed. Encoding processing such as DCT processing and quantization processing is performed.

JP 2005-295215 A

  The present disclosure improves encoding efficiency in encoding processing performed using filter processing.

  The moving image generation method and the moving image generation apparatus according to the present disclosure generate an encoded moving image by encoding an encoding target image constituting a moving image based on a reference image. The reference image includes a long-term reference image that is a reference image having a longer temporal distance from the encoding target image than other reference images. The moving image generation method determines whether or not to perform a filtering process on at least a partial area of the encoding target image, and the area of the encoding target image that is not subjected to the filtering process is predetermined. An image obtained by decoding an image obtained by encoding an image equal to or greater than the value is used as the long term reference image.

  According to the moving image generation method and the moving image generation apparatus of the present disclosure, it is possible to improve the encoding efficiency in the encoding process performed using the filter process.

FIG. 1 is a conceptual diagram of the moving picture transmission system according to the first embodiment. FIG. 2 is a block diagram of the moving image generating apparatus according to the first embodiment. FIG. 3 is a block diagram of the video playback device in the first embodiment. FIG. 4 is a flowchart showing the moving image generation processing in the first embodiment. FIG. 5 is a diagram for explaining filter processing in the moving image generation processing according to the first embodiment. FIG. 6 is a diagram showing the concept of inter-frame prediction in the moving image generation process in the first embodiment.

  Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of already well-known matters and repeated descriptions for substantially the same configuration may be omitted. This is to avoid the following description from becoming unnecessarily redundant and to facilitate understanding by those skilled in the art.

  The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

(Embodiment 1)
Hereinafter, Embodiment 1 will be described with reference to FIGS.

[1-1. Constitution]
FIG. 1 is a conceptual diagram of the moving picture transmission system according to the first embodiment.

  The moving image transmission system 100 includes a moving image generating device 110 and a moving image reproducing device 120.

  The moving image generation device 110 generates a moving image. An example of the moving image generation device 110 is a camera that encodes a moving image shot by an input unit, or an editing computer that encodes an unencoded image recorded in a storage unit.

  The video playback device 120 plays back the video generated by the video generation device 110. An example of the moving image reproducing device 120 is a television set or a personal computer equipped with a processor having a decoding function.

  The moving image generated by the moving image generating device 110 is sent to the moving image reproducing device via the communication path 130.

  There is a limit to the transmission capacity of the communication path 130. In order not to exceed the transmission capacity of the communication path 130, it is necessary to reduce the data amount of the moving image generated by the moving image generating device 110. In the present embodiment, the moving image generating apparatus 110 improves encoding efficiency in moving image generation in order to reduce the amount of moving image data.

  FIG. 2 is a block diagram of the moving image generating apparatus according to the first embodiment.

  The moving image generating apparatus 110 includes a processor 201, a storage unit 202, an input unit 203, an output unit 204, a communication unit 205, and a bus 206.

  The processor 201 controls other elements of the moving image generating apparatus 110 via the bus 206. As an example, the processor 201 can be configured by using a general-purpose CPU (Central Processing Unit). Further, the processor 201 can execute a predetermined program. An encoded moving image can be generated by the processor 201 executing a predetermined program.

  The storage unit 202 acquires various information from other elements and holds the information temporarily or permanently. The storage unit 202 is a general term for so-called primary storage devices and secondary storage devices, and a plurality of storage units 202 may be physically arranged. For example, a DRAM (Direct Random Access Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive) is used for the configuration of the storage unit 202.

  The input unit 203 receives information from the outside. The external information received by the input unit 203 includes information related to input from the operator of the video generation device 110, information that configures a video before encoding, information that configures a video after encoding, and the like. As an example, the input unit 203 can be configured by using an existing input / output interface.

  The output unit 204 presents information to the outside. Information presented by the output unit includes information that configures a moving image before encoding, information that configures a moving image after encoding, and the like. As an example, the output unit 204 can be configured by using an existing input / output interface.

  The communication unit 205 communicates with an external device via the communication path 130. The apparatus with which the communication unit 205 communicates includes the moving image playback device 120. As an example, the communication unit 205 can be configured by using an existing communication interface.

  The above-described configuration of the moving image generating apparatus 110 is an example. A part of each component of the moving image generating device 110 may be integrated. A part of each component of the moving image generating apparatus 110 may be divided into a plurality of elements. Some of the components of the moving image generating apparatus 110 may be omitted. The moving image generating apparatus 110 can be configured by adding other elements.

  In the present embodiment, at least one processor may generate a moving image. That is, a plurality of moving image generation devices 110 may cooperate to generate a moving image.

  FIG. 3 is a block diagram of the video playback device in the first embodiment.

  The moving image playback device 120 includes a processor 301, a storage unit 302, an input unit 303, an output unit 304, a communication unit 305, and a bus 306.

  The processor 301 controls other elements of the video playback device 120 via the bus 306. As an example, the processor 301 can be configured by using a general-purpose CPU (Central Processing Unit). Further, the processor 301 can execute a predetermined program. The encoded video can be reproduced by the processor 301 executing a predetermined program.

  The storage unit 302 acquires various information from other elements and holds the information temporarily or permanently. The storage unit 302 is a generic name for so-called primary storage devices and secondary storage devices, and a plurality of storage units 302 may be physically arranged. For example, a DRAM (Direct Random Access Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive) is used as the configuration of the storage unit 302.

  The input unit 303 receives information from the outside. Information from the outside received by the input unit 303 includes information related to input from the operator of the video playback device 120. As an example, the input unit 303 can be configured by using an existing input / output interface.

  The output unit 304 presents information to the outside. The information presented by the output unit includes information constituting the encoded moving image, information notifying the result of tampering detection, and the like. As an example, the output unit 304 can be configured by using an existing input / output interface.

  The communication unit 305 communicates with an external device via the communication path 130. The apparatus with which the communication unit 305 communicates includes the moving image playback device 120. As an example, the communication unit 305 can be configured by using an existing communication interface.

  The above-described configuration of the moving image playback device 120 is an example. A part of each component of the moving image playback device 120 may be integrated. A part of each component of the video playback device 120 can be divided into a plurality of components. Some of the components of the moving image generating apparatus 110 may be omitted. The moving image generating apparatus 110 can be configured by adding other elements.

  In the present embodiment, at least one processor may generate a moving image. That is, a plurality of moving image generation devices 110 may cooperate to generate a moving image.

  Note that the communication path 130 may be either wired or wireless.

[1-2. Operation]
FIG. 4 is a flowchart showing the moving image generation processing in the first embodiment.

  In the present embodiment, the moving image generation process is mainly performed by the processor 201 of the moving image generation apparatus 110.

  In step S400, the processor 201 starts a moving image generation process. The moving image generation process includes a process of generating an encoded moving image by encoding an unencoded moving image based on a reference image. When starting the moving image generation process, the processor 201 reads from the storage unit 202 an image constituting the moving image recorded in the storage unit 202. The unit for counting this image is called a frame. The read image is called a frame image in the subsequent processing. In the present embodiment, it is assumed that the moving image recorded in the storage unit 202 has not been encoded yet. The timing at which the processor starts the moving image generation processing may be performed according to a moving image generation processing instruction for the input unit 203 performed by the user of the moving image generation device 110, or each time the input unit 203 acquires a moving image before encoding from the outside. You may go to

  In step S401, the processor 201 determines whether or not the processing target block of the image read out in step S400 (frame image to be processed = encoding target image) is an attention area block. In the present embodiment, the processor 201 performs processing by dividing a frame image into a plurality of areas called blocks. The processor 201 processes the divided blocks in order. A block to be processed by the processor 201 is called a processing target block. The attention area block is an area where a special flag is set in the block. The attention area block is also called a ROI (Region Of Interest) area. In general, the attention area block is configured by a group of blocks that occupy an area (= attention area) in an image of high interest of a person viewing a moving image. The method of determining the attention area is as follows: (1) The central area of the image (2) If the object of interest (person, face, specific object, moving object, predetermined pattern, etc.) is detected from the image, the area is detected It can be considered. The attention area block is determined so as to occupy a part or all of the attention area. Specifically, the coordinate group (x, y) of the attention area in the image is recorded in the storage unit 202, and the processor 201 performs processing by comparing the coordinate group occupied by the processing target block with the coordinate group of the attention area. It is determined whether the target block is an attention area block. A plurality of attention areas may be defined in the image. In addition, when the attention area block is a block that is encoded with a larger amount of code compared to other blocks, the effect of reducing the encoding efficiency of the present disclosure is greater, so the application of the present disclosure Is preferred.

  The attention area block is a block that exists in an area where the interest of the person browsing the moving image is high. For this reason, in the present embodiment, in order to express a moving image clearly, a filtering process such as a smoothing filter is not performed on the region occupied by the region-of-interest block. From this, it can be said that the process of step S401 in the present embodiment is a process of determining whether or not to perform a filter process on at least a partial region of the encoding target image.

  In step S <b> 402 (Yes in step S <b> 401), the processor 201 records in the storage unit 202 information indicating that the filtering process is not performed on the area occupied by the processing target block that is the determination target in step S <b> 401.

  In step S403 (No in step S401), the processor 201 determines whether the processing target block that is the determination target in step S401 belongs to the first GOP (Group Of Picture) in the moving image that is the target of the moving image generation processing. Determine whether or not.

  A GOP is a group of frame images constituting a moving image and includes an I frame image. An I-frame image is a type of image constituting an encoded moving image. When a moving image is encoded, information on frame images before and after the frame image to be encoded may be referred to. Referencing the information of previous and subsequent frame images is called interframe prediction. A frame image encoded using inter-frame prediction is called a B frame or a P frame. A frame image encoded without using inter-frame prediction is called an I frame or the like. Note that the frame name depends on the encoding method.

  In this embodiment, the processor 201 performs a moving image generation process in units of GOPs. In the present disclosure, “the processing target block belongs to the first GOP in the moving image subjected to the moving image generation processing” means that the image to which the processing target block belongs is first processed by the processor 201 in the moving image generation processing. It belongs to.

  When the processing target block belongs to the first GOP in the moving image that is the target of the moving image generation processing (Yes in step S403), the process proceeds to step S402 described above.

  In step S <b> 404 (No in step S <b> 403), the processor 201 records in the storage unit 202 information indicating that the filtering process is performed on the area occupied by the processing target block that is the determination target in step S <b> 401.

  In step S405, the processing target block is encoded according to the encoding rule. An example of an encoding rule is H.264. 265 or the like. In the encoding process in step S405, encoding using inter-frame prediction is performed (excluding I frame encoding). Interframe prediction is a technique for performing encoding based on an image called a reference image. According to inter-frame prediction, an image can be encoded based on a difference from a reference image or a motion vector. Therefore, if inter-frame prediction is used, the information amount of the encoded moving image can be reduced as compared to encoding the encoding target image itself.

  In the present embodiment, in step S405, filter processing is performed on the processing target block prior to encoding. In the present embodiment, the processor 201 refers to the information recorded in the storage unit 202 in steps S402 and S404, and determines whether or not to filter the processing target block. Specifically, the processor 201 does not perform the filtering process on a block in which information indicating that the filtering process is not performed is stored in the storage unit, and the filtering process is performed on a block in which information indicating that the filtering process is performed is described. I do.

  FIG. 5 is a diagram for explaining filter processing in the moving image generation processing according to the first embodiment. In FIG. 5, the frame image 501 is divided into blocks so that the concept of blocks can be easily understood. The hatched blocks in FIG. 5 are blocks that are not subjected to filter processing. In FIG. 5, blocks that are not hatched are blocks on which filter processing is performed. In FIG. 5, a block group surrounded by a thick line is a region-of-interest block. As shown in FIG. 5, the filtering process is not performed on the attention area of the frame image 501. In the present embodiment, except for the frame image belonging to the first GOP, in principle, filter processing is performed on the area other than the attention area block as shown in FIG.

  The filter process here may be any process that has an effect of reducing the amount of encoding in the encoding process, and an example thereof is a smoothing filter. By using the smoothing filter, the values (luminance, color) of the pixels constituting the image are smoothed according to the values of surrounding pixels. When encoding is performed using inter-frame prediction, if the images in the reference relationship are smoothed, the amount of encoding can be reduced. This is because when the images in the reference relationship are smoothed, the difference between the pixel values of the same coordinates between the images becomes small (the encoding becomes possible when the difference between the pixel values in the preceding and following images becomes small). The amount can be reduced).

  In step S406, the processor 201 determines whether there is an unprocessed block in the frame image. If there is an unprocessed block (Yes in step S406), the processor 201 changes the process target block to the unprocessed block and then returns to the process of step S401.

  In step S407 (No in step S406), the processor 201 determines that the image obtained by encoding the encoding target image is (1) not all the blocks have been subjected to the filter processing, and (2) an I frame image. It is determined whether or not there is.

  In step S408 (Yes in step S407), the processor 201 updates the long term reference image. An image that becomes a new long-term reference image by updating is an image obtained by further decoding an image obtained by encoding the encoding target image. In the present embodiment, the decoding performed in step S408 is so-called local decoding (decoding performed during the encoding process).

  A long term reference image is a type of reference image used in inter-frame prediction. The long term reference image is an image having a longer time distance from the encoding target image than other reference images. In addition, when a long-term reference image is defined in a standard for video coding, or when a concept equivalent to the long-term reference image of the present disclosure is defined, the defined image is the long-term reference image of the present disclosure. Included in the reference image.

  The concept of the long term reference image will be described with reference to FIG. FIG. 6 is a diagram showing the concept of inter-frame prediction in the moving image generation process in the first embodiment. In FIG. 6, the rhombus figure indicates a frame image after encoding. In FIG. 6, the frame images are arranged on the time axis. In FIG. 6, the older the image (the image existing on the start side of the moving image) is arranged on the left side. In FIG. 6, it is assumed that four GOPs are a set for simplifying the description. In FIG. 6, it is assumed that the frame images constituting the GOP are one I picture and three subsequent P pictures. Here, the P picture refers to an encoding target image (on the time axis) one previous image (short term reference image) and long term reference image (frame image surrounded by a thick frame in FIG. 6). It is an image obtained by encoding using inter-frame prediction based on both. As shown in FIG. 6, the long-term reference image 601 is used not only for the immediately following frame image 602 but also for encoding other P pictures (for example, the frame image 603) in inter-frame prediction. The long term reference image 601 is recorded in the storage unit 202.

  In step S409, the processor 201 determines whether there is a frame image other than the frame image to which the processing target block belongs in the moving image recorded in the storage unit 202. When the moving image recorded in the storage unit 202 includes a frame image other than the frame image to which the processing target block belongs (Yes in step S409), the processor 201 selects the processing target block as a block in the frame image (new encoding target image). Then, the process returns to step S401.

  In step S409, the processor 201 ends the moving image generation process. The encoded moving image can be generated as described above.

  In the present disclosure, by performing the process of step S408, the image obtained by encoding the encoding target image is not a filter processed on all blocks, and an image that is an I frame image is a long term. It will be used as a reference image for subsequent encoding. If it does in this way, the encoding efficiency in the encoding process performed using a filter process can be improved.

  The inventors of the present disclosure have focused on the fact that, in the prior art, the amount of encoding increases in the short term when the region of interest fluctuates between frame images. If the region of interest fluctuates between frame images, it may occur that the filter processing is not performed on the next image in the region where the filter processing has been performed on the previous image. The inventors of the present disclosure have concluded that the cause of the increase in the coding amount is that the filtering process is not performed on the area where the filtering process has been performed. That is, if only a part of the images in the reference relationship is filtered, the difference between the values of pixels having the same coordinates between the images becomes large. Encoding efficiency decreases when the difference between the values of pixels with the same coordinates between images increases. Therefore, if the filtering process is not performed on the area where the filtering process has been performed, the encoding amount will increase. It is done.

  According to the present disclosure, it is possible to use an image with a large number of regions that are not subjected to filter processing for a long-term reference image. In this way, an image having an area that is not subjected to filter processing and an area having a small difference can be used as a reference image in inter-frame prediction. Therefore, even if the filter processing is not performed on the next image in the region where the filter processing has been performed on the previous image, it is possible to suppress the reduction in encoding efficiency. As a result, the encoding efficiency in the encoding process performed using the filter process can be improved.

  In the present disclosure, the long-term reference image may be updated by another method as long as an image having an area that is not filtered and an area having a small difference can be used as the long-term reference image. That is, the process in step S407 may be replaced with another process.

  For example, the requirement that “(1) all the blocks have not been subjected to the filtering process” in step S407 may be replaced with the requirement that “the area of the area not subjected to the filter area is equal to or larger than a predetermined value”. . Even if it does in this way, even if it does not perform a filter process by the next image in the area | region where the filter process was performed by the image until then, the reduction of an encoding efficiency can be suppressed. As a result, the encoding efficiency in the encoding process performed using the filter process can be improved.

  In addition, the requirement “(2) Is it an I frame image” in step S407 may be eliminated. That is, a frame image that satisfies the condition (1) and is a B frame image or a P frame image may be used as the long term reference image. However, it is preferable from the viewpoint of encoding efficiency that the I-frame image is a long-term reference image. This is because the I frame image has little difference from the image before encoding.

  In the present embodiment, by performing the process of step S403, an image that becomes Yes is created by the process of step S407. In this way, the I frame image in the first GOP is used as the long term reference image.

  Here, the process in step S403 may be replaced with another process so that the long-term reference image can be periodically updated.

  For example, the condition of “determining whether the processing target block that is the determination target in step S401 belongs to the first GOP in the moving image generation processing target” in step S403 is “in step S401. It is determined whether or not the processing target block that is the target of determination belongs to the Nth GOP in the moving image that is the target of the moving image generation process (where N is the remainder of N ÷ M is zero). It may be changed to the condition “ In this way, the long term reference image is updated every M GOPs.

[1-3. Effect]
As described above, in the present embodiment, the moving image generating apparatus 110 generates an encoded moving image by encoding the encoding target images constituting the moving image based on the reference image. The reference image includes a long-term reference image that is a reference image having a longer temporal distance from the encoding target image than other reference images. The moving image generation method determines whether or not to perform a filtering process on at least a partial area of the encoding target image, and the area of the encoding target image that is not subjected to the filtering process is predetermined. An image obtained by decoding an image obtained by encoding an image equal to or greater than the value is used as the long term reference image.

  If it does in this way, as mentioned above, the encoding efficiency in the encoding process performed using a filter process can be improved.

  Further, in the present embodiment, the moving image generating apparatus 110 does not perform the filtering process on at least one of the encoding target images, and the long image is an image obtained by decoding the encoding target image that has not been subjected to the filtering process. Used as a term reference image.

  If it does in this way, as mentioned above, a long term reference picture can be updated appropriately. As a result, the encoding efficiency in the encoding process performed using the filter process can be further improved.

  In the present embodiment, moving image generation apparatus 110 uses an image obtained by decoding the encoding target image as the long-term reference image on condition that the encoding target image is an I frame image.

  In this way, the encoding efficiency can be further improved as described above.

(Other embodiments)
As described above, the first embodiment has been described as an example of the technique disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to an embodiment in which changes, replacements, additions, omissions, and the like are appropriately performed. Moreover, it is also possible to combine each component demonstrated in the said Embodiment 1, and it can also be set as a new embodiment.

  The above-described embodiments are for illustrating the technique in the present disclosure, and various modifications, replacements, additions, omissions, and the like can be made within the scope of the claims and their equivalents.

  The present disclosure can be applied to a camera that generates a moving image, a video editing device, and the like.

DESCRIPTION OF SYMBOLS 100 Movie transmission system 110 Movie generator 120 Movie player 130 Communication path 201 Processor 202 Storage unit 203 Input unit 204 Output unit 205 Communication unit 206 Bus 301 Processor 302 Storage unit 303 Input unit 304 Output unit 305 Communication unit 306 Bus 501 Frame image 601 Long term reference image 602 Frame image 603 Frame image

Claims (6)

A moving image generating method for generating an encoded moving image by encoding an encoding target image constituting a moving image based on a reference image,
The reference image is
Including a long-term reference image that is a reference image having a long temporal distance from the encoding target image compared to other reference images;
The video generation method includes:
Determining whether to perform filtering on at least a portion of the encoding target image;
An image obtained by decoding an image obtained by encoding an image in which the size of an area of the encoding target image that is not subjected to filter processing is a predetermined value or more is used as the long-term reference image.
Movie generation method. Filtering is not performed on at least one of the encoding target images, and an image obtained by decoding the encoding target image that has not been subjected to the filtering process is used as the long term reference image.
The moving image generation method according to claim 1. On the condition that the encoding target image is an I frame image, an image obtained by decoding the encoding target image is used as the long term reference image.
The moving image generation method according to claim 1. A moving image generating apparatus that generates an encoded moving image by encoding an encoding target image constituting a moving image based on a reference image,
The reference image is
Including a long-term reference image that is a reference image having a long temporal distance from the encoding target image compared to other reference images;
The moving image generating device includes:
Determining whether to perform filtering on at least a portion of the encoding target image;
An image obtained by decoding an image obtained by encoding an image in which the area of the region to be encoded that is not subjected to filter processing is a predetermined value or more is used as the long-term reference image.
Movie generator. Filtering is not performed on at least one of the encoding target images, and an image obtained by decoding the encoding target image that has not been subjected to the filtering process is used as the long term reference image.
The moving image generating apparatus according to claim 4. On the condition that the encoding target image is an I frame image, an image obtained by decoding the encoding target image is used as the long term reference image.
The moving image generating apparatus according to claim 4.

Images