JP5870835B2 - Moving image processing apparatus, moving image processing method, and moving image processing program - Google Patents

Description

  The present invention relates to a moving image processing apparatus, a moving image processing method, and a moving image processing program.

  In a video taken by a surveillance camera or the like, when a crime or an incident occurs, it is required to enlarge and display a region of interest (ROI) such as a criminal's face in detail. When no crime or incident has occurred, it is not necessary to enlarge a specific area and display it in detail. In the method of recording the entire image at a high resolution, the storage capacity becomes enormous and the recording cost increases. For example, there is a technique for making the image quality of the region of interest higher than the image quality of the region other than the region of interest while suppressing the increase in the data amount by making the compression rate of the region of interest lower than the compression rate of the region other than the region of interest. Are known.

  In addition, there has been proposed an image reproducing device that selects image data to be decoded in accordance with the processing capability of the mounted CPU (see, for example, Patent Document 1). For example, the image reproduction apparatus receives compressed reduced image data obtained by compression encoding a reduced image obtained by thinning a part of pixels from an original image, and additional information obtained by compression encoding a portion to be thinned. Then, for example, an image reproducing device equipped with a CPU having a processing capability higher than a predetermined reference value restores both the compressed and reduced image data and the additional information, and generates a reproduced image with higher image quality than the reduced image. In addition, for example, an image playback device equipped with a CPU having a processing capability lower than a predetermined reference value restores only compressed / reduced image data and generates a playback image in order to maintain a constant playback speed.

JP-A-8-289290

  In the method in which the compression rate of the region of interest is lower than the compression rate of the region other than the region of interest, the resolution of the region of interest is the same as the resolution of the region other than the region of interest. For this reason, in the method in which the compression rate of the region of interest is lower than the compression rate of the region other than the region of interest, even if the region of interest is enlarged and displayed, the region of interest is not always displayed finely. In addition, the method of selecting image data to be decoded in accordance with the processing capability of the CPU requires data (for example, compressed / reduced image data and additional information) recorded on the entire image at a high resolution, so that the amount of data is enormous. Become.

  In one aspect, an object of the present invention is to enlarge a region of interest and enable fine display while suppressing an increase in the amount of data.

  In one aspect of the present invention, the moving image processing apparatus includes a first image generation unit that generates first image data corresponding to a first image obtained by reducing an input image at a preset reduction rate, and an interest in the input image. A second image generating unit that generates a second image data corresponding to a second image including at least one of the plurality of region-of-interest images by thinning out the region at a thinning-out rate corresponding to the reduction rate and dividing the region into a plurality of regions-of-interest images When the first image data and the second image data are encoded to generate stream data and the second image data is encoded, a prediction is made based on the first image corresponding to the time associated with the second image. And an image encoding unit capable of performing processing.

  While suppressing an increase in the amount of data, the region of interest can be enlarged and displayed finely.

1 illustrates an example of a moving image processing system according to an embodiment. 2 illustrates an example of a moving image encoding apparatus illustrated in FIG. 1. An example of division of a region of interest image is shown. An example of allocation of a region-of-interest image is shown. The other example of allocation of the region-of-interest image is shown. The other example of allocation of the region-of-interest image is shown. An example of a frame reference relationship at the time of encoding is shown. 3 shows an example of the operation of the video encoding apparatus shown in FIG. 2 shows an example of a moving image playback apparatus shown in FIG. 10 shows an example of the operation of the moving image playback apparatus shown in FIG. The example of the moving image encoder in another embodiment is shown. 12 illustrates an example of the operation of the moving image encoding device illustrated in FIG. 11. An example of the stream configuration is shown. FIG. 12 illustrates an example of a moving image reproduction device corresponding to the moving image encoding device illustrated in FIG. 11. Fig. 15 illustrates an example of an operation of the moving image reproduction device illustrated in Fig. 14.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 shows an example of a moving image processing system SYS in one embodiment. The moving image processing system SYS includes a moving image encoding device 10 that is an aspect of the moving image processing device, a stream storage unit 100 that stores the compressed stream STRM generated by the moving image encoding device 10, and a compressed stream STRM. And a moving image reproduction device 110 for decoding. The moving image playback device 110 is also an aspect of the moving image processing device. The moving image processing system SYS is also an aspect of the moving image processing apparatus.

  The moving image encoding apparatus 10 is, for example, H.264. A compressed stream STRM based on the image data IIMG is generated using an encoding method compliant with H.264 or the like. Image data IIMG is image data of an input image. Hereinafter, the image corresponding to the image data is indicated by the same reference numerals as the image data. For example, symbol IIMG indicates an input image and input image data. The moving image encoding apparatus 10 includes, for example, a reduced image generation unit 20, an interest image generation unit 30, and an image encoding unit 40.

  The reduced image generation unit 20 receives the image data IIMG and generates image data RIMG (hereinafter also referred to as reduced image data RIMG) of an image obtained by reducing the input image IIMG. That is, the reduced image generation unit 20 generates reduced image data RIMG corresponding to the reduced image RIMG obtained by reducing the input image IIMG at a preset reduction rate.

  The interest image generation unit 30 receives the image data IIMG and generates image data SIMG (hereinafter also referred to as interest image data SIMG) for displaying a region of interest (ROI) in the input image IIMG. The region of interest is, for example, a human face. For example, the interest image generation unit 30 thins out the region of interest in the input image IIMG at a thinning rate corresponding to the reduction rate of the reduced image RIMG, and divides it into a plurality of region of interest images. Then, the interest image generation unit 30 generates interest image data SIMG corresponding to the interest image SIMG including at least one of the plurality of region-of-interest images.

  In this way, the interest image data SIMG has image data for enlarging the region of interest and displaying it finely, for example. For example, the image of the region of interest generated based on the interested image data SIMG has a higher resolution than the region of interest in the reduced image RIMG generated based on the reduced image data RIMG.

  The image encoding unit 40 encodes the reduced image data RIMG and the interest image data SIMG to generate a compressed stream STRM. The compressed stream STRM is stream data including, for example, data obtained by encoding the reduced image data RIMG and data obtained by encoding the image data of interest SIMG. For example, the image encoding unit 40 receives the reduced image data RIMG generated by the reduced image generating unit 20 and the interested image data SIMG generated by the interested image generating unit 30. Then, the image encoding unit 40 converts the reduced image data RIMG and the interest image data SIMG to H.264. A compressed stream STRM is generated by encoding with an encoding method compliant with H.264 or the like.

  For example, the image encoding unit 40 converts the reduced image data RIMG and the interest image data SIMG into the H.264 format. The encoding is performed using an encoding method compliant with H.264 MVC (Multiview Video Coding). Hereinafter, data obtained by encoding the reduced image data RIMG is also referred to as encoded data of the reduced image data RIMG, and data obtained by encoding the interest image data SIMG is also referred to as encoded data of the interest image data SIMG.

  H. In the encoding method compliant with the H.264 MVC, the image encoding unit 40 encodes the reduced image data RIMG as a base view frame, and encodes the interest image data SIMG as a non-base view frame. For example, the interest image data SIMG is encoded with reference to the reduced image data RIMG and other interest image data SIMG. As a result, H.C. A compressed stream STRM conforming to H.264 MVC is generated. That is, when encoding the image-of-interest data SIMG, the image encoding unit 40 can execute a prediction process based on the reduced image RIMG corresponding to the time associated with the image of interest SIMG.

  The compressed stream STRM generated by the moving image encoding device 10 is stored in the stream storage unit 100. The compressed stream STRM stored in the stream storage unit 100 is decoded by, for example, the moving image playback device 110. For example, the moving image reproduction device 110 reads the compressed stream STRM stored in the stream storage unit 100 and decodes the compressed stream STRM. That is, the moving image reproduction device 110 decodes the compressed stream STRM generated by the moving image encoding device 10. Thereby, an image OIMG to be displayed on a display or the like is generated.

  For example, the moving image reproduction apparatus 110 is H.264. An image decoding unit 120 that functions as an encoder compliant with H.264 and the like, and a display image generation unit 130 are included. The image decoding unit 120 decodes the compressed stream STRM received from the stream storage unit 100, and generates reduced image data RDIMG and interest image data SDIMG. Note that when there is no instruction for fine display of the region of interest (for example, a fine display request for the region of interest or a region of interest designation), the image decoding unit 120 may not generate the interest image data SDIMG. That is, the image decoding unit 120 does not have to decode the encoded data of the image data of interest SIMG when there is no instruction for fine display of the region of interest (for example, at normal time).

  The display image generation unit 130 receives the reduced image data RDIMG and the interest image data SDIMG generated by the image decoding unit 120. Then, for example, the display image generation unit 130 combines the reduced image data RDIMG and the interest image data SDIMG to generate display image image data OIMG (hereinafter also referred to as display image data OIMG). When there is no instruction for fine display of the region of interest (normal time), the display image generation unit 130 outputs reduced image data RDIMG as display image data OIMG, for example.

  When there is an instruction for fine display of the region of interest, for example, the display image generation unit 130 extracts the region of interest instructed by the user from the image of interest SDIMG, and synthesizes the extracted region of interest into the reduced image RDIMG. . Thereby, display image data OIMG including image data for finely displaying a region of interest of interest (region of interest designated by the user) is generated. As described above, in this embodiment, it is possible to display an image in which a region of interest of interest (region of interest instructed by the user) is higher in definition than other regions.

  FIG. 2 shows an example of the moving picture encoding apparatus 10 shown in FIG. In the example of FIG. 2, transfer of the image data IIMG or the like is executed via the memory 50. For example, the moving image encoding device 10 includes a reduced image generation unit 20, an interest image generation unit 30, an image encoding unit 40, and a memory 50. The memory 50 may be provided in a module such as the image encoding unit 40 or may be provided outside the moving image encoding device 10.

  The memory 50 sequentially stores image data IIMG of an image IIMG taken by a digital video camera or the like. The reduced image generation unit 20 reads the image data IIMG from the memory 50, reduces the image IIMG at a preset reduction ratio (for example, horizontal 1/2, vertical 1/2 reduction ratio), and reduces the reduced image data RIMG. Generate. Then, the reduced image generation unit 20 writes the reduced image data RIMG in the memory 50. The reduced image data RIMG is encoded by the first encoding unit 42 of the image encoding unit 40.

  The interest image generation unit 30 includes, for example, a region of interest detection unit 32, a cutout unit 34, and an image composition unit 36. The region of interest detection unit 32 reads the local decoded image data LDEC1 from the memory 50, and detects the region of interest using a face detection algorithm or the like. The local decoded image data LDEC1 is image data obtained by decoding the encoded data of the reduced image data RIMG, and is generated during the encoding process of the first encoding unit 42.

  The number, position, and size of the region of interest detected by the region of interest detection unit 32 are uniquely determined for the local decoded image LDEC1 (detection target image) by using a predetermined algorithm such as a face detection algorithm. Determined. The region-of-interest detection unit 32 notifies the segmentation unit 34 and the image composition unit 36 of the region-of-interest information CINF (number of regions of interest, position, size, etc.) regarding the detected region of interest.

  The cutout unit 34 reads the image data IIMG from the memory 50, and cuts out the region of interest from the image IIMG based on the region of interest information CINF. Thereby, the region of interest image ROI corresponding to the region of interest in the image IIMG is generated. Hereinafter, the region of interest in the image IIMG corresponding to the region of interest image ROI is indicated by the same symbol (ROI or the like) as the region of interest image.

  Here, the position (coordinates) of the region of interest indicated by the region-of-interest information CINF is a value based on the image (local decoded image LDEC1) obtained by reducing the image IIMG. Therefore, when the region of interest ROI is extracted from the image IIMG, the clipping unit 34 corrects the position (coordinates) of the region of interest indicated by the region of interest information CINF to a value that matches the image size of the image IIMG. For example, the cutout unit 34 doubles the coordinates of the region of interest indicated by the region-of-interest information CINF in the horizontal direction and the vertical direction, respectively, when the reduction ratio in the horizontal direction and the vertical direction of the reduced image RIMG is ½. To do.

  The cutout unit 34 outputs image data ROI of the region of interest ROI cut out from the image IIMG (hereinafter also referred to as region of interest image data ROI) to the image composition unit 36. Note that the cutout unit 34 may write the region-of-interest image data ROI into the memory 50. At this time, the image composition unit 36 reads the region-of-interest image data ROI from the memory 50.

  The image composition unit 36 generates the interest image data SIMG based on the region-of-interest image data ROI received from the clipping unit 34. Then, the image composition unit 36 writes the interest image data SIMG in the memory 50. Note that the interest image SIMG is generated to have the same image size as that of the reduced image RIMG, for example. For example, the image composition unit 36 thins out the pixels of the region of interest image ROI cut out by the cutout unit 34 at a thinning rate corresponding to the reduction rate of the reduced image RIMG with respect to the image IMG. Thereby, the region-of-interest image (for example, the region-of-interest image SROI shown in FIG. 3) is generated in the same image size as the region of interest in the reduced image RIMG. At this time, for example, the image composition unit 36 generates a plurality of region-of-interest images having different pixels to be thinned out.

  For example, when the respective reduction ratios in the horizontal direction and the vertical direction of the reduced image RIMG are ½ times, the image composition unit 36 samples the pixels of the region-of-interest image ROI by thinning out at four sampling phases. Thus, the region of interest image ROI is divided into four sampling phase images (region of interest images SROIa, SROIb, SROIc, SROId) as shown in FIG. When a plurality of regions of interest ROI are detected, the image composition unit 36 performs the above-described division processing (thinning sampling using different sampling phases) for each region of interest image ROI.

  Then, the image synthesis unit 36 generates the interest image data SIMG by combining the divided region-of-interest images. The divided region-of-interest image is allocated to one region of interest image SIMG, for example, as shown in FIG. Alternatively, the divided region-of-interest images may be assigned to a plurality of images of interest SIMG as shown in FIGS. 5 and 6, for example.

  The image encoding unit 40 includes a first encoding unit 42, a second encoding unit 44, and a stream synthesis unit 46. The first encoding unit 42 reads the reduced image data RIMG from the memory 50, encodes the reduced image data RIMG, and generates a compressed stream CST1. For example, the first encoding unit 42 is configured as H.264. The reduced image data RIMG is encoded as a frame of a base view by an encoding method compliant with H.264 MVC. Then, the first encoding unit 42 outputs the compressed stream CST1 generated by encoding the reduced image data RIMG to the stream synthesis unit 46.

  The first encoding unit 42 writes the local decoded image data LDEC1 generated during the encoding process of the reduced image data RIMG in the memory 50. For example, the first encoding unit 42 uses the local decoded image data LDEC1 as a reference image as necessary during the encoding process of the reduced image data RIMG.

  The second encoding unit 44 reads out the interest image data SIMG from the memory 50, encodes the interest image data SIMG, and generates a compressed stream CST2. For example, the second encoding unit 44 includes the H.264 standard. The image data SIMG of interest is encoded as a non-base view frame by an encoding method compliant with H.264 MVC. Then, the second encoding unit 42 outputs the compressed stream CST2 generated by encoding the interest image data SIMG to the stream synthesizing unit 46.

  The second encoding unit 42 writes the local decoded image data LDEC2 generated during the encoding process of the image data of interest SIMG in the memory 50. For example, the second encoding unit 44 uses the local decoded image data LDEC2 as a reference image as necessary when encoding the image data of interest SIMG. The local decoded image data LDEC2 is image data obtained by decoding the encoded data of the image data of interest SIMG.

  In addition, the second encoding unit 44 uses the local decoded image data LDEC1 as a reference image for inter-view prediction as necessary during the encoding process of the image data of interest SIMG. For example, the second encoding unit 44 reads the local decoded image data LDEC1 from the memory 50 when performing inter-view prediction.

  The stream synthesizing unit 46 receives the compressed stream CST1 generated by the first encoding unit 42 and the compressed stream CST2 generated by the second encoding unit 44, and converts the compressed streams CST1 and CST2 to H.264. It is synthesized in a format compliant with H.264 MVC. Then, the stream synthesizing unit 46 outputs the compressed stream STRM obtained by synthesizing the compressed streams CST1 and CST2 to the stream storage unit 100 shown in FIG.

  In addition, the structure of the moving image encoder 10 is not limited to this example. For example, the stream synthesis unit 46 may receive the compressed streams CST1 and CST2 from the first encoding unit 42 and the second encoding unit 44 via the memory 50. At this time, the first encoding unit 42 writes the compressed stream CST1 into the memory 50, and the second encoding unit 44 writes the compressed stream CST2 into the memory 50.

  FIG. 3 shows an example of division of the region of interest image ROI. FIG. 3 shows an example of division of the region-of-interest image ROI when the respective reduction ratios in the horizontal direction and the vertical direction of the reduced image RIMG are ½ times. The region of interest image ROI shown on the left side of FIG. 3 is the region of interest ROI cut out from the image IIMG by the cutout unit 34. Pixels Pa, Pb, Pc, and Pd indicate the respective pixels P sampled at four sampling phases A, B, C, and D. In FIG. 3, the position of each pixel P will be described with the top row of the region of interest image ROI shown on the left as the odd-numbered row and the leftmost column as the odd-numbered column.

  For example, the pixels Pa in the odd rows and the odd columns of the region-of-interest image ROI are sampled at the sampling phase A. Thereby, the region-of-interest image SROIa of the sampling phase A is generated. In other words, the image composition unit 36 performs sampling by sampling pixels P (Pa) of the region of interest image ROI at the sampling phase A, thereby generating the region of interest image SROIa that is ½ times the region of interest image ROI in the horizontal direction and the vertical direction. Generate.

  The pixels Pb in the odd rows and even columns in the region of interest image ROI are sampled at the sampling phase B, for example. Thereby, the region-of-interest image SROIb having the sampling phase B is generated. That is, the image compositing unit 36 performs sampling by sampling pixels P (Pb) of the region of interest image ROI at the sampling phase B, thereby generating the region of interest image SROIb obtained by halving the region of interest image ROI in the horizontal direction and the vertical direction. Generate.

  The pixels Pc in even rows and odd columns in the region of interest image ROI are sampled at the sampling phase C, for example. Thereby, the region-of-interest image SROIc of the sampling phase C is generated. That is, the image compositing unit 36 thins and samples the pixels P (Pc) of the region of interest image ROI at the sampling phase C, thereby generating the region of interest image SROIc that is ½ times the region of interest image ROI in the horizontal direction and the vertical direction. Generate.

  The pixels Pd in even rows and even columns in the region of interest image ROI are sampled at the sampling phase D, for example. Thereby, the region-of-interest image SROId having the sampling phase D is generated. That is, the image compositing unit 36 thins and samples the pixels P (Pd) of the region of interest image ROI at the sampling phase C, thereby obtaining the region of interest image SROId that is ½ times the region of interest image ROI in the horizontal direction and the vertical direction. Generate.

  As described above, the image composition unit 36 converts the region-of-interest image ROI into four regions-of-interest images SROIa, SROIb, SROIc, and SROId when the respective reduction ratios of the reduced image RIMG in the horizontal direction and the vertical direction are ½ times. To divide. Note that the thinning rate when the region-of-interest image ROI is divided into the region-of-interest images SROIa, SROIb, SROIc, and SROId corresponds to the reduction rate of the reduced image RIMG. Thereby, in this embodiment, the region-of-interest images SROIa, SROIb, SROIc, and SROId can be set to the same resolution as the reduced image RIMG.

  FIG. 4 shows an example of allocation of the region of interest image ROI. FIG. 4 shows an example of the allocation of the region-of-interest image ROI when the respective reduction ratios of the reduced image RIMG in the horizontal direction and the vertical direction are ½ times. The broken lines in FIG. 4 indicate the boundaries of macroblocks (16 × 16 pixels). In the example of FIG. 4, two regions of interest ROI1 and ROI2 are detected from the input image IIMG. The regions of interest RROI1 and RROI2 in the reduced image RIMG correspond to images obtained by reducing the regions of interest ROI1 and ROI2 in the input image IIMG, respectively.

  The regions of interest SROI1a, SROI1b, SROI1c, and SROI1d in the image of interest SIMG are images when the region of interest ROI1 cut out from the input image IIMG is sampled at the sampling phases A, B, C, and D. The regions of interest SROI2a, SROI2b, SROI2c, and SROI2d in the image of interest SIMG are images obtained by sampling the region of interest ROI2 cut out from the input image IIMG at the sampling phases A, B, C, and D.

  Each region of interest ROI (ROI1, ROI2, etc.) is cut out from the input image IIMG so that, for example, the region of interest image SROI (SROI1a-SROI1d, SROI2a-SROI2d, etc.) of each sampling phase can be processed in units of macroblocks. . For example, the region of interest ROI1 is cut out from the input image IIMG so that the size of the region of interest images SROI1a, SROI1b, SROI1c, and SROI1d of each sampling phase is a multiple of the size of the macroblock. Similarly, the region of interest ROI2 is cut out from the input image IIMG so that the size of the region of interest images SROI2a, SROI2b, SROI2c, and SROI2d of each sampling phase is a multiple of the size of the macroblock.

  In the example of FIG. 4, the size of the region of interest RROI1 in the reduced image RIMG and the regions of interest SROI1a, SROI1b, SROI1c, and SROI1d in the image of interest SIMG are 32 × 32 pixels. The size of the region of interest RROI2 in the reduced image RIMG and the regions of interest SROI2a, SROI2b, SROI2c, SROI2d in the image of interest SIMG are 48 × 48 pixels.

  The region-of-interest image SROI obtained by dividing the region of interest ROI is, for example, arranged together in one region-of-interest image SIMG so as to be one minute image data of the region-of-interest ROI. For example, in the block in which the region-of-interest image SROI1 is collectively arranged, the region-of-interest images SROI1a, SROI1b, SROI1c, and SROI1d of the sampling phases A, B, C, and D are respectively disposed in the upper left, upper right, lower left, and lower right. ing.

  In this way, the interest image generation unit 30 divides the region of interest ROI cut out from the input image IIMG and assigns it to one interest image SIMG. That is, the interest image generation unit 30 assigns a plurality of region-of-interest images SROI1 to a common interest image SIMG, and generates interest image data SIMG corresponding to the interest image SIMG to which the plurality of region-of-interest images SROI1 are assigned.

  When a plurality of regions of interest ROI are detected, for example, a block (SROI1a-SROI1d block, SROI2a-SROI2d block, etc.) in which the region of interest images SROI obtained by dividing the region of interest ROI are collected for each region of interest ROI. Are arranged in order from the upper left. That is, even when a plurality of regions of interest ROI are detected, the region of interest image SROI obtained by dividing each region of interest ROI becomes the image data that displays each region of interest ROI in a fine manner in one region of interest SIMG. Assigned.

  Further, for example, when the sum of the sizes of the plurality of detected regions of interest ROI is larger than the size of the image of interest SIMG, the region of interest ROI gives priority to each region of interest ROI. Then, the interest image generation unit 30 assigns the divided region of interest images SROI in the region of interest SIMG in order from the region of interest ROI with the highest priority. A region of interest ROI that has not been allocated in the image of interest SIMG (region of interest ROI with a relatively low priority) is excluded from the candidates for the region of interest ROI that can be finely decoded.

  In this embodiment, the arrangement of the region-of-interest image SROI allocated by the image-of-interest generation unit 30 is uniquely determined with respect to the detected region of interest ROI. Thereby, the moving image reproduction apparatus 110 detects the region of interest RROI (RROI1, RROI2, etc.) in the reduced image RDIMG (the image obtained by decoding the encoded data of the reduced image data RIMG), for example, thereby detecting the inside of the interested image SDIMG. Of the region of interest image SROI can be recognized.

  Since the region other than the region-of-interest image SROI in the image of interest SIMG is a region that is not required on the decoding side, any image may be inserted. For example, the region other than the region-of-interest image SROI in the image of interest SIMG may be a copy of an image at a corresponding position in the reduced image RIMG, or may be a fixed color (single color) image. In this embodiment, by making a region other than the region of interest image SROI in the image of interest SIMG a copy of an image at a corresponding position in the reduced image RIMG, image data of the region other than the region of interest image ROI in the image of interest SIMG The compression ratio can be increased.

  In this embodiment, as described with reference to FIG. 3, the region-of-interest image SROI is generated by thinning out pixels of the region-of-interest image ROI at a thinning rate corresponding to the reduction rate of the reduced image RIMG. Thereby, the size and resolution of the region-of-interest image SROI become the same as the size and resolution of the reduced image RIMG. Here, for example, in the interest image SIMG in which the region of interest ROI in the input image IIMG is copied as it is, even if the reduced image RIMG is referred to, the size and resolution of the region of interest are different, so that the compression rate cannot be improved.

  In contrast, in this embodiment, as described above, the size and resolution of each region of interest image SROI are the same as the size and resolution of each region of interest RROI in the reduced image RIMG corresponding to each region of interest image SROI. . For this reason, in this embodiment, the probability that the prediction between the views of the image of interest SIMG and the reduced image RIMG will be improved. As a result, in this embodiment, the compression rate when encoding the region-of-interest image SROI can be increased, and an increase in the amount of data can be suppressed.

  Furthermore, in this embodiment, since the positional relationship between each region of interest image SROI in the image of interest SIMG and each region of interest RROI in the reduced image RIMG is known, the search range for obtaining the motion vector can be narrowed. Thereby, in this embodiment, the motion vector search process can be executed efficiently.

  Note that the arrangement of the region-of-interest images SROI of the sampling phases A, B, C, and D is not limited to this example. For example, the region-of-interest images SROI1a, SROI1b, SROI1c, and SROI1d of the sampling phases A, B, C, and D may be sequentially arranged from the left side to the right side. For example, when the reduced image RIMG is generated by pixel thinning, the region-of-interest image SROI having a sampling phase corresponding to the sampling phase when the reduced image RIMG is generated may not be allocated in the image of interest SIMG. Good.

  Alternatively, the image composition unit 36 selects at least one sampling phase from a plurality of sampling phases calculated according to the reduction rate of the reduced image RIMG, and only the region-of-interest image SROI of the selected sampling phase is included in the image of interest SIMG. May be assigned. In this case, the data amount can be reduced as compared with the case where the region-of-interest images SROI of all the sampling phases calculated according to the reduction ratio of the reduced image RIMG are allocated in the image of interest SIMG.

  In the moving image processing system SYS that selects the region-of-interest image SROI to be allocated in the image of interest SIMG, the moving image reproduction device 110, for example, corresponds the pixel P corresponding to the unselected sampling phase to the selected sampling phase. Interpolation is performed using the pixel P. Also at this time, since the image information regarding the region of interest ROI is larger than when only the reduced image RIMG is used, it is possible to display the region of interest ROI with higher definition than when only the reduced image RIMG is used.

  FIG. 5 shows another example of allocation of the region of interest image ROI. FIG. 5 shows an example of the allocation of the region-of-interest image ROI when the respective reduction ratios of the reduced image RIMG in the horizontal direction and the vertical direction are ½ times. The meaning of the broken line in FIG. 5 is the same as that in FIG. The input image IIMG and the reduced image RIMG are the same as in the example of FIG. For example, two regions of interest ROI1 and ROI2 are detected from the input image IIMG.

  In the example of FIG. 5, the region-of-interest images SROI having a plurality of sampling phases calculated according to the reduction rate of the reduced image RIMG are allocated to the plurality of images of interest SIMG at the common time. The images of interest SIMGa, SIMGb, SIMGc, and SIMGd are images of interest SIMG at a common time. For example, the images of interest SIMGa, SIMGb, SIMGc, and SIMGd are each encoded as non-base-view frames of different sequences.

  For example, the region-of-interest images SROI1a, SROI1b, SROI1c, and SROI1d of four sampling phases are assigned to the four images of interest SIMGa, SIMGb, SIMGc, and SIMGd, respectively. Note that the region-of-interest image SROI1 in each image of interest SIMG is arranged at a position (for example, the same coordinates) corresponding to the region of interest RROI1 in the reduced image RIMG, for example. Similarly, the region-of-interest images SROI2a, SROI2b, SROI2c, and SROI2d are respectively assigned to the images of interest SIMGa, SIMGb, SIMGc, and SIMGd at positions (for example, the same coordinates) corresponding to the region of interest RROI2 in the reduced image RIMG.

  In this way, the interested image generation unit 30 assigns the regions of interest image SROI having a plurality of sampling phases to each interested image SIMG by one phase. The region other than the region-of-interest image SROI in the image of interest SIMG is, for example, an image obtained by copying an image at a corresponding position in the reduced image RIMG. At this time, in the region other than the region-of-interest image SROI in the image of interest SIMG, the difference value and motion in inter-view prediction using the local decoded image LDEC1 (image data obtained by decoding the encoded data of the reduced image data RIMG) as a reference image The vector can be almost zero. The region other than the region-of-interest image SROI in the image of interest SIMG may be, for example, a fixed color (single color) image. Also at this time, the compression rate of the image data in the region other than the region of interest image ROI in the image of interest SIMG can be increased.

  As described above, the method shown in FIG. 5 can achieve the same effect as the method shown in FIG. For example, also in the method illustrated in FIG. 5, it is possible to improve the probability that prediction between the views of the image of interest SIMG and the reduced image RIMG will be achieved. In the method shown in FIG. 5 as well, the search range for obtaining the motion vector can be narrowed, and the motion vector search process can be executed efficiently.

  Further, in the method shown in FIG. 5, the region of interest image SROI is interested so that the coordinates (position) of the region of interest image SROI in the image of interest SIMG are equal to the coordinates (position) of the region of interest RROI in the reduced image RIMG. Assigned to image SIMG. For this reason, in the method shown in FIG. 5, the code amount of the motion vector can be reduced as compared with the method shown in FIG.

  Note that the arrangement of the region-of-interest images SROI of the sampling phases A, B, C, and D is not limited to this example. For example, when the reduced image RIMG is generated by thinning out pixels, the region-of-interest image SROI having a sampling phase corresponding to the sampling phase when the reduced image RIMG is generated may not be allocated in the image of interest SIMG. At this time, compared with the case of generating the image of interest SIMG corresponding to each of a plurality of sampling phases calculated according to the reduction ratio of the reduced image RIMG, the number of images of interest SIMG at the common time can be reduced, and the compressed stream STRM The amount of data can be reduced.

  Further, for example, the image composition unit 36 selects at least one sampling phase from a plurality of sampling phases calculated according to the reduction ratio of the reduced image RIMG, and selects only the region of interest image SROI of the selected sampling phase. You may allocate in SIMG. Also at this time, the number of images of interest SIMG at a common time can be reduced compared with the case of generating images of interest SIMG respectively corresponding to a plurality of sampling phases calculated according to the reduction ratio of the reduced image RIMG. The data amount of the stream STRM can be reduced.

  FIG. 6 shows another example of allocation of the region of interest image ROI. FIG. 6 shows an example of the allocation of the region-of-interest image ROI when the respective reduction ratios of the reduced image RIMG in the horizontal direction and the vertical direction are ½ times. Each frame time t (t0, t1, t2, t3,...) In FIG. 6 indicates the time associated with the reduced image RIMG or the like, for example. In FIG. 6, for the reduced image RIMG and the interest image SIMG corresponding to each frame time t (t0, t1, t2, t3,...), The number at the end of the code of the frame time t is added to the end of the code. doing. For example, the reduced image RIMG0 and the interest image SIMG0 are the reduced image RIMG and the interest image SIMG at the frame time t0.

  In FIG. 6, the input image IIMG is not shown. For example, the input image IIMG at the time (frame time t0) corresponding to the reduced image RIMG0 and the image of interest SIMG0 is the same as the input image IIMG shown in FIG. Therefore, the reduced image RIMG0 is the same as the reduced image RIMG shown in FIG. The meaning of the broken line in FIG. 6 is the same as that in FIG. In the interest images SIMG1, SIMG2, and SIMG3 in FIG. 6, the broken lines are omitted for easy understanding of the drawing.

  In the example of FIG. 6, one image of interest SIMG is divided into a plurality of areas AR (AR0 to AR3), and the plurality of areas AR correspond to a plurality of frames (images) at different times of the reduced image RIMG. Yes. For example, the region-of-interest images SROI having a plurality of sampling phases calculated according to the reduction ratio of the reduced image RIMG are respectively assigned to the predetermined regions AR of the plurality of images of interest SIMG at different frame times t. That is, the interested image generation unit 30 assigns the region-of-interest images SROI having a plurality of sampling phases one phase at a time to the plurality of images of interest SIMG at different frame times t.

  For example, a region of interest image SROI (such as SOI 10a) obtained by dividing the region of interest ROI in the input image IIMG at the frame times t0, t4, t8,... Is assigned to the upper left region AR0 in the image of interest SIMG. Further, for example, a region of interest image SROI (SOI11a or the like) obtained by dividing the region of interest ROI in the input image IIMG at frame times t1, t5, t9,... Is assigned to the upper right region AR1 in the image of interest SIMG. .

  Then, for example, a region-of-interest image SROI (SOI12a or the like) obtained by dividing the region-of-interest ROI in the input image IIMG at frame times t2, t6, t10,... . Further, for example, a region of interest image SROI (SOI13a or the like) obtained by dividing the region of interest ROI in the input image IIMG at frame times t3, t9, t11,... Is assigned to the lower right region AR3 in the image of interest SIMG. It is done.

  That is, the region of interest images SROI10a and SROI20a of the sampling phase A obtained by dividing the region of interest ROI of the input image IIMG at the frame time t0 are assigned to the region AR0 of the image of interest SIMG0 at the frame time t0. Then, the region-of-interest images SROI10b and SROI20b at the sampling phase B are assigned to the region AR0 of the image of interest SIMG1 at the frame time t1. The region-of-interest images SROI10c and SROI20c at the sampling phase C are allocated to the region RA0 of the image of interest SIMG2 at the frame time t2. The region-of-interest images SROI10d and SROI20d of the sampling phase D are allocated to the region RA0 of the image of interest SIMG3 at the frame time t3.

  Similarly, the region of interest images SROI of the sampling phases A, B, C, and D obtained by dividing the region of interest ROI of the input image IIMG at the frame time t1 are respectively allocated to the region RA1 of the image of interest SIMG at the frame times t1 to t4. Then, the region of interest images SROI of the sampling phases A, B, C, and D obtained by dividing the region of interest ROI of the input image IIMG at the frame time t2 are respectively assigned to the region RA2 of the image of interest SIMG at the frame time t2-t5. Further, the region of interest images SROI of the sampling phases A, B, C, and D obtained by dividing the region of interest ROI of the input image IIMG at the frame time t3 are respectively assigned to the region RA3 of the image of interest SIMG at the frame time t3 to t6.

  The region other than the region-of-interest image SROI in the image of interest SIMG at each frame time t may be, for example, a copy of an image at a corresponding position of the reduced image RIMG at each frame time t, or may be a fixed color (single color) image. . Also in the method shown in FIG. 6, the same effect as the method shown in FIG. 4 can be obtained. For example, also in the method illustrated in FIG. 6, it is possible to improve the probability that the prediction between the views of the image of interest SIMG and the reduced image RIMG is successful. In the method shown in FIG. 6 as well, the search range for obtaining the motion vector can be narrowed, and the motion vector search process can be executed efficiently.

  Furthermore, in the method shown in FIG. 6, it is also possible to refer to the region-of-interest image SROI of different sampling phases generated from the common input image IIMG in the inter-frame prediction (inter prediction) of the image of interest SIMG. For example, in the method illustrated in FIG. 6, the region-of-interest image SROI 10 a of the image of interest SIMG 0 can be referred to when the region-of-interest image SROI 10 b of the image of interest SIMG 1 is encoded. For this reason, in the method shown in FIG. 6, prediction with a higher compression rate can be selected between inter-view prediction and inter-frame prediction, and the compression rate can be improved.

  Note that the arrangement of the region-of-interest images SROI of the sampling phases A, B, C, and D is not limited to this example. For example, when the reduced image RIMG is generated by thinning out pixels, the region-of-interest image SROI having a sampling phase corresponding to the sampling phase when the reduced image RIMG is generated may not be allocated in the image of interest SIMG. Further, for example, the image composition unit 36 selects at least one sampling phase from a plurality of sampling phases calculated according to the reduction ratio of the reduced image RIMG, and selects only the region of interest image SROI of the selected sampling phase. You may allocate in SIMG.

  FIG. 7 shows an example of a frame reference relationship at the time of encoding. Note that FIG. 2 illustrates an example of a frame reference relationship when encoding is performed using an encoding method compliant with H.264 MVC. In the figure, I and P in parentheses indicate an I frame and a P frame, respectively. For example, the moving image encoding apparatus 10 processes the reduced image RIMG as a frame of a base view, and processes the image of interest SIMG as a frame of a non-base view.

  The I frame of the reduced image RIMG processed as the base view is encoded using, for example, intra-frame prediction (intra prediction). Also, the P frame of the reduced image RIMG is encoded using inter-frame prediction. The P frame of the image of interest SIMG processed as the non-base view is encoded using, for example, inter-frame prediction that refers to the frame of the non-base view or inter-view prediction that refers to the corresponding frame of the base view. By using inter-view prediction, the moving image encoding apparatus 10 can encode a non-base view interest image SIMG without using an I frame with a large code amount. As a result, in this embodiment, an increase in the data amount of the compressed stream STRM can be suppressed.

  FIG. 8 shows an example of the operation of the moving picture encoding apparatus 10 shown in FIG. The operation of FIG. 8 may be realized only by hardware, or may be realized by controlling the hardware by software. For example, software such as a moving image processing program causes a computer to execute the operation of FIG.

  In process S100, the moving image encoding device 10 acquires the input image IIMG. For example, the moving image encoding device 10 sequentially stores in the memory 50 the image data IIMG of the image IIMG taken by a digital video camera or the like.

  In process S110, the reduced image generation unit 20 generates a reduced image RIMG of the input image IIMG. For example, the reduced image generation unit 20 reads the image data IIMG from the memory 50, reduces the image IIMG at a preset reduction rate, and generates reduced image data RIMG. The reduced image data RIMG is stored in the memory 50, for example.

  In the process S120, the first encoding unit 42 encodes the reduced image RIMG generated in the process S110 as a frame of the base view. For example, the first encoding unit 42 reads the reduced image data RIMG from the memory 50, encodes the reduced image data RIMG, and generates a compressed stream CST1. Note that the local decoded image data LDEC1 generated during the encoding process of the reduced image data RIMG is stored in the memory 50, for example. The local decoded image data LDEC1 stored in the memory 50 is used as a reference image as necessary. For example, the first encoding unit 42 reads the local decoded image data LDEC1 from the memory 50 when performing inter-frame prediction.

  In process S130, the region-of-interest detection unit 32 detects a region of interest using the local decoded image LDEC1 generated during the encoding process in step S120. For example, the region of interest detection unit 32 reads the local decoded image data LDEC1 from the memory 50, and detects the region of interest using a face detection algorithm or the like.

  In the process S140, the cutout unit 34 cuts out the region of interest ROI from the input image IIMG based on the detection result of the process S130. For example, the cutout unit 34 reads the input image data IIMG from the memory 50, and extracts the region-of-interest image data ROI corresponding to the region of interest detected in step S130 from the input image data IIMG. As described with reference to FIG. 2, the clipping unit 34 extracts information (coordinates and the like) related to the region of interest ROI detected in step S130 when extracting the region of interest image data ROI from the input image data IIMG. The value is adjusted to match the image size. That is, the cutout unit 34 calculates the position of the region of interest ROI in the input image IIMG based on the detection result of the process S130.

  In the process S150, the image composition unit 36 divides the region of interest ROI cut out in the process S140 into the region of interest images SROI having a plurality of sampling phases, and synthesizes the divided region of interest image SROI. As a result, an interest image SIMG that is a composite image of the region of interest image SROI is generated. For example, the image composition unit 36 generates the interest image SIMG using one of the region of interest image ROI allocation methods illustrated in FIGS. 4 to 6. The image composition unit 36 may generate the interest image SIMG using a method other than the region of interest image ROI allocation method illustrated in FIGS. 4 to 6. The region-of-interest image SROI is stored in the memory 50, for example.

  In the process S160, the second encoding unit 44 encodes the composite image (interest image SIMG) generated in the process S150 as a non-base view frame. For example, the second encoding unit 44 reads out the interest image data SIMG from the memory 50, encodes the interest image data SIMG, and generates a compressed stream CST2. Note that the local decoded image data LDEC2 generated during the encoding process of the interest image data SIMG is stored in the memory 50, for example.

  The local decoded image data LDEC1 and LDEC2 stored in the memory 50 are used as reference images as necessary. For example, the second encoding unit 44 reads the local decoded image data LDEC2 from the memory 50 when executing inter-frame prediction with reference to a non-base view frame. For example, the second encoding unit 44 reads the local decoded image data LDEC1 from the memory 50 when performing inter-view prediction.

  In process S170, the stream synthesis unit 46 synthesizes the compressed stream CST1 generated in process S120 and the compressed stream CST2 generated in process S160 to generate a compressed stream STRM. That is, the compressed stream STRM includes encoded data of reduced image data RIMG and encoded data of interest image data SIMG.

  The moving image encoding device 10 sequentially encodes the input image IIMG by repeating the processes S100 to S170. Note that the compressed stream STRM is stored in, for example, the stream storage unit 100 illustrated in FIG. The compressed stream STRM stored in the stream storage unit 100 is reproduced by, for example, the moving image reproduction apparatus 110 illustrated in FIG.

  Note that the operation of the video encoding device 10 is not limited to this example. For example, the interested image generation unit 30 performs a process of dividing the region of interest ROI into a plurality of regions of interest image SROI. The frame interval of the input image IIMG (for example, the interval from the frame time t0 to the frame time t1 illustrated in FIG. 6). It may be performed at larger intervals. That is, the region-of-interest detection unit 32 may execute, for example, detection of a region of interest at an interval larger than the frame interval. In addition, the cutout unit 34 may execute the region of interest ROI cutout processing (processing of cutting out the region of interest ROI from the input image IIMG) at an interval larger than the frame interval.

  The interest image SIMG of the frame in which the region of interest ROI cut-out process is not executed is the same as the interest image SIMG of the previous frame, for example. Note that the interest image SIMG of the frame in which the region of interest ROI cut-out process is not executed may be the same as the interest image SIMG of the previous frame. For this reason, in the moving image encoding apparatus 10 in which the execution interval of the clipping process is larger than the frame interval, the data amount of the compressed stream CST1 including the encoded data of the image data of interest SIMG is compared with that when the clipping process is performed for each frame. Can be reduced.

  FIG. 9 shows an example of the moving image playback device 110 shown in FIG. In the example of FIG. 9, transfer of image data RDIMG or the like is executed via the memory 140. For example, the moving image reproduction device 110 includes an image decoding unit 120, a display image generation unit 130, and a memory 140. Note that the memory 140 may be provided in a module such as the image decoding unit 120 or may be provided outside the moving image reproduction apparatus 110.

  The image decoding unit 120 includes, for example, a stream separation unit 122, a first decoding unit 124, and a second decoding unit 126. For example, the stream separation unit 122 receives the compressed stream STRM from the stream storage unit 100 illustrated in FIG. Then, the stream separation unit 122 separates the compressed stream STRM into a compressed stream CST1 for base view and a compressed stream CST2 for non-base view. The compressed stream CST1 has encoded data of reduced image data RIMG, and the compressed stream CST2 has encoded data of interest image data SIMG.

  The first decoding unit 124 receives the compressed stream CST1 from the stream separation unit 122, and decodes the compressed stream CST1. For example, the first decoding unit 124 may H.264-compliant base view decoding processing is executed on the compressed stream CST1. As a result, the compressed stream CST1 is decoded, and reduced image data RDIMG is generated. For example, the first decoding unit 124 writes the reduced image data RDIMG in the memory 140.

  The reduced image data RDIMG is decoded image data LDEC1 obtained by decoding the encoded data of the reduced image data RIMG, and is used as a display image. Hereinafter, the reduced image data RDIMG is also referred to as decoded image data LDEC1. The decoded image data LDEC1 is used as necessary as a reference image for decoding processing. For example, the first decoding unit 124 reads the decoded image data LDEC1 from the memory 140 when using the decoded image data LDEC1 as a reference image for decoding processing.

  The second decoding unit 126 receives the compressed stream CST2 from the stream separation unit 122. Then, the second decoding unit 126 decodes the compressed stream CST2 when there is an instruction for fine display of the region of interest, for example. For example, the second decoding unit 126 is configured to transmit the H.264 data. H.264 compliant non-base view decoding processing is performed on the compressed stream CST2. Thereby, the compressed stream CST2 is decoded, and the interest image data SDIMG is generated. For example, the second decoding unit 126 writes the interest image data SDIMG in the memory 140.

  The interest image data SDIMG is decoded image data LDEC2 obtained by decoding the encoded data of the interest image data SIMG, and is used when the region of interest is displayed finely. Hereinafter, the interest image data SDIMG is also referred to as decoded image data LDEC2. The decoded image data LDEC2 is used as necessary as a reference image for decoding processing. For example, the second decoding unit 126 reads the decoded image data LDEC2 from the memory 140 when using the decoded image data LDEC2 as a reference image for decoding processing.

  The display image generation unit 130 includes, for example, a region of interest detection unit 132, a cutout unit 134, and a display image synthesis unit 136. The region-of-interest detection unit 132 reads the reduced image data RDIMG from the memory 50 when there is an instruction for fine display of the region of interest, for example. Then, the region of interest detection unit 132 detects the region of interest using the read reduced image data RDIMG. That is, the region-of-interest detection unit 132 detects the region of interest when there is an instruction for fine display of the region of interest.

  For example, the region of interest detection unit 132 detects the region of interest using a region of interest detection algorithm executed by the region of interest detection unit 32 of the video encoding device 10. Then, the region-of-interest detection unit 32 notifies the segmentation unit 134 and the display image synthesis unit 136 of the region-of-interest information CINF (number of regions of interest, position, size, etc.) regarding the detected region of interest.

  Here, in the region-of-interest detection algorithm used by the region-of-interest detection unit 132 and the region-of-interest detection unit 32, the number, position, and size of the region of interest to be detected are uniquely determined with respect to the detection target image. For example, since the reduced image data RDIMG is image data obtained by decoding the encoded data of the reduced image data RIMG, the reduced image data RDIMG is the same image data as the local decoded image data LDEC1 used in the region of interest detection unit 32.

  Therefore, the region of interest detection unit 132 detects a region of interest similar to the region of interest detected by the region of interest detection unit 32 of the moving image encoding device 10. For this reason, in this embodiment, the moving image encoding apparatus 10 does not have to embed the position information of the region of interest or the like in the compressed stream STRM.

  The cutout unit 134 reads out the interest image data SDIMG from the memory 140 when, for example, there is an instruction for fine display of the region of interest. Then, the cutout unit 134 cuts out the region-of-interest image SROI corresponding to the designated region of interest from the region-of-interest image SDIMG based on the region-of-interest information CINF. In this embodiment, the arrangement of the region-of-interest image SROI allocated by the image-of-interest generation unit 30 of the moving image encoding device 10 is uniquely determined for the detected region of interest. Therefore, the cutout unit 134 can recognize the arrangement of the region of interest image SROI in the image of interest SDIMG, for example, by using the region of interest information CINF.

  The display image composition unit 136 reads the reduced image data RDIMG from the memory 140. The display image composition unit 136 outputs the reduced image data RDIMG as the display image data OIMG when there is no instruction for fine display of the region of interest (normal time). Further, when there is an instruction for fine display of the region of interest, the display image composition unit 136 synthesizes the reduced image data RDIMG and the region of interest image data SROI received from the clipping unit 134 based on the region of interest information CINF and the like. Display image data OIMG is generated. Thereby, display image data OIMG including image data for finely displaying a region of interest of interest (region of interest designated by the user) is generated.

  For example, the display image composition unit 136 receives the region-of-interest image data SROI of the region of interest cut out by the cut-out unit 134 when there is an instruction for fine display of the region of interest. Then, the display image composition unit 136 generates the region-of-interest image data ROI of the region of interest of interest (region of interest designated by the user) using the region-of-interest image data SROI of each sampling phase received from the clipping unit 134. .

  As a result, a region of interest image ROI that is finer than the reduced image RDIMG is generated. The region-of-interest image data ROI generated based on the region-of-interest image data SROI is combined with the reduced image data RDIMG and included in the display image data OIMG. Thereby, in this embodiment, the designated region of interest ROI can be displayed more finely than the reduced image RIMG.

  For example, when the respective reduction ratios in the horizontal direction and the vertical direction of the reduced image RIMG are ½ times, the display image composition unit 136 enlarges the reduced image RDIMG in the horizontal direction and the vertical direction and the region-of-interest image. The region of interest image ROI generated from the data SROI is synthesized to generate a display image OIMG. The display image composition unit 136 arranges the region of interest image ROI generated from the region of interest image data SROI (region of interest image ROI enlarged with respect to the reduced image RDIMG) at an arbitrary position of the reduced image RDIMG, and displays it. An image OIMG may be generated.

  FIG. 10 shows an example of the operation of the moving image playback apparatus 110 shown in FIG. The operation of FIG. 10 may be realized only by hardware, or may be realized by controlling the hardware by software. For example, software such as a moving image processing program causes a computer to execute the operation of FIG.

  In process S200, the moving image playback device 110 acquires the compressed stream STRM. For example, the stream separation unit 122 receives the compressed stream STRM from the stream storage unit 100 illustrated in FIG.

  In process S210, the stream separation unit 122 separates the compressed stream STRM acquired in process S200 into a compressed stream CST1 for base view and a compressed stream CST2 for non-base view.

  In the process S220, the first decoding unit 124 applies, for example, H.264 to the compressed stream CST1 separated in the process S210. H.264-compliant base view decoding processing is executed. Thereby, the encoded data of the reduced image data RIMG is decoded, and reduced image data RDIMG is generated. The reduced image data RDIMG is stored in the memory 140, for example.

  In process S230, the display image composition unit 136 outputs the reduced image data RDIMG decoded in process S220 as display image data OIMG. Thereby, the display image OIMG based on the reduced image data RDIMG is displayed. The display device that displays the display image OIMG may be provided as a part of the moving image reproduction device 110 or may be provided outside the moving image reproduction device 110.

  In process S240, the moving image playback device 110 determines whether or not there is a request for fine display of the region of interest. For example, when the user wants to display the region of interest in detail, the user notifies the moving image playback device 110 of a fine display request using the user interface of the video playback device 110 or the like. Thereby, the display image generation unit 130 or the like receives a fine display request from the user interface or the like when there is a fine display request for the region of interest.

  When there is no request for fine display of the region of interest (No in process S240), the moving image reproducing apparatus 110 ends the series of processes. That is, the decoding process is always performed on the compressed stream CST1 for base view (encoded data of the reduced image data RIMG). When there is no fine display request from the user, only an image based on the reduced image RDIMG is displayed. On the other hand, when there is a request for fine display of the region of interest (Yes in process S240), the operation of the moving image reproduction apparatus 110 proceeds to process S250.

  In process S250, the region of interest detection unit 132 detects the region of interest using the reduced image data RDIMG decoded in process S220. Note that the moving image playback device 110 may reflect a mark or the like indicating the detected region of interest on the display image.

  In step S260, the moving image playback device 110 determines whether there is a designation of a region of interest for fine display. For example, when the user wants to display the region of interest in detail, the user designates the region of interest to be displayed in detail using the user interface of the moving image reproduction apparatus 110 or the like. Accordingly, the image decoding unit 120, the display image generation unit 130, and the like receive, for example, information indicating the designated region of interest from the user interface or the like. When there is no region of interest designation (No in process S260), the moving image reproducing apparatus 110 ends a series of processes. On the other hand, when there is a region of interest designation (Yes in process S260), the operation of the moving image reproduction device 110 proceeds to process S270.

  In the process S270, the second decoding unit 126 applies, for example, H.264 to the compressed stream CST2 separated in the process S210. H.264 compliant non-base view decoding processing is executed. Thereby, the encoded data of the interest image data SIMG is decoded, and the interest image data SDIMG is generated. That is, for the compressed stream CST2 for non-base view (encoded data of the image data SIMG of interest), the decoding process is executed when the region of interest is designated. The interest image data SDIMG is stored in the memory 140, for example.

  In the process S280, the cutout unit 134 cuts out the region of interest image SROI corresponding to the designated region of interest from the image of interest SDIMG based on the region of interest information CINF regarding the region of interest detected in the process S250.

  In process S290, the region-of-interest image SROI cut out in process S280 and the reduced image RDIMG decoded in process S220 are combined to generate a display image OIMG. Thereby, in process S300, the synthesized image (display image OIMG) including the region of interest with higher definition than other regions is displayed. Thus, in this embodiment, the designated region of interest ROI can be displayed with higher definition than the reduced image RIMG. For example, in this embodiment, the region of interest ROI can be enlarged and displayed finely.

  As described above, in this embodiment, the moving image processing system SYS encodes the reduced image RIMG of the input image IIMG and the interest image SIMG for finely displaying the region of interest ROI in the input image IIMG. have. For example, as described with reference to FIG. 3, the moving image encoding apparatus 10 thins out the region of interest ROI in the input image IIMG at a thinning rate corresponding to the reduction rate of the reduced image RIMG, and assigns it to the interested image SIMG.

  For example, when encoding the image of interest SIMG, the moving image encoding device 10 uses the reduced image RIMG as a reference image as necessary. Thereby, in this embodiment, the compression rate at the time of encoding the interested image SIMG including the region-of-interest image SROI can be increased, and an increase in the data amount can be suppressed. Note that the plurality of region-of-interest images SROI obtained by dividing the region of interest ROI may be allocated to one region-of-interest image SIMG, for example, as described with reference to FIGS. May be.

  For example, in this embodiment, as shown in FIG. 5, the interest image generation unit 30 assigns a plurality of region-of-interest images SROI to a plurality of interest images SIMG at a common time, and corresponds to the plurality of interest images SIMG, respectively. Interest image data SIMG is generated. Thereby, in this embodiment, the code amount of a motion vector can be reduced relatively.

  Further, for example, in this embodiment, as shown in FIG. 6, the interest image generation unit 30 assigns a plurality of region-of-interest images SROI to a plurality of interest images SIMG at different times, and assigns each of the plurality of interest images SIMG. Corresponding image data of interest SIMG may be generated. Thereby, in this embodiment, when encoding the image of interest SIMG, the prediction with the higher compression rate can be selected between the inter-view prediction and the inter-frame prediction, and the compression rate can be improved.

  Furthermore, in this embodiment, the moving image processing system SYS includes a moving image reproduction device 110 that decodes the compressed stream STRM generated by the moving image encoding device 10. The moving image reproduction apparatus 110 includes, for example, a region of interest detection unit 132 that calculates the position of the region of interest.

  The region of interest detection unit 132 calculates the position of the region of interest ROI in the input image IIMG by using the method of calculating the position of the region of interest ROI by the region of interest detection unit 32 of the moving image encoding device 10. For example, the region of interest detection unit 132 detects a region of interest from the reduced image data RDIMG obtained by decoding the encoded data of the reduced image data RIMG, and calculates the position of the region of interest ROI in the input image IIMG based on the detection result. . Thereby, in this embodiment, the moving image encoding device 10 does not have to embed the position information of the region of interest or the like in the compressed stream STRM. As a result, in this embodiment, the data amount of the compressed stream STRM can be reduced.

  FIG. 11 shows an example of a moving image encoding apparatus 10A according to another embodiment. Elements similar to those described in FIGS. 1 to 10 are denoted by the same reference numerals, and detailed description thereof will be omitted. In this embodiment, the moving image processing system SYS includes, for example, the moving image encoding device 10A, the stream storage unit 100 illustrated in FIG. 1, and the moving image reproduction device 110A illustrated in FIG. The moving image encoding apparatus 10A includes an interest image generation unit 30A and an image encoding unit 40A instead of the interest image generation unit 30 and the image encoding unit 40 illustrated in FIG. Other configurations of the moving image encoding device 10A are the same as those in the embodiment described with reference to FIGS.

  For example, the moving image encoding device 10A includes a reduced image generation unit 20, an interest image generation unit 30A, an image encoding unit 40A, and a memory 50. The reduced image generation unit 20 is the same as the embodiment described with reference to FIGS. For example, the reduced image generation unit 20 is the same as the reduced image generation unit 20 described in FIG. The interest image generation unit 30A includes, for example, a region of interest detection unit 32A, a cutout unit 34, and an image composition unit 36. The region of interest detection unit 32A reads the image data IIMG from the memory 50, and detects the region of interest ROI using a face detection algorithm or the like.

  That is, the region of interest detection unit 32A detects the region of interest ROI from the input image IIMG. The region-of-interest detection unit 32A notifies the region-of-interest information CINF (the number, position, size, etc. of the region of interest ROI) regarding the detected region of interest ROI to the cutout unit 34, the image composition unit 36, and the image encoding unit 40A. . Note that the position (coordinates) of the region of interest indicated by the region-of-interest information CINF is a value based on the image IIMG.

  The cutout unit 34 and the image composition unit 36 are the same as those in the embodiment described with reference to FIGS. Note that the cutout unit 34 does not need to correct the value of the region-of-interest information CINF because the value (coordinates) of the region-of-interest information CINF is a value based on the image IIMG.

  The image encoding unit 40A includes a first encoding unit 42, a second encoding unit 44, and a stream synthesis unit 46A. The 1st encoding part 42 and the 2nd encoding part 44 are the same as that of embodiment demonstrated in FIGS. 1-10. For example, the first encoding unit 42 and the second encoding unit 44 are the same as the first encoding unit 42 and the second encoding unit 44 described in FIG.

  The stream synthesis unit 46A receives the compressed stream CST1 generated by the first encoding unit 42 and the compressed stream CST2 generated by the second encoding unit 44, and also receives the region-of-interest information CINF from the region-of-interest detection unit 32A. Then, the stream synthesis unit 46A converts the compressed streams CST1 and CST2 to H.264. It is synthesized in a format compliant with H.264 MVC. Further, the stream synthesizing unit 46A embeds position information of the region of interest ROI or the like (for example, region of interest information CINF) in the compressed stream STRM obtained by synthesizing the compressed streams CST1 and CST2.

  That is, the compressed stream STRM includes encoded data of the reduced image data RIMG, encoded data of the interest image data SIMG, and position information of the region of interest ROI. In this way, the stream synthesis unit 46A outputs the compressed stream CST1, CST2 and the position information of the region of interest ROI as one compressed stream STRM to the stream storage unit 100 shown in FIG.

  Note that the configuration of the moving image encoding device 10A is not limited to this example. For example, the moving image encoding device 10A may include the functions of the moving image encoding device 10 illustrated in FIG. 2 and execute the operations of the moving image encoding device 10 as necessary.

  FIG. 12 shows an example of the operation of the moving picture encoding apparatus 10A shown in FIG. The operation of FIG. 12 may be realized only by hardware, or may be realized by controlling the hardware by software. For example, software such as a moving image processing program causes a computer to execute the operation of FIG. In the operation of FIG. 12, processes S132 and S172 are executed instead of the processes S130 and S170 shown in FIG. Detailed description of the processing described in FIG. 8 is omitted.

  In process S132, the region-of-interest detection unit 32A detects the region of interest ROI using the input image IIMG acquired in process S100. For example, the region of interest detection unit 32 reads the input image data IIMG from the memory 50 and detects the region of interest ROI using a face detection algorithm or the like.

  In the process S172, the stream synthesizing unit 46A synthesizes the compressed stream CST1 generated in the process S120 and the compressed stream CST2 generated in the process S160, and generates a compressed stream STRM. Furthermore, the stream synthesizing unit 46A embeds the position information of the region of interest ROI detected in step S132 (for example, the region of interest information CINF) in the compressed stream STRM obtained by synthesizing the compressed streams CST1 and CST2. Accordingly, a compressed stream STRM including the position information of the region of interest ROI, the encoded data of the reduced image data RIMG, and the encoded data of the image of interest data SIMG is generated.

  Note that the operation of the moving image encoding device 10A is not limited to this example. For example, the interest image generation unit 30A performs the process of dividing the region of interest ROI into a plurality of region of interest images SROI, for example, the frame interval of the input image IIMG (for example, the interval from the frame time t0 to the frame time t1 shown in FIG. 6). It may be performed at larger intervals. That is, the region-of-interest detection unit 32A may perform detection of the region of interest ROI at an interval larger than the frame interval, for example. Further, the cutout unit 34 may execute the cutout process of the region of interest ROI at an interval larger than the frame interval. At this time, the data amount of the compressed stream CST1 including the encoded data of the interest image data SIMG can be reduced as compared with the case where the clipping process is executed for each frame.

  FIG. 13 shows an example of the stream configuration. Note that FIG. 2 illustrates an example of a configuration of a compressed stream STRM compliant with H.264 MVC. The compressed stream STRM has at least one sequence SEQ. The sequence SEQ has at least one access unit AU. The access unit includes various header information of AUD (Access Unit Delimiter), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), SEI (Supplemental Enhancement Information), and a view component VC.

  AUD, SPS, PPS, and SEI may be omitted. In this embodiment, the position information of the region of interest ROI is embedded in SEI as one of user_data SEI, for example. The position information of the region of interest ROI embedded in the SEI is, for example, the coordinates and size of the region of interest ROI in the input image IIMG and the coordinates of the region of interest SROI in the image of interest SIMG.

  Base view image data (encoded data of reduced image data RIMG) and non-base view image data (encoded data of image data of interest SIMG) are embedded in the view component VC, respectively. Each view component VC has a slice SL, for example. The slice SL has a plurality of macro blocks MB.

  FIG. 14 illustrates an example of a moving image reproduction device 110A corresponding to the moving image encoding device 10A illustrated in FIG. Elements similar to those described in FIGS. 1 to 10 are denoted by the same reference numerals, and detailed description thereof will be omitted. The moving image reproduction apparatus 110A of this embodiment includes an image decoding unit 120A and a display image generation unit 130A instead of the image decoding unit 120 and the display image generation unit 130 illustrated in FIG. Other configurations of the moving image playback device 110A are the same as those in the embodiment described with reference to FIGS.

  For example, the moving image reproduction device 110A includes an image decoding unit 120A, a display image generation unit 130A, and a memory 140. The image decoding unit 120A includes, for example, a stream separation unit 122A, a first decoding unit 124, and a second decoding unit 126.

  For example, the stream separation unit 122A receives the compressed stream STRM from the stream storage unit 100 illustrated in FIG. Then, the stream separation unit 122A separates the compressed stream STRM into a compressed stream CST1 for base view and a compressed stream CST2 for non-base view. The compressed stream CST1 has encoded data of reduced image data RIMG, and the compressed stream CST2 has encoded data of interest image data SIMG.

  Further, the stream separation unit 122A acquires the region of interest information CINF (position information of the region of interest ROI, etc.) from the compressed stream STRM. Then, the stream separation unit 122A outputs the compressed streams CST1 and CST2 to the first decoding unit 124 and the second decoding unit 126, respectively, and outputs the region-of-interest information CINF to the display image generation unit 130A. The 1st decoding part 124 and the 2nd decoding part 126 are the same as that of embodiment demonstrated in FIGS. For example, the first decoding unit 124 and the second decoding unit 126 are the same as the first decoding unit 124 and the second decoding unit 126 described in FIG.

  In the display image generation unit 130A, for example, the region of interest detection unit 132 is omitted from the display image generation unit 130 illustrated in FIG. For example, the display image generation unit 130A includes a cutout unit 134 and a display image composition unit 136. The cutout unit 134 and the image composition unit 136 are the same as those in the embodiment described with reference to FIGS. Note that the cutout unit 134 and the image composition unit 136 receive the region-of-interest information CINF from the stream separation unit 122A. Thereby, for example, when there is an instruction for fine display of the region of interest, the display image generation unit 130A can display the designated region of interest ROI with higher definition than the reduced image RIMG.

  Note that the configuration of the moving image playback device 110A is not limited to this example. For example, the moving image reproduction device 110A may include the functions of the moving image reproduction device 110 illustrated in FIG. 9 and execute the operation of the moving image reproduction device 110 as necessary.

  FIG. 15 shows an example of the operation of the moving image playback device 110A shown in FIG. The operation of FIG. 15 may be realized only by hardware, or may be realized by controlling the hardware by software. For example, software such as a moving image processing program causes a computer to execute the operation of FIG. In the operation of FIG. 15, processes S212 and S252 are executed instead of the processes S210 and S250 shown in FIG. Detailed description of the processing described in FIG. 10 is omitted.

  In process S212, the stream separation unit 122A separates the compressed stream STRM acquired in process S200 into a compressed stream CST1 for base view and a compressed stream CST2 for non-base view. Further, the stream separation unit 122A acquires the position information (region of interest information CINF) of the region of interest ROI from the compressed stream STRM.

  The process S252 is executed when there is a fine display request for the region of interest (Yes in the process S240). Note that the operation of the moving image playback device 110A when there is no fine display request for the region of interest (No in step S240) is the same as that of the moving image playback device 110.

  In step S252, the moving image playback device 110A processes the position information (region of interest information CINF) of the region of interest ROI acquired in step S212 as valid information. For example, the moving image playback device 110 may reflect a mark indicating the detected region of interest in the display image. Then, the moving image reproduction device 110A can display the designated region of interest ROI with higher definition than the reduced image RIMG by executing the processes S260 to S300. Note that step S252 may be omitted. For example, in the operation in which the process S252 is omitted, when there is a region of interest designation (Yes in the process S260), the position information (region of interest information CINF) of the region of interest ROI acquired in the process S212 is processed as valid information. S270 to S300 are executed.

  As described above, also in this embodiment, the same effect as that of the embodiment described with reference to FIGS. 1 to 10 can be obtained. Furthermore, in this embodiment, the moving image playback device 110A does not have to include the region of interest detection unit 132. For this reason, in this embodiment, the configuration of the moving image playback apparatus 110A can be simplified.

The invention described in the above embodiments is organized and disclosed as an appendix.
(Appendix 1)
A first image generation unit that generates first image data corresponding to a first image obtained by reducing an input image at a preset reduction rate;
The region of interest in the input image is thinned out at a thinning rate corresponding to the reduction rate and divided into a plurality of region of interest images, and second image data corresponding to a second image including at least one of the plurality of regions of interest images A second image generation unit for generating
When the first image data and the second image data are encoded to generate stream data, and the second image data is encoded, the first image corresponding to the time associated with the second image is added to the first image data. And a video encoding unit capable of performing prediction processing based on the video encoding unit.
(Appendix 2)
The second image generation unit allocates the plurality of region-of-interest images to the plurality of second images at a common time, and generates the second image data respectively corresponding to the plurality of second images. The moving image processing apparatus according to appendix 1.
(Appendix 3)
The second image generation unit allocates the plurality of region-of-interest images to the plurality of second images at different times, and generates the second image data corresponding to the plurality of second images, respectively. The moving image processing apparatus according to appendix 1.
(Appendix 4)
The second image generation unit allocates the plurality of region-of-interest images to the common second image, and generates the second image data corresponding to the second image to which the plurality of region-of-interest images are allocated. The moving image processing apparatus according to appendix 1, characterized by:
(Appendix 5)
The second image generation unit performs the process of dividing the region of interest into the plurality of region-of-interest images at intervals larger than the frame interval of the input image. The moving image processing apparatus according to the item.
(Appendix 6)
The second image generation unit detects the region of interest using local decoded image data generated when the first image data is encoded, and the region of interest in the input image is detected based on the detection result. The moving image processing apparatus according to any one of appendix 1 to appendix 5, wherein the position is calculated.
(Appendix 7)
In the moving image processing apparatus for decoding the stream data generated by the moving image processing apparatus according to attachment 6,
A detection unit configured to detect the region of interest by using the image data obtained by decoding the data obtained by encoding the first image data and detecting the region of interest in the region of interest detection method executed on the encoding side that generated the stream data; A moving image processing apparatus.
(Appendix 8)
The moving image processing apparatus according to any one of Supplementary Note 1 to Supplementary Note 5, wherein the image encoding unit includes position information indicating a position of the region of interest in the input image in the stream data. .
(Appendix 9)
In the moving image processing device for decoding the stream data generated by the moving image processing device according to attachment 8,
A moving image processing apparatus comprising: a third image generation unit that calculates a position of the region of interest based on the position information included in the stream data.
(Appendix 10)
A moving image processing method for encoding a moving image,
Generating first image data corresponding to a first image obtained by reducing the input image at a preset reduction rate;
The region of interest in the input image is thinned out at a thinning rate corresponding to the reduction rate and divided into a plurality of region of interest images, and second image data corresponding to a second image including at least one of the plurality of regions of interest images Produces
Encoding the first image data and the second image data to generate stream data;
The moving image processing method characterized in that the encoding processing for encoding the second image data can perform prediction processing based on the first image corresponding to the time associated with the second image. .
(Appendix 11)
11. The moving image according to claim 10, wherein the plurality of region-of-interest images are respectively assigned to the plurality of second images at a common time, and the second image data corresponding to the plurality of second images is generated. Processing method.
(Appendix 12)
11. The moving image according to claim 10, wherein the plurality of region-of-interest images are respectively assigned to the plurality of second images at different times, and the second image data corresponding to the plurality of second images is generated. Processing method.
(Appendix 13)
The additional image according to claim 10, wherein the plurality of region-of-interest images are allocated to the common second image, and the second image data corresponding to the second image to which the plurality of region-of-interest images are allocated is generated. Video processing method.
(Appendix 14)
The moving image processing method according to any one of appendix 10 to appendix 13, wherein the process of dividing the region of interest into the plurality of region-of-interest images is performed at an interval larger than a frame interval of the input image. .
(Appendix 15)
Detecting the region of interest using local decoded image data generated when the first image data is encoded, and calculating a position of the region of interest in the input image based on a detection result. 15. The moving image processing method according to any one of appendix 10 to appendix 14.
(Appendix 16)
In the moving image processing method for decoding the stream data generated by the moving image processing method according to attachment 15,
Using the image data obtained by decoding the data obtained by encoding the first image data, and detecting the region of interest by the region-of-interest detection method executed on the encoding side that generated the stream data. Video processing method.
(Appendix 17)
15. The moving image processing method according to claim 10, wherein position information indicating the position of the region of interest in the input image is included in the stream data.
(Appendix 18)
In the moving image processing method for decoding the stream data generated by the moving image processing method according to attachment 17,
A moving image processing method, comprising: calculating a position of the region of interest based on the position information included in the stream data.
(Appendix 19)
A moving image processing program to be executed by a computer,
A moving image processing program for causing the computer to operate as the moving image processing apparatus according to any one of appendix 1 to appendix 9.
(Appendix 20)
A first image generation process for generating first image data corresponding to a first image obtained by reducing an input image at a preset reduction rate;
The region of interest in the input image is thinned out at a thinning rate corresponding to the reduction rate and divided into a plurality of region of interest images, and second image data corresponding to a second image including at least one of the plurality of regions of interest images A second image generation process for generating
When the first image data and the second image data are encoded to generate stream data, and the second image data is encoded, the first image corresponding to the time associated with the second image is added to the first image data. A moving image processing program that causes a computer to execute an image encoding process capable of performing a prediction process based on the computer program.
(Appendix 21)
In the second image generation process, the plurality of region-of-interest images are respectively assigned to the plurality of second images at a common time, and the second image data corresponding to the plurality of second images is generated. The moving image processing program according to appendix 20.
(Appendix 22)
In the second image generation process, the plurality of region-of-interest images are respectively assigned to the plurality of second images at different times, and the second image data corresponding to the plurality of second images is generated. The moving image processing program according to appendix 20.
(Appendix 23)
In the second image generation processing, the plurality of region-of-interest images are allocated to the common second image, and the second image data corresponding to the second image to which the plurality of region-of-interest images are allocated is generated. The moving image processing program according to appendix 20, characterized by:
(Appendix 24)
Any one of Supplementary notes 20 to 23, wherein in the second image generation process, the process of dividing the region of interest into the plurality of region-of-interest images is performed at an interval larger than the frame interval of the input image. The moving image processing program according to the item.
(Appendix 25)
In the second image generation process, the region of interest is detected using local decoded image data generated when the first image data is encoded, and the region of interest in the input image is detected based on the detection result. The moving image processing program according to any one of supplementary notes 20 to 24, wherein the position is calculated.
(Appendix 26)
In the moving image processing program for decoding the stream data generated using the moving image processing program according to attachment 25,
Using the image data obtained by decoding the data obtained by encoding the first image data, a detection process for detecting the region of interest is performed on the computer by the region of interest detection method executed on the encoding side that generated the stream data. A moving image processing program that is executed.
(Appendix 27)
25. The moving image processing program according to any one of supplementary notes 20 to 24, wherein in the image encoding processing, position information indicating a position of the region of interest in the input image is included in the stream data. .
(Appendix 28)
In the moving image processing program for decoding the stream data generated using the moving image processing program according to attachment 27,
A moving image processing program for causing a computer to execute a third image generation process for calculating the position of the region of interest based on the position information included in the stream data.

  From the above detailed description, features and advantages of the embodiments will become apparent. This is intended to cover the features and advantages of the embodiments described above without departing from the spirit and scope of the claims. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and changes, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to rely on suitable improvements and equivalents within the scope disclosed in.

  DESCRIPTION OF SYMBOLS 10, 10A ... Moving image encoding apparatus; 20 ... Reduced image generation part; 30, 30A ... Interest image generation part; 32, 32A, 132 ... Interest area detection part; 34, 134 ... Extraction part; 40, 40A ... Image encoding unit; 42 ... First encoding unit; 44 ... Second encoding unit; 46, 46A ... Stream composition unit; 50, 140 ... Memory; 100 ... Stream storage unit; 120, 120A, image decoding unit; 122, 122A, stream separation unit; 124, first decoding unit; 126, second decoding unit; 130, 130A, display image generation unit; 136, display image synthesis unit; Video processing system

Claims (8)

A first image generation unit that generates first image data corresponding to a first image obtained by reducing an input image at a preset reduction rate;
The region of interest in the input image is thinned out at a thinning rate corresponding to the reduction rate and divided into a plurality of region of interest images, and second image data corresponding to a second image including at least one of the plurality of regions of interest images A second image generation unit for generating
When the first image data and the second image data are encoded to generate stream data, and the second image data is encoded, the first image corresponding to the time associated with the second image is added to the first image data. And a video encoding unit capable of performing prediction processing based on the video encoding unit. The second image generation unit allocates the plurality of region-of-interest images to the plurality of second images at a common time, and generates the second image data respectively corresponding to the plurality of second images. The moving image processing apparatus according to claim 1. The second image generation unit allocates the plurality of region-of-interest images to the plurality of second images at different times, and generates the second image data corresponding to the plurality of second images, respectively. The moving image processing apparatus according to claim 1. The said 2nd image generation part implements the process which divides | segments the said region of interest into these several region of interest image by the space | interval larger than the frame space | interval of the said input image. The moving image processing apparatus according to claim 1. The second image generation unit detects the region of interest using local decoded image data generated when the first image data is encoded, and the region of interest in the input image is detected based on the detection result. The moving image processing apparatus according to claim 1, wherein the position is calculated. The moving image processing apparatus for decoding the stream data generated by the moving image processing apparatus according to claim 5,
A detection unit configured to detect the region of interest by using the image data obtained by decoding the data obtained by encoding the first image data and detecting the region of interest in the region of interest detection method executed on the encoding side that generated the stream data; A moving image processing apparatus. A moving image processing method for encoding a moving image,
Generating first image data corresponding to a first image obtained by reducing the input image at a preset reduction rate;
The region of interest in the input image is thinned out at a thinning rate corresponding to the reduction rate and divided into a plurality of region of interest images, and second image data corresponding to a second image including at least one of the plurality of regions of interest images Produces
Encoding the first image data and the second image data to generate stream data;
The moving image processing method characterized in that the encoding processing for encoding the second image data can perform prediction processing based on the first image corresponding to the time associated with the second image. . A moving image processing program to be executed by a computer,
A moving image processing program causing the computer to operate as the moving image processing apparatus according to claim 1.

Images