JP2016171525A - Image processing device, control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, a control method, and a program which reduce a processing amount in the case of collectively distributing plural images as a single image.SOLUTION: The image processing device comprises: an acquisition part which acquires encode information for encoding plural input images, for the input images; an encoding part which uses the encode information acquired by the acquisition part to encode the input images as a single image including the input images; and a distribution part which distributes the single image encoded by the encoding part.SELECTED DRAWING: Figure 8

Description

  The present invention relates to an image processing apparatus, a control method, and a program.

  In recent years, with the widespread use of the Internet and the widening of access lines, a technique for distributing captured moving images via the Internet has attracted attention. For example, in the technique described in Patent Document 1, a plurality of types of moving image distribution are prepared by arranging a plurality of cameras, and a user can select a desired moving image. With this technology, the user can receive the selected moving image.

  In relation to the above technique, there is a technique in which a plurality of moving images are arranged in a tile to form one moving image and the moving image is distributed. In this case, a plurality of input moving images are first decoded, and the decoded moving images are combined into one moving image, and the combined moving image is encoded and distributed. As described above, in the related art, encoding is performed after a plurality of moving images are combined into one moving image.

JP2011-103522A

When combining multiple images into one, even if the combination of the images included in the combined image is the same or some of the images are in common, the order of arranging the images is changed to one When they are summarized, encoding is performed for each moving image.
In this case, although a common image is included, vector search and mode determination must be performed for each grouped image, and a combination of images is distributed for each distribution destination. In practice, it was difficult in terms of processing amount and cost.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing apparatus, a control method, and a program that reduce the processing amount when a plurality of images are delivered together as one image. There is to do.

  In order to solve the above-described problem, the present invention uses, for each of a plurality of input images, an acquisition unit that acquires encoding information for encoding the input image, and the encoding information acquired by the acquisition unit. And an encoding unit that encodes as one image including the plurality of input images, and a distribution unit that distributes the one image encoded by the encoding unit.

  As described above, according to the present invention, it is possible to provide an image processing device, a control method, and a program that reduce the amount of processing when a plurality of images are delivered together as one image.

It is a block diagram which shows the structure of the moving image delivery system as an image processing apparatus by this embodiment. It is a figure which shows the example of two types of synthesized images delivered from the encoding apparatus EN. It is a figure which shows the structural example of the decoding apparatus DE and the encoding apparatus EN. It is a figure which shows the structural example of the decoding apparatus DE in the case of outputting the image of several types of sizes. It is a figure which shows an example of the coordinate transformation in the case of combining in the offset process performed by the motion detection / mode determination unit. It is a figure which shows an example of the coordinate conversion in the case of superimposing and combining in the offset process performed by a motion detection and mode determination part. It is a flowchart which shows the process performed by the decoding apparatus DE. It is a flowchart which shows the process performed by the encoding apparatus EN. It is a flowchart which shows the process performed by the composite image controller. It is a block diagram which shows the modification of a structure of a moving image delivery system.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating a configuration of a moving image distribution system 100 as an image processing apparatus according to the present embodiment. As an example, the moving image distribution system 100 receives images from m terminals, and distributes one image including a plurality of input images (hereinafter referred to as “composite images”) to n terminals. It is.

As shown in FIG. It includes a plurality of decoding devices DE, a plurality of encoding devices EN, a composite image controller 110, and a directional database 120 compliant with H.264 (MPEG-4 Part 10: AVC).
M decoding devices DE are provided from the decoding device DE1 to the decoding device DEm, and can handle a maximum of m input images.
The decoding device DE first receives an input image as a bit stream. The decoding device DE decodes an input image input as a bit stream, determines a mode such as a motion vector or intra or inter from the decoded image, and uses it as encoding information. Then, the decoding device DE outputs the encoded information and the decoded image to the encoding device EN.
In FIG. 1, the decoding device DE and the encoding device EN are connected by a solid line and a broken line because the encoded information and the decoded image are output separately. In the present embodiment, the solid line indicates the output of the image, and the broken line indicates the output of the encoding information. A specific data transfer example will be described later.
Further, the decoding device DE outputs the image feature information and tag information (described later) output from the decoding device DE and an identifier (DE_ID) for identifying the decoding device DE to the composite image controller 110.

There are n encoding devices EN from the encoding device EN1 to the encoding device ENn, and a maximum of n synthesized images can be distributed.
The encoding device EN first outputs an identifier (user ID) for identifying a distribution destination (viewer) to distribute an image to the composite image controller 110, or distribution conditions (for example, only girls) When the image is output to the composite image controller 110 as necessary, specific information for specifying the decoding device DE that acquires the input image necessary for the composite image is acquired from the composite image controller 110. Arrangement information indicating the arrangement position where the input image is arranged is notified. Hereinafter, the specific information and the arrangement information are combined and expressed as composite information.

The encoding device EN distributes the composite image encoded as one image including a plurality of input images from the input image output from the decoding device DE, the encoding information, and the composite information to the distribution destination.
The composite image controller 110 notifies the encoding device EN of the composite information by referring to the directional database based on the above-described feature information, user ID, distribution conditions, and the like. The directional database 120 is a database that stores user information (for example, sex, age, hobbies, and other preferences) associated with a user ID, and feature information and tag information associated with an ED_ID.

  Through the processes described above, the encoding device EN can distribute different composite images for each distribution destination as shown in the composite images 1, 2, and 3. FIG. 2 is a diagram illustrating two types of composite images distributed from the encoding device EN.

  FIG. 2A is a diagram illustrating a composite image example in which a plurality of input images are arranged. As shown in the figure, the composite image is composed of regions divided into 5 × 5 tiles, and is a single image in which a plurality of input images A, B, C,. In the case of FIG. 2A, they are arranged in order of A, B, C from the upper left, but as shown in the composite image C in FIG. 1, they are arranged in the order of B, C,. It is also possible to distribute images in which the order of arrangement using images is changed. The number of input images to be arranged is not limited to 5 × 5, and may be 2 to 24, or 26 or more input images.

  FIG. 2B is a diagram illustrating an example of a composite image in which a plurality of input images are superimposed (overlapped). As shown in the figure, the composite image is an image in which the input images B and C are superimposed on the input image A. In addition to the composite image example shown in FIG. 2B, for example, an image obtained by superimposing a part or all of the input image C on the input image B can also be distributed. The number of input images to be superimposed is not limited to 3, and may be 2 or 4 or more.

Next, configuration examples of the decoding device DE and the encoding device EN will be described. FIG. 3 is a diagram illustrating a configuration example of the decoding device DE and the encoding device EN. First, the decoding device DE will be described. Note that the configuration of the decoding device DE shown in FIG. 3 shows a configuration when the size of the image output to the encoding device EN is one type.
The decoding device DE includes a decoder 10, a feature detection unit 11, an image processing unit 12, an image storage unit 13, an encode information acquisition unit 14, and an encode information storage unit 15.
The decoder 10 decodes the input image input as a bit stream, and outputs the decoded input image to the feature detection unit 11. The feature detection unit 11 analyzes the image, detects feature information of the image (for example, information indicating the feature of the image such as an outdoor image, a snow scene image, an image including a person), and the detected feature information is detected. In addition to outputting to the image composition controller 110, the input image is output to the image processing unit 12.

  The feature detection unit 11 also outputs tag information (for example, man, woman, sports, automobile, train, cooking, animation, karaoke, musical instrument performance, etc.) set in the image to the image composition controller 110. Therefore, for example, in the case of an image of a woman who is outdoors with snow, “snow” and “outdoor” are detected as the feature information and “woman” is set as the tag information. Is output.

The image processing unit 12 changes the size, brightness, and color of the image input from the feature detection unit 11, cuts out an object, cuts out a background, and various effects (for example, Image processing such as adding an effect by animation-like processing or skin filter processing is performed, and the input image subjected to the image processing is stored in the image storage unit 13.
For example, in the case of a 5 × 5 composite image (1920 × 1200) as shown in FIG. 2A, the image processing unit 12 sets the size of the input image to 384 × 240.

The encoding information acquisition unit 14 encodes the input image by performing motion vector search, motion compensation, weighted prediction, intra prediction, mode determination, and the like for the input image stored in the image storage unit 13. Encoding information for this is acquired, and the acquired encoding information is stored in the encoding information storage unit 15. The encoding information is vector information or mode information, and more specifically, information indicating a mode such as a motion vector, intra / inter, or direction prediction.
Thus, the encoding information acquisition unit 14 performs in advance processing that occupies most of the processing load among the processing required for encoding. With the encoding information acquired in this manner, the encoding apparatus EN can perform encoding after performing processing relating to the arrangement of input images to be collected, and the like, by simply performing processing with a relatively small processing load such as frequency conversion and quantization.

  The decoding device DE outputs the input image stored in the image storage unit 13 and the encoding information stored in the encoding information storage unit 15 to the encoding device EN that requires the input image. As described above, a generation unit (corresponding to an encoding information acquisition unit described later) that generates encoding information is provided for each of a plurality of input images.

  The encoding apparatus EN includes a map synthesis unit 20, a motion detection / mode determination unit 21, a frequency conversion / quantization unit 22, and a syntax conversion unit 23. The map composition unit 20 outputs the user ID and distribution conditions to the composite image controller 110, receives the composite information notified from the composite image controller 110, and obtains the encode information for each input image from the encode information storage unit 15. The input image and the encoding information corresponding to the input image are associated with each other and output to the motion detection / mode determination unit 21. Thus, the map composition unit 20 is an acquisition unit that acquires encoding information for encoding the input image for each of the plurality of input images. The map composition unit 20 is a designation unit for designating the conditions of the input image included in one image.

The motion detection / mode determination unit 21 performs synthesis processing by performing offset processing for converting the input image and the encoding information input from the map synthesis unit 20 into encoding information corresponding to the arrangement position indicated in the synthesis information. A motion detection / mode determination process for determining image encoding information is performed. The motion detection / mode determination unit 21 then outputs the determined encoding information to the frequency conversion / quantization unit 22.
The frequency conversion / quantization unit 22 performs frequency conversion and quantization on the encoded information input from the motion detection / mode determination unit 21 and outputs it to the syntax conversion unit 23. The syntax conversion unit 23 synthesizes the encoded information as a bit stream. Deliver images.

  The motion detection / mode determination unit 21 and the frequency conversion / quantization unit 22 described above are encoding units that encode as a single image including a plurality of input images using the encoding information acquired by the map synthesis unit 20. Yes. The syntax conversion unit 23 is a distribution unit that distributes one encoded image.

Here, an example of data transfer from the decoding device DE to the encoding device EN will be described. In this embodiment, data is transferred by line. Specifically, the decoding device DE outputs an image to the encoding device EN via the line memory. For example, when distributing the composite image 1 shown in FIG. 1, the encoding device EN receives the first line of the input image 1 (A), and subsequently receives the first line of the input image 2 (B). By receiving the data of the first line of the input image 3 (C), the line buffer stores one line of data of the input images 1, 2, and 3 (ABC).
By repeating this line transfer, when a line of codec processing units (for example, macroblock 16 × 16) is stored in the line buffer, the macroblock of input image 1 (A) can be processed. Therefore, since the encoding device EN can receive the encoding information from the decoder device DE and can encode the encoder device EN without providing a frame memory, the amount of memory is reduced compared to the case where a frame memory is provided. can do.

Next, the configuration of the decoding device DE when outputting images of a plurality of types of sizes will be described. FIG. 4 is a diagram illustrating a configuration example of the decoding device DE when outputting images of a plurality of types of sizes. Here, a configuration different from FIG. 3 will be described. In addition, the configuration example illustrated in FIG. 4 illustrates a configuration in the case of outputting three types of images (L size, M size, and S size).
When outputting images of a plurality of types of sizes, an image storage unit 13, an encode information acquisition unit 14, and an encode information storage unit 15 are provided for each type. In the case of FIG. 4, since there are three types, the image storage units 13-1, 13-2, and 13-3, the encoding information acquisition units 14-1, 14-2, and 14-3, and the encoding information storage unit 15-1 15-2 and 15-3.

  The image storage units 13-1, 13-2, and 13-3 store L size, M size, and S size images, respectively. The encoding information acquisition units 14-1, 14-2, and 14-3 acquire L size, M size, and S size encoding information, respectively. The encoding information storage units 15-1, 15-2, and 15-3 store L size, M size, and S size encoding information, respectively. Further, the encode information storage unit 15-3 that stores the S-size encode information outputs the stored encode information to the encode information storage unit 15-2 that stores the M-size encode information. The encode information storage unit 15-2 that stores M size encode information outputs the stored encode information to the encode information storage unit 15-1 that stores L size encode information.

  4 is provided with a filter for changing the size of the image output from the image processing unit 12 (in the case of FIG. 4, reduction). In the case of the configuration shown in FIG. 4, since the L size is the size of the image output from the image processing unit 12, no filter is provided between the image processing unit 12 and the image storage unit 13-1. Reduction filters F1 and F2 are provided between the image processing unit 12 and the image storage unit 13-2 and between the image processing unit 12 and the image storage unit 13-3, respectively.

The reduction filter F1 is a filter that reduces the image output from the image processing unit 12 to M size. The reduction filter F2 is a filter that reduces the image output from the image processing unit 12 to an S size.
Also, the encoding information acquisition units 14-1, 14-2, and 14-3 search for motion vectors and perform motions on the images stored in the corresponding image storage units 13-1, 13-2, and 13-3. Compensation, weighted prediction, intra prediction, mode determination, and the like may be performed independently. However, in this embodiment, the encoding information acquisition units 14-3, 14-2, and 14-1 are arranged in the order (smallest size). ) Start processing.

  This is because the images for which the encoding information acquisition units 14-1, 14-2, and 14-3 acquire the encoding information are only different in size. This is because, by starting the processing on the image, the encoding information obtained by the processing can be reused to further increase the speed.

Therefore, in this embodiment, the encoding information obtained by the processing of the encoding information acquisition unit 14-3 is output to the encoding information acquisition unit 14-2, and is reused by the encoding information acquisition unit 14-2. The encoding information obtained by the processing of the unit 14-2 is output to the encoding information acquisition unit 14-1, and is reused by the encoding information acquisition unit 14-1. By doing so, for example, the circuit size when the decoding device DE is realized by hardware can be suppressed, and the bandwidth of the memory can be reduced. Furthermore, quantization parameters and the like may be individually set according to the size.
By outputting images of a plurality of types in this way, it is possible to simultaneously distribute images of the same image but different sizes, such as the composite image 1 and the composite image 3 shown in FIG. For example, in the case of the composite image 2 in FIG. 1, the encoding device EN2 distributes the composite image 2 by acquiring an L-size image A, an M-size image B, and an S-size image C.

In addition to the configuration shown in FIG. 4, a configuration in which a reduction filter is provided inside the encode information acquisition unit 14 may be used as a configuration when outputting images of a plurality of types of sizes. In this case, first, the image is reduced to S size, and encoding information of the image reduced to S size is acquired. Next, the image is reduced to the M size, and the encoding information acquired at the S size is reused to acquire the encoding information of the image reduced to the M size. Then, the encoding information acquired in the M size is reused to acquire the encoding information of the L size image. At this time, the circuit area can be reduced by providing a line memory for storing the encoded information and reusing the encoded information stored in the line memory.
The configuration shown in FIG. 4 is a configuration for outputting images of a plurality of types, but it is also possible to output a plurality of types of images subjected to different image processing. For example, when outputting three types of images subjected to three different types of image processing, image processing units that perform different image processing corresponding to each of the image storage units 13-1, 13-2, and 13-3 are provided. The image processing unit outputs an image for each corresponding image storage unit.

As described above, when outputting a plurality of types of images or outputting a plurality of types of images subjected to different image processing, the output size and the type of image processing are output to the composite image controller 110. To do. When outputting images of a plurality of types, the decoding device DE outputs “1.L size”, “2.M size”, and “3.S size” to the composite image controller 110 together with ED_ID. Similarly, when outputting a plurality of types of images subjected to different image processing, the decoding device DE outputs “1. animation style”, “2. beautiful skin”, and “3. illustration style” together with ED_ID. To 110.
The composite image controller 110 stores these size and image processing information in the directional database 120 in association with the ED_ID. Then, the composite image controller 110 notifies the encoding device EN of the composite information by referring to the directional database 120 based on the above-described feature information, user ID, distribution conditions, and the like.

FIG. 5 is a diagram illustrating an example of coordinate transformation in the case of side-by-side synthesis in the offset processing executed by the motion detection / mode determination unit 21. In this example, it is assumed that the arrangement information of the composite information indicates that the input image is arranged at the nth position with the number of areas m × m. As shown in FIG. 5, the n-th indicates the order in which the input images are arranged horizontally from the upper left, return to the left end when reaching the right end, and are arranged again from the left end.
Also, the size of the composite image is K × L, and K = mk and L = ml. Since the size of the input image is adjusted by the image processing unit 12 according to the composite image, the size of the input image is k × l.

Here, the point at the ST coordinate in the input image is (s, t), the point at the XY coordinate in the composite image corresponding to (s, t) is (x, y), and q and r are n = Assuming q and r satisfying mq + r (0 ≦ r <m), (x, y) = (s + kr, t + L−1 (q + 1)).
As shown in this equation, the image composition controller 110 only has to notify m indicating the number m × m of input images and n indicating the order regarding the arrangement position, and the motion detection / mode determination unit 21 sets m And n can be used to perform coordinate transformation.

FIG. 6 is a diagram illustrating an example of coordinate conversion in the case of superimposing and synthesizing in the offset processing executed by the motion detection / mode determination unit 21. In this example, it is assumed that the arrangement information of the composite information indicates the position (x1, y1) in the XY coordinates where the origin O in the ST coordinates of the input image is arranged.
Here, when the point at the ST coordinate in the input image is (s, t) and the point at the XY coordinate in the composite image corresponding to (s, t) is (x, y), (x, y) = (S + x1, t + y1).

As shown in this equation, the image composition controller 110 only has to notify the position (x1, y1) in the XY coordinates where the origin O in the ST coordinate of the input image is arranged, and the motion detection / mode determination unit 21 Coordinate conversion can be performed using the position (x1, y1).
The coordinate conversion method is not limited to the method described with reference to FIGS. 5 and 6 described above. When the images are combined side by side, the number of input images to be combined from the beginning, such as 5 × 5, is fixed and converted. In the case of superimposing, there is a method in which the position to be superimposed is fixedly determined from the beginning and converted.

FIG. 7 is a flowchart showing processing executed by the decoding device DE. In FIG. 7, first, the decoder 10 decodes an input image input as a bit stream (step S101). Next, the feature detection unit 11 detects feature information and outputs the detected feature information to the image composition controller 110 (step S102).
Next, the image processing unit 12 performs the above-described image processing (step S193), and stores the input image subjected to the image processing in the image storage unit 13 (step S104). The encoding information acquisition unit 14 acquires the encoding information by performing the above-described motion vector search for the input image stored in the image storage unit 13 (step S105), and converts the acquired encoding information into the encoding information. It memorize | stores in the memory | storage part 15 (step S106).

  Then, the input image and the encoding information are output to the encoding device EN (step S107), and it is determined whether or not to end the decoding (step S108). If the decoding is not completed (step S108: NO), the process returns to step S101. On the other hand, when decoding is terminated when a transmission stop is received from the transmission source of the bitstream (step S108; YES), this processing is terminated.

FIG. 8 is a flowchart showing processing executed by the encoding apparatus EN. In FIG. 8, when a distribution request is received from a distribution destination (step S201: YES), the distribution destination user ID is acquired (step S202).
Furthermore, it is determined whether or not there is a delivery condition when the delivery condition is notified from the delivery destination (step S203). When the distribution condition is designated (step S203: YES), the distribution condition is acquired (step S204), and the process proceeds to step S205.
On the other hand, when the distribution condition is not specified (step S203: YES) or when the distribution condition is acquired in step S204, the map composition unit 20 outputs the user ID to the composite image controller 110 (step S205). . At this time, if the distribution condition is acquired in step S204, the distribution condition is also output.

The map composition unit 20 receives composition information from the composition image controller 110 (step S206), and acquires an input image and encoding information corresponding to the input image according to the arrangement information (step S207).
Next, the motion detection / mode determination unit 21 performs the motion detection / mode determination process described above (step S208), and the frequency conversion / quantization unit 22 performs frequency conversion and quantization of the encoded information (step S209), and syntax. The conversion unit 23 distributes the composite image as a bit stream (step S210).

  Next, it is determined whether or not a distribution stop has been received from the distribution destination (step S211). When the distribution stop is received from the distribution destination (step S211: YES), this process is terminated. On the other hand, when the distribution stop has not been received from the distribution destination (step S211: NO), the process returns to step S207.

FIG. 9 is a flowchart showing processing executed by the composite image controller 110. First, the composite image controller 110 determines whether or not feature information and meta information have been received from the decoding device DE (step S301). When the feature information and meta information are received (step S301: YES), the composite image controller 110 stores the feature information and meta information in the directional database 120 in association with the ED_ID (step S311), and returns to step S301. .
If feature information and meta information have not been received (step S301: NO), the composite image controller 110 determines whether a user ID has been received from the encoding device EN (step S302). If the user ID has not been received (step S302: NO), the process returns to step S301.

When the user ID is received from the encoding device EN (step S302: YES), the composite image controller 110 determines whether there is a distribution condition (step S303). If there is a distribution condition (step S303: YES), the composite image controller 110 generates specific information based on the distribution condition (step S309).
Specifically, the composite image controller 110 first searches for a decoding device DE that outputs an image whose feature information or meta information satisfies a distribution condition. Then, among the searched decoding devices DE, the decoding devices DE are specified in descending order of the feature information or meta information satisfying the distribution conditions by the number of input images required for the composite image (for example, 25 for 5 × 5), A list of ED_IDs of the specified decoding device DE is generated as specific information.

Next, the composite image controller 110 generates arrangement information. Specifically, the composite image controller 110 generates arrangement information such that an image with more feature information or meta information satisfying the distribution condition is more conspicuous (step S310), and the process proceeds to step S308.
For example, in the case of a 5 × 5 composite image, the placement information is generated so that the image having the largest feature information or meta information satisfying the delivery condition is placed in the center image. In this way, arrangement information may be generated such that the more characteristic information or meta information that satisfies the distribution condition, the closer to the center.

Returning to step S303, if there is no distribution condition (step S303: NO), the composite image controller 110 refers to the directional database 120 and acquires user information from the user ID (step S304).
Next, the composite image controller 110 calculates a score for each encoding device EN by performing principal component analysis using the feature information, meta information, and user information stored in the directional database 120 (step S305).
Then, the composite image controller 110 generates specific information based on the score (step S306).

Specifically, the synthesized image controller 110 identifies the decoding device DE in descending order of the number of input images necessary for the synthesized image among the decoding devices DE, and identifies the list of ED_IDs of the identified decoding device DE. Generate as information.
Next, the composite image controller 110 generates arrangement information. Specifically, similarly to the arrangement condition, the composite image controller 110 generates arrangement information such that an image with a large score is conspicuous (step S307), and the process proceeds to step S308.
For example, in the case of a 5 × 5 composite image, the arrangement information is generated so that the image with the highest score is arranged in the center image, as in the arrangement condition. Thus, arrangement information may be generated such that the larger the score, the closer to the center.
Then, the composite image controller 110 notifies the encoding information to the encoding device EN (step S308), and ends this processing.

  In this way, by using distribution conditions and principal component analysis, for example, when the viewer is a woman, a composite image obtained by collecting and synthesizing male images is distributed, or when the viewer likes anime, images related to anime If the viewer likes a game, or if the viewer likes a game, or if the viewer likes skiing, the snow scene or other image related to snow The synthesized composite image can be distributed.

As described above, according to the present embodiment, the encoding information for encoding the input image is obtained for each of the plurality of input images, and the plurality of input images are included using the acquired encoding information. Since encoding is performed as one image, the processing amount can be significantly reduced as compared with the case of encoding after combining the images into one.
In addition, even when the combination and arrangement of images to be distributed are frequently changed, the arrangement is simply changed and there is no need to search for motion vectors that occupy most of the processing load. Compared with the case of encoding after combining them into one, the processing amount can be significantly reduced.

  Further, in the present embodiment, the configuration example of the moving image distribution system 100 illustrated in FIG. 1 has a configuration in which the decoding device DE and the encoding device EN are directly connected, but with the increase in the decoding device DE or the encoding device EN, Wiring will increase. Therefore, the decoding device DE and the encoding device EN may be connected via a path switch.

FIG. 10 is a block diagram illustrating a modified example of the configuration of the moving image distribution system 100. In FIG. 10, the moving image distribution system 100 has a configuration in which a route switch SW is provided. The path switch SW is a PCIe (PCI-Express) switch, and connects the decoding device DE and the encoding device EN by Ethernet (registered trademark) Over PCIe.
According to the configuration shown in FIG. 10, the number of wirings can be reduced and the degree of freedom of the wiring can be increased as compared with the configuration shown in FIG.

  Note that, in the example shown in FIG. 2A in which the images are arranged side by side, all the input images are collected in the same size, but may be different sizes. For example, the larger the distribution condition or the larger the principal component analysis score, the larger the display may be.

  Further, as described above, the moving image distribution system 100 as the image processing apparatus according to the present embodiment is roughly composed of the decoding device DE and the encoding device EN, but as an example of the operation mode, like a blade server. There is an operation mode in which the decoding device DE and the encoding device EN are mounted on one blade and the blade is mounted in a casing.

Further, there is an operation mode in which the decoding device DE and the encoding device EN are different devices, and the decoding device DE and the encoding device EN are connected by a network having an intelligent switch. In this case, instead of the above-described line transfer, for example, when transfer is performed for each frame, the encoding device EN is provided with a memory capable of storing a frame, not a line buffer.
Also, by providing a device that operates as a composite image controller on the network, it is possible to realize distribution of the composite image based on the distribution conditions and scores described in FIG. Furthermore, in order to efficiently transfer data, the Ethernet switch and the composite image controller 110 may be combined to efficiently control the information transmission path.

  Further, there is an operation mode in which the decoding device DE and the encoding device EN are implemented by software. In particular, since the processing load of the encoding device EN is smaller than that of the decoding device DE, there is an operation mode in which the decoding device DE is implemented by hardware and the encoding device EN is implemented by software.

  Note that the decoding device DE performs motion vector search, motion compensation, weighted prediction, intra prediction, mode determination, and the like in order to obtain encoding information, and some of these processes are executed by the encoding device EN. You may make it do.

The moving image distribution system 100 according to the present embodiment is very effective when the image size is a multiple of 16 (for example, when the image size is K × L, both K and L are multiples of 16). Even if it is not a multiple of 16, the motion detection / mode determination unit 21 determines a motion vector re-search or a mode such as intra or inter based on the offset-encoded encoding information. In this case, the processing amount is not reduced as much as when the image size is a multiple of 16, but the processing amount can be considerably reduced as compared with the case where there is no encoding information.
It should be noted that the motion vector re-search by the motion detection / mode determination unit 21 and the determination of the mode such as intra or inter perform prediction in intra prediction even when the image size is a multiple of 16 or not. For the purpose of mode optimization and motion vector reduction, it may be executed only when encoding information can be used effectively.

  In addition, as described above, the moving image distribution system 100 according to the present embodiment is the H.264 standard. H.264, but H.264 MPEG2 (H.222 / H.262, ISO / IEC 13818), which performs the same processing as H.264, It is also applicable to H.265 (ISO / IEC 23008-2 HEVC). Specifically, MPEG2 or H.264 is used. In 265, the encoding information acquisition unit 14 obtains encoding information in advance by performing processing that occupies most of the processing load among processing necessary for encoding such as motion vector search, and the encoding device EN acquires encoding information. Encoding is performed using the encoding information acquired by the unit 14.

A processing amount reduction example according to the present embodiment will be described using a specific example. For example, when five input images are simply arranged side by side, there are 120 ways to arrange them. In the prior art, a search for a motion vector or the like has to be performed for each of the 120 synthesized images. On the other hand, according to the present embodiment, if a motion vector search or the like is performed for each of the five input images, only processing with a small processing load is required.
Thus, even if only the number of executions of motion vector search is taken into account, the number of times in the conventional technique is 120, but in the present embodiment, it is only 5 (and the image is a small size). Thus, it can be seen that the processing amount can be remarkably reduced.
More generally, the number of motion vector searches when n input images are simply arranged side by side is n! However, according to the present embodiment, since the order is n (O (n)), the processing amount can be remarkably reduced. Therefore, when processing is performed on 5 × 5 input images as shown in FIG. 2A, the reduction amount of the processing is enormous.

  When the processing of each device in the above-described embodiment is realized by a computer, a program for realizing this function is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by a computer system. , May be realized by executing. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. You may implement | achieve using programmable logic devices, such as FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

100 video distribution system, DE, DE1-DEm decoding device,
EN, EN1-ENn Encoding device, 10 Decoder, 11 Feature detection unit, 12 Image processing unit, 13, 13-1, 13-2, 13-3 Image storage unit,
14, 14-1, 14-2, 14-3 Encoding information acquisition unit 15, 15-1, 15-2, 15-3 Encoding information storage unit,
20 map synthesis unit, 21 motion detection / mode determination unit, 22 frequency conversion / quantization unit, 23 syntax conversion unit, F1, F2 reduction filter

The encoding device EN distributes the composite image encoded as one image including a plurality of input images from the input image output from the decoding device DE, the encoding information, and the composite information to the distribution destination.
The composite image controller 110 notifies the encoding device EN of the composite information by referring to the directional database based on the above-described feature information, user ID, distribution conditions, and the like. The directional database 120 is a database in which user information (for example, gender, age, hobbies, and other preferences) of a delivery destination is associated with the user ID, and feature information and tag information are stored in association with the DE_ID .

FIG. 2A is a diagram illustrating a composite image example in which a plurality of input images are arranged. As shown in the figure, the composite image is composed of regions divided into 5 × 5 tiles, and is a single image in which a plurality of input images A, B, C,. In the case of FIG. 2A, they are arranged in order of A, B, C from the upper left, but as shown in the composite image 3 of FIG. 1, they are arranged in the order of B, C,. It is also possible to distribute images in which the order of arrangement using images is changed. The number of input images to be arranged is not limited to 5 × 5, and may be 2 to 24, or 26 or more input images.

The image processing unit 12, the input image inputted from the feature detection unit 11, the size of the image, or change the brightness, and color, etc., and objects of the cut, or perform excision of background, various Effects (e.g. performs image processing such or to add Effects) due animation like processing and skin beautifying filtering, stores the input image subjected to image processing in the image storage unit 13.
For example, in the case of a 5 × 5 composite image (1920 × 1200) as shown in FIG. 2A, the image processing unit 12 sets the size of the input image to 384 × 240.

4 is provided with a filter that changes the size of the image output from the image processing unit 12 (in the case of FIG. 4, it is reduced). In the case of the configuration shown in FIG. 4, since the L size is the size of the image output from the image processing unit 12, no filter is provided between the image processing unit 12 and the image storage unit 13-1. Reduction filters F1 and F2 are provided between the image processing unit 12 and the image storage unit 13-2 and between the image processing unit 12 and the image storage unit 13-3, respectively.

Therefore, in this embodiment, the encoding information obtained by the processing of the encoding information acquisition unit 14-3 is output to the encoding information acquisition unit 14-2, and is reused by the encoding information acquisition unit 14-2. The encoding information obtained by the processing of the unit 14-2 is output to the encoding information acquisition unit 14-1, and is reused by the encoding information acquisition unit 14-1. By doing so, for example, the circuit size when the decoding device DE is realized by hardware can be suppressed, and the bandwidth of the memory can be reduced. Furthermore, quantization parameters and the like may be individually set according to the size.
By outputting images of a plurality of types in this way, it is possible to simultaneously distribute images of the same image but different sizes, such as the composite image 1 and the composite image 2 shown in FIG. For example, in the case of the composite image 2 in FIG. 1, the encoding device EN2 distributes the composite image 2 by acquiring an L-size image A, an M-size image B, and an S-size image C.

As described above, when outputting a plurality of types of images or outputting a plurality of types of images subjected to different image processing, the output size and the type of image processing are output to the composite image controller 110. To do. When outputting an image of a plurality of sizes, the decoding device DE, together with DE_ID, outputs "1.L size", "2.M size", the "3.S size" in the composite image controller 110. Similarly, when outputting a plurality of types of images subjected to different image processing, the decoding device DE, together with DE_ID , generates “1. Animation style”, “2. Beautiful skin”, and “3. Illustration style” as a composite image controller. To 110.
The composite image controller 110 stores these size and image processing information in the directional database 120 in association with the DE_ID . Then, the composite image controller 110 notifies the encoding device EN of the composite information by referring to the directional database 120 based on the above-described feature information, user ID, distribution conditions, and the like.

FIG. 7 is a flowchart showing processing executed by the decoding device DE. In FIG. 7, first, the decoder 10 decodes an input image input as a bit stream (step S101). Next, the feature detection unit 11 detects feature information and outputs the detected feature information to the image composition controller 110 (step S102).
Next, the image processing unit 12 performs the above-described image processing (step S1 0 3), and stores the input image subjected to the image processing in the image storage unit 13 (step S104). The encoding information acquisition unit 14 acquires the encoding information by performing the above-described motion vector search for the input image stored in the image storage unit 13 (step S105), and converts the acquired encoding information into the encoding information. It memorize | stores in the memory | storage part 15 (step S106).

FIG. 8 is a flowchart showing processing executed by the encoding apparatus EN. In FIG. 8, when a distribution request is received from a distribution destination (step S201: YES), the distribution destination user ID is acquired (step S202).
Furthermore, it is determined whether or not there is a delivery condition when the delivery condition is notified from the delivery destination (step S203). When the distribution condition is designated (step S203: YES), the distribution condition is acquired (step S204), and the process proceeds to step S205.
On the other hand, when the distribution condition is not specified (step S203: NO ), or when the distribution condition is acquired in step S204, the map composition unit 20 outputs the user ID to the composite image controller 110 (step S205). . At this time, if the distribution condition is acquired in step S204, the distribution condition is also output.

FIG. 9 is a flowchart showing processing executed by the composite image controller 110. First, the composite image controller 110 determines whether or not feature information and meta information have been received from the decoding device DE (step S301). When the feature information and meta information are received (step S301: YES), the composite image controller 110 stores the feature information and meta information in the directional database 120 in association with the DE_ID (step S311), and returns to step S301. .
If feature information and meta information have not been received (step S301: NO), the composite image controller 110 determines whether a user ID has been received from the encoding device EN (step S302). If the user ID has not been received (step S302: NO), the process returns to step S301.

When the user ID is received from the encoding device EN (step S302: YES), the composite image controller 110 determines whether there is a distribution condition (step S303). If there is a distribution condition (step S303: YES), the composite image controller 110 generates specific information based on the distribution condition (step S309).
Specifically, the composite image controller 110 first searches for a decoding device DE that outputs an image whose feature information or meta information satisfies a distribution condition. Then, among the searched decoding devices DE, the decoding devices DE are specified in descending order of the feature information or meta information satisfying the distribution conditions by the number of input images required for the composite image (for example, 25 for 5 × 5), A list of DE_IDs of the specified decoding device DE is generated as specific information.

Specifically, the synthesized image controller 110 identifies the decoding device DE in descending order of the number of input images necessary for the synthesized image among the decoding devices DE, and identifies the DE_ID list of the identified decoding device DE. Generate as information.
Next, the composite image controller 110 generates arrangement information. Specifically, similarly to the arrangement condition, the composite image controller 110 generates arrangement information such that an image with a large score is conspicuous (step S307), and the process proceeds to step S308.
For example, in the case of a 5 × 5 composite image, the arrangement information is generated so that the image with the highest score is arranged in the center image, as in the arrangement condition. Thus, arrangement information may be generated such that the larger the score, the closer to the center.
Then, the composite image controller 110 notifies the encoding information to the encoding device EN (step S308), and ends this processing.

Claims (9)

For each of a plurality of input images, an acquisition unit that acquires encoding information for encoding the input image;
An encoding unit that encodes as one image including the plurality of input images using the encoding information acquired by the acquisition unit;
An image processing apparatus comprising: a distribution unit that distributes the one image encoded by the encoding unit. A generation unit that generates the encoding information is provided for each of the plurality of input images,
The image processing apparatus according to claim 1, wherein the acquisition unit acquires the encoding information generated by the generation unit.   The image processing apparatus according to claim 1, wherein the generation unit generates vector information or mode information for encoding the input image as the encoding information. A designating unit for designating a condition of the input image included in the one image;
4. The image processing apparatus according to claim 1, wherein the encoding unit encodes the input image satisfying a condition specified by the specifying unit as one image. 5.   The image processing apparatus according to claim 4, wherein the input image satisfying more of the conditions is arranged at the center of the one image.   The image processing apparatus according to claim 1, wherein the one image is an image in which the plurality of input images are arranged.   The image processing apparatus according to claim 1, wherein the one image is an image obtained by superimposing the plurality of input images. A control method in an image processing apparatus that distributes a plurality of input images as one image,
For each of the plurality of input images, an obtaining step for obtaining encoding information for encoding the input image;
An encoding step of encoding as one image including the plurality of input images using the encoding information acquired in the acquisition step;
A distribution step of distributing the one image encoded by the encoding step. A program for causing a plurality of input images to function as an image processing device that distributes as a single image,
For each of the plurality of input images, an obtaining step for obtaining encoding information for encoding the input image;
An encoding step of encoding as one image including the plurality of input images using the encoding information acquired in the acquisition step;
A distribution step of distributing the one image encoded by the encoding step.

Images