JP6387511B2 - Image data processing method - Google Patents

Description

  The present invention relates to an image data processing method, and more particularly to an image data processing method in the case of using an image processing apparatus that displays a large number of small-resolution moving images while performing decoding processing by a hardware decoder.

  In recent video playback processing in various information devices, the encoded video information that has been encoded and stored in advance from the viewpoint of capacity is read or downloaded via the net, and it is encoded. A video is reproduced on a liquid crystal device or the like by decoding the data by the decoding process and supplying the data to the display device. The encoding process can be processed by software because there is a time margin, but in the decoding process, it is hard to ensure that even a high-definition video is reproduced without stagnation. It is also common to provide a wear circuit, particularly a dedicated image processor.

  In other words, the higher-level processing control unit only gives processing instructions to the dedicated image processing processor, and then the image processing processor autonomously performs image processing, thereby improving the efficiency of image processing. It is the structure which aims at.

An outline of the above concept is shown in FIG.
Here, the image data ROM 200 stores, for example, one or more pieces of moving image information encoded based on the H.264 (MPEG 4 / AVC) standard. Here, the case where the storage device for storing the encoded video information is provided in the information device main body is shown, but the same applies to a device downloaded from a server on the network. .

  Here, the H.264 standard will be briefly described. In MPEG 2 and MPEG 4, the syntax hierarchy is defined, and the information arranged in the bitstream is in line with the hierarchy, whereas in H.264, the parameter set and the slice that references it are There are few restrictions on arrangement compared to MPEG 2 and the like. The parameter set includes SPS (Sequence Parameter Set) for storing information related to encoding of the entire sequence, PPS (Picture Parameter Set) indicating the encoding mode of the entire picture, and arbitrary information. SEI (Supplemental Enhancement Information) that can be encoded, and SPS and PPS can be arranged for each frame, so that a plurality of sequences can be handled in one bit stream.

  FIG. 12 is a diagram illustrating a general bitstream of encoded data of an image encoded based on the H.264 standard. Here, for convenience of explanation, the set of SPS and PPS is shown as one set. In the H.264 standard, a layer called NAL (Network Abstraction Layer) separated from VCL (Video Coding Layer) is defined, and encoded data and parameters generated by VCL are defined. The set can be handled as a NAL unit, and encoded data is handled as a VCL NAL unit, and SPS and PPS are handled as a non-VCL NAL unit.

  Here, each slice has a header area, in which at least information of the first macroblock coordinate (first_mb_in_slice) of the own slice is stored. In addition, as in MPEG 2 and MPEG 4, in H.264, one picture is generally composed of one or more slices. However, when one moving image is actually encoded, one frame is encoded as one picture composed of one slice. A technique for obtaining a certain effect and encoding one frame as a plurality of slices, that is, dividing one image into a plurality of slices and encoding it as a plurality of slices, is disclosed in, for example, Japanese Patent Application Laid-Open No. 2004-122867.

  However, in the conventional technology represented by Patent Document 1, each image of one moving image is divided into a plurality of pieces, and each of the plurality of moving images is divided as the feature of the present invention described later. There is no conventional concept of coding and configuring them as each slice.

  Returning to FIG. 11, the host CPU 100 issues a processing command generally called an API (Application Program Interface) to the image processing processor 300. For example, if the operating system (OS) from which the device operates is Windows, it is an API called DXVA2 (DirectX Video Acceleration 2), and if it is Linux (registered trademark), VDPAU (Video Decode and Presentation) (API for Unix) is unique to the software platform used by each device. The upper CPU 100 reads necessary encoded image data to be displayed on the display unit 400 at a predetermined timing from the image data ROM 200, and displays the encoded image data together with a display command (display timing, display location, etc.). Send to 300. The image processor 300 that has received the encoded image data corresponding to the command restores the original image by performing a decoding process corresponding to the encoding process by an internal hardware decoder (not shown). When the image processing processor 300 supplies the decoded image data to the display unit 400, a moving image is reproduced on the liquid crystal screen or the like of the display unit 400.

  Note that Patent Document 2 uses a tile obtained by encoding an I picture that can be decoded without referring to other image information, and corrects the position of the tile read by the stream correction unit in the frame. A moving image distribution system that can arbitrarily set a moving image viewing area and interactively change the viewing area is disclosed.

  However, the invention described in Patent Document 2 relates to a technique for arbitrarily changing the viewing area of a moving image, and does not avoid a reduction in efficiency of decoding processing when handling a plurality of images with small resolution as described later. Absent.

JP 2012-191513 A International Publication No. 2012/060459

  By the way, moving image reproduction in a gaming machine such as a pachinko machine or a pachislot machine has a feature that is significantly different from general moving image reproduction in which a single moving image is reproduced. FIG. 13 is a diagram for explaining moving image reproduction in a gaming machine such as a pachinko machine or a pachislot machine. Specifically, as shown in FIG. 5 (a), in gaming machines such as pachinko machines and pachislot machines, videos of various sizes, large and small, are displayed in a plurality of arbitrary areas such as one liquid crystal screen. The Furthermore, as shown in FIG. 5B, the playback start time and playback time of each of the moving images are various. In addition, as in the relationship between the moving image E and the moving image G, there is also an aspect in which the moving image E is interrupted without being played to the end and the moving image G starts to be played instead as the game progresses.

  Here, when attention is paid to the decoding processing by the hardware decoder of the images having different resolutions, the processing performance does not depend on the resolution but is not constant but depends on the resolution. In other words, the decoding processing time is not proportionally determined according to the resolution. For example, just because the resolution is halved does not mean that the decoding processing time is halved. FIG. 14 is a diagram illustrating a relationship between image resolution (horizontal axis) and processing performance (Gdps) (vertical axis). Here, for reference, a case where decryption processing is performed by software is also shown. Theoretically, the processing performance should not change depending on the resolution, but as shown in the figure, according to the software processing, the processing performance decreases proportionally as the resolution decreases, while the hardware processing According to the above, it can be seen that the processing performance extremely decreases as the resolution decreases.

  As described above, in the decoding process by the hardware decoder, there are mainly two reasons why the processing performance extremely decreases as the resolution decreases.

One reason is that the hardware decoder performs parallel processing in a predetermined resolution unit in the vertical direction of the image.
FIG. 15 is a diagram for explaining vertical parallel processing in the hardware decoder. FIG. 4A shows a case where parallel processing is not performed, and FIG. 4B shows a case where parallel processing is performed. In general, decoding processing is performed in units of macroblocks (MB) (for example, 16 × 16 pixels), and if parallel processing is not performed, as shown in FIG. One decoder core corresponding to one column sequentially decodes each macroblock in the column. Here, if intra prediction is adopted as in the H.264 standard, decoding is advanced line by line while referring to information of a predetermined macroblock around the already decoded process. Go.

  On the other hand, in the case of having a parallel processing function, a plurality of decoder cores corresponding to a plurality of macroblocks are provided as shown in FIG. On the other hand, the decoding process will proceed in parallel. In this case as well, if intra prediction is adopted, the processing typically proceeds in a wave shape as shown in FIG. 4B. Therefore, such parallel processing is called wavefront parallel processing. (Wave front parallel processing).

  In parallel processing by a plurality of hardware decoder cores as shown in FIG. 15 (b), an image having a sufficiently large resolution in the vertical direction can enjoy the advantages of the parallel processing, but the resolution in the vertical direction. For small images, the benefits may not be enjoyed. For example, if a macroblock is 16 × 16 pixels and eight hardware decoder cores are provided on a one-to-one basis with respect to the macroblock (5 in the figure (b)), 16 pixels × 8 = 128 pixels, that is, the number of vertical pixels processed at one time is 128 pixels. In theory, an image having a smaller number of vertical pixels than 128 enjoys the advantages of the parallel processing. It will not be possible.

  As described above, depending on how much parallel processing function is provided, in general, when decoding processing is performed on an image having a small resolution as compared to processing an image having a large resolution, Processing performance will drop.

  The second reason is that both the low-resolution image and the high-resolution image are one picture, and in the process of decoding one picture, in addition to the image data encoding process itself, initialization of the decoder, etc. Therefore, even if the number of images with a small resolution increases, the number of processing steps increases dramatically. As a result, compared with the case of processing an image with a large resolution. Then, even if the number of images with a small resolution is increased, the performance of the decoding process is lowered.

  From the above, there is a problem that the processing speed becomes extremely slow in an apparatus that displays a large number of moving images with a small resolution, particularly in parallel, such as a gaming machine represented by a pachinko machine and a pachislot machine.

  The present invention has been made for the above-described circumstances, and an object of the present invention is to provide a decoding processing capability even in an image processing apparatus included in a gaming machine that displays a large number of small resolution moving images. An object of the present invention is to provide an image data processing method that does not drop.

In order to achieve the above object, an image data processing method of the present invention outputs encoded image data relating to a moving image, and performs parallel processing by a host CPU that issues a command relating to reproduction of the moving image and a plurality of data cores. has a hardware decoder for, based on the instruction input, and an image processor for decoding the encoded image data according to the video, the video based on the decoded image data by the image processing processor An image data processing method using an image processing apparatus comprising: a display unit to be played back , wherein a vertical resolution of the plurality of moving images to be played back on the display unit is the hardware decoder Among those below the number of pixels involved in parallel processing, with respect to the timing to be reproduced on the display unit, a set that is within a predetermined proximity to each other, If for each video that is a set, is encoded previously by generating the coded image data, a plurality of moving image on the display unit based on the command is reproduced in the same GOP interval, the higher the CPU , By combining the moving images in the set with the same picture type at the level of the slice which is each picture encoded image data, and rewriting the information of the start macroblock coordinates included in the header area of each slice. constitute the coded image data supplied to the image processor collectively to resemble the one picture of a slice, the hardware decoder decodes the lumped coded image data, the higher the CPU Requested the image processor to separate each video from the decoded image data obtained from the hardware decoder, and separated Video is summarized as Rukoto reproduced on the display unit.

In this case, the sum of the vertical resolution of the video is the said set is, it is Ru preferred der exceed the number of pixels parallel processing by the plurality of data cores.

  On the other hand, when the image processor reproduces each moving image on the display unit, the decoded image obtained from the hardware decoder is used to ensure a desired display timing for each frame of each moving image. In some cases, image data is stored in an image data storage unit.

  In addition, among the encoded image data related to the moving image, it is preferable that each moving image related to the coalescence is encoded with the horizontal resolution aligned by supplementing the data in the horizontal direction. At this time, it is one idea to provide a plurality of reference values for the horizontal resolution, and to match the horizontal resolution by matching one of them. In addition, among the encoded image data related to the moving image, it is also preferable that each moving image related to the coalescence is encoded with the vertical resolution aligned by supplementing the data also in the vertical direction. At the time of such compensation, information on the size of the original image of each moving image is stored in SEI or a container.

  In addition, in the decoding process by the hardware decoder of the encoded image data related to each moving image related to the merging, when the encoding process related to any moving image is completed, or the reproduction of any moving image is interrupted If it should be, a new moving image is also incorporated in the GOP unit.

  According to the image data processing method of the present invention, the CPU combines each moving image at the level of a slice that is each picture-encoded image data, and decodes it to a hardware decoder as a single picture of a plurality of slices. Therefore, the decoding processing speed is improved.

  In particular, even when a large number of moving images having a small resolution in the vertical direction are reproduced, the advantage of the parallel processing function of the hardware decoder can be enjoyed, which is effective.

  Further, since the moving images are combined in a manner in which the slices are grouped into a plurality of slices in units of pictures of each moving image, it is not necessary to change the configuration or function of the hardware decoder itself.

It is a figure for demonstrating the outline | summary of one Embodiment in the image data processing method of this invention. It is a flowchart which shows the procedure of the encoding process of each moving image. It is a figure for demonstrating the filling (padding) process of image data. It is a figure which shows the bit stream of each moving image after an encoding. It is a block diagram which shows the structure of the image processing apparatus used in one Embodiment of the image data processing method of this invention. It is a figure for demonstrating the time adjustment which unites the picture type between each moving image in the pre-processing before decoding. It is a figure for demonstrating combining each moving image at the level of the slice which is each picture coding data, and decoding collectively considering it as one picture of several slices. It is a figure for demonstrating the case where a certain moving image interrupts and switches to another moving image. It is a figure for demonstrating the case where a certain moving image interrupts and switches to another moving image. It is a figure for demonstrating the process of the post process part 312. FIG. It is a figure which shows the structure of the apparatus for reproducing a moving image. It is a figure which shows the general bit stream of the coding data of the image encoded based on H.264 specification. It is a figure for demonstrating the moving image reproduction | regeneration in game machines, such as a pachinko machine and a pachislot machine. It is a figure which shows the relationship between image resolution (horizontal axis) and processing performance (Gdps) (vertical axis). It is a figure for demonstrating the vertical direction parallel processing in a hardware decoder.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
<Overview>
FIG. 1 is a diagram for explaining an outline of an embodiment of an image data processing method according to the present invention. In short, in the present invention, encoded image data of a plurality of moving images with small vertical resolution are combined, the combined image is decoded, and the plurality of original moving images are obtained from the obtained image data. By performing processing such as separation, even if there are many moving images with small vertical resolution, the decoding processing performance is not degraded.

  More specifically, in FIG. 2A, first, an image processing apparatus (for example, one incorporated in a pachinko machine) that is intended to perform a predetermined process including a predetermined moving image display process. From the processing design (especially for video display, what video is displayed at what timing, etc.), among the videos displayed on the device, the video that should be combined and decoded The decoding process design of selecting is performed (step S1). For example, as shown in FIG. 5B, it is designed that the moving image X may be processed independently, and the moving image Y and the moving image Z having a small vertical resolution may be combined and decoded. Then, the moving image X, the moving image Y, and the moving image Z are encoded (step S2). Next, with respect to the moving image X, the encoded data is decoded alone, the moving image X is restored, and displayed on the display unit of the image processing apparatus at a predetermined timing. On the other hand, for video Y and video Z, depending on their display timing, the encoded data is merged and decoded, video X and video Y are restored, and further separated into the display unit, respectively. Is displayed at the timing (step S3).

Hereinafter, an embodiment in which the above-described outline is made more specific will be described in detail including some predetermined conditions.
<Decryption design process (step S1)>
From the viewpoint of resolution, the conditions for selecting moving images to be combined at the time of decoding are as follows.
-The vertical resolution of each video to be merged is less than or equal to the "number of pixels" of parallel processing by multiple decoder cores.-The total vertical resolution of each video to be merged is parallel by multiple decoder cores. Exceeding the “number of pixels” of processing ・ However, the total vertical resolution of the videos to be merged must not exceed the “maximum resolution” of the decoder in the vertical direction.

  Further, from the viewpoint of the timing of displaying each moving image on the image processing device, the moving images to be combined are those having display timings as close as possible. As long as the above-mentioned resolution viewpoint conditions are satisfied, videos displayed at any timing may be used. However, the longer the time between the decoding process and the display time, the more The staying time in the image data buffer for temporarily storing the image data becomes longer. Therefore, the selection of the moving images to be combined from the viewpoint of display timing depends on the capacity of the image data buffer. Note that, taking a gaming machine as an example, the decoding design processing is performed as data to be decoded by combining video that is played simultaneously or simultaneously in one event and that satisfies the above conditions. Is preferred.

<Encoding process (step S2)>
Next, the encoding process of each moving image will be described. FIG. 2 is a flowchart showing the processing procedure. Therefore, a moving image to be encoded is selected (step S21). Then, it is determined whether the selected moving image is a moving image to be combined with another moving image based on the above-described decoding design process (step S22). In the case of the moving image A as illustrated in FIG. 13, since the decoding process is performed alone, the determination in step S22 is negative, and step S23 is skipped and the process proceeds to step S24. On the other hand, in the case of each of the moving image B to the moving image G (the vertical resolution is, for example, 128 pixels or less) as illustrated in FIG. 13, the decoding process is performed by combining with any of the other. Is affirmative, and the process proceeds to step S23.

  In step S23, image data filling (padding) processing is performed. FIG. 3 is a diagram for explaining the compensation process. Thus, for example, it is assumed that the moving image B and the moving image F can be combined into a set, and the moving image C, the moving image D, and the moving image E (moving image G) should be combined together. Note that the moving image G is a moving image to be reproduced from the moving image E when the moving image E is interrupted during the reproduction. Therefore, at least the same set of moving images needs to have the same horizontal resolution from the decoding processing method described later. Therefore, in order to make the resolution in the horizontal direction uniform, image data compensation processing is performed in the horizontal direction of each moving image. At this time, the maximum processing resolution in the horizontal direction of the decoder may be uniformly set. By doing so, the same maximum processing resolution can be set for all the pairs, so that the compensation process itself becomes simple. However, even when the moving images having only a small resolution in the horizontal direction are combined, if only the same is done, only the compensation data becomes large, and the decoding process is wasted. Therefore, it is a good idea to provide several alignment standards, such as 120, 240, 480, 960, 1920 pixels in the horizontal direction. The standard to align is determined by the resolution of the video having the maximum horizontal resolution among the videos to be merged, so the number of these standards and the setting of the values are determined for each horizontal resolution of the videos to be merged. Depends on variation. In other words, horizontal resolution can be included as a condition for selecting moving images to be combined.

  In step S24, each moving image is encoded. For example, it is typically performed by software processing based on the H.264 standard. The condition at this time is that the encoding processing is performed so that the number of frames included in one GOP is the same (for example, 30) for at least the moving images in the same set. This is because the decryption process can be performed by combining them in the decryption process described later. FIG. 4 is a diagram showing a bit stream of each moving image after encoding. As in the conventional case, one picture is configured as one slice, and the macroblock start coordinates are included in the slice header as in the conventional case. However, as a characteristic feature of the present invention, “data size of the original image” (resolution) (for example, 110 × 48 in the case of the moving image C shown in FIG. 3) is additionally provided. In other words, processing to be put in a container containing NAL such as SEI or MP4. This is because an original image can be extracted after the decoding processing in an image processing apparatus that performs decoding processing of each moving image and displays it, as will be described later.

  In step S25, it is determined whether or not an unprocessed moving image remains. If it remains (affirmative determination), the process returns to step S21, the above processing is repeated, and if all moving images are completed, the processing ends.

  In the above-described embodiment, the vertical resolution is set as it is for each moving image, that is, it is divided, but for the moving images to be combined and decoded, the vertical resolution is also made uniform by the compensation process. It can be considered. If the resolution in the vertical direction is also made uniform, the moving images to be decoded and combined as the same set can be exchanged uniformly, and the processing becomes simple in the above-described encoding design processing.

<Image Data Processing by Actual Operation of Image Processing Device with Decoder (Step S3)>
FIG. 5 is a block diagram showing a configuration of an image processing apparatus used in an embodiment of the image data processing method of the present invention.
The image processing apparatus shown in FIG. 1 includes a host CPU 1, an image data ROM 2, an image processing processor 3, and a display unit 4. The host CPU 1 includes a preprocessing unit 11, and the preprocessing unit 11 includes each moving image reproduction timing information generation unit 111 and each inter-moving picture uniting unit 112. The image processor 3 includes a post-processing unit 31, a hardware decoder 32, and a video memory 33, and the post-processing unit 31 further includes a command interpreting unit 311 and a drawing circuit 312. The drawing circuit 312 includes each moving image separation unit 3121, an original image cutout unit 3122, and each moving image display timing control unit 3123. Here, the image data ROM 2 stores data of each moving image encoded by the above-described <encoding process (step S2)>.

The outline of the image data processing in the image processing apparatus shown in FIG. 5 is as follows. Details of the pre-processing unit 11 and the post-processing unit 31 will be described later.
The upper CPU 1 reads out the encoded image data relating to the necessary moving image from the image data ROM 2 based on the design operation in the image processing apparatus. When the encoded image data relating to a plurality of moving images that can be merged is read out from the image data ROM 2 based on the information relating to each moving image that can be merged, as described in detail later, In 112, the moving picture is merged as each slice in a state where the picture types are aligned between the moving pictures. At this time, the information of the leading macro block coordinates included in the header area of the slice to be merged is rewritten. Then, each inter-moving picture uniting unit 112 supplies the encoded image data after the uniting composed of a plurality of slices to the hardware decoder 32 of the image processing processor 3.

  Since each moving image based on encoded image data after combining composed of a plurality of slices usually has different timings to be displayed, the upper CPU 1 needs to inform the image processor 3 of the reproduction timing information of each moving image. is there. Each moving image reproduction timing information generation unit 111 performs processing for generating the reproduction timing information. That is, each moving image reproduction timing information generation unit 111 sets an adjustment time such as the reproduction timing of a plurality of moving images displayed in one event of the gaming machine, and supplies the information to the image processor 3.

Hardware decoder 32 performs decoding processing on the encoded image data after merging supplied from the host CPU1, supplies decoding result to the video memory 3 3, outputs the decoding completion notification (not shown) to the upper CPU1 To do. The host CPU 1 that has received the decoding completion notification from the hardware decoder 32 issues a post-processing command to the post-processing unit 31 of the image processor 3, and the command interpretation unit 311 includes at least the coalescing included in the command. Information relating to each moving image and information relating to the reproduction timing of each moving image are supplied to the drawing circuit 312. Each moving image separation unit 3121, original image cutout unit 3122, and each moving image display timing control unit 3123 included in the drawing circuit 312 includes the information and the original image included in the decoded image data itself stored in the video memory 33. After performing predetermined post-processing on the decoded image data stored in the video memory 33 based on the data size information (see FIG. 4), the result is supplied to the display unit 4 and displayed as each moving image. Let

Detailed processing contents of the pre-processing unit 11 and the post-processing unit 31 will be described below by way of example with reference to FIGS. 6 to 10.
The host CPU 1 issues a moving image display command based on an operation process designed for the image processing apparatus (in the case of a gaming machine, a design operation for advancing the game in the gaming machine in which the image processing apparatus is incorporated). . For example, in the example shown in FIG. 13, “Play video A (B,...) From a predetermined timing” or while video E is being played, “playback of video E is interrupted. Then, instead, a command such as “Start playing video G” is issued.

  The upper CPU 1 obtains encoded image data from the image data ROM 2 based on the issued command. For example, if the command is “Reproduce the moving image A from a predetermined timing”, the upper CPU 1 converts the encoded image data related to the moving image A to image data. After obtaining the image data obtained from the ROM 2 and supplied to the hardware decoder 32 of the image processor 3 and stored in the video memory 33, an image is created using the drawing circuit 312 and displayed at a predetermined timing. 4 is played back on the display unit 4.

  Further, when a command for reproducing a plurality of moving images, for example, moving images C, D, E, and G, is generated by the upper CPU 1, a slice, which is necessary encoded image data, is read from the image data ROM 2, and each inter-moving picture combining unit 112 is read. Rewrites the information of the head macroblock coordinates included in the header area of the slice to be merged, merges a plurality of slices, generates merged compressed data, and supplies it to the hardware decoder 32 of the image processor 3 To do. Note that the moving images C, D, and E read from the image data ROM 2 may be subjected to compensation processing after reading.

  Also, the coordinate information rewriting of the first macroblock included in the header area of the slice means that the coordinate information of the first macroblock of each slice is initially stored as “0”. The coordinate information of the leading macroblock is changed to a desired value in accordance with the slice arrangement. For example, as shown in FIG. 10A, when the moving image C, the moving image D, and the moving image E are combined and combined as one large image, the coordinate information of the first macroblock included in the header area of the slice of the moving image C needs to be rewritten. However, the coordinate information of the first macroblock included in the header areas of the slices of the moving image D and the moving image E is rewritten according to the information on the positions where they are arranged. This is to generate an effective H.264 stream that can normally decode the encoded image data including a plurality of slices by the hardware decoder 32.

  After the encoded image data is decoded by the hardware decoder 32 and stored in the video memory 33, information for cutting out the combined decoded image data into individual images, information such as moving image separation and moving image display timing, etc. The CPU 1 generates the information, and the information is interpreted by the instruction interpretation unit 311 of the image processor 3.

  For example, the commands interpreted by the command interpretation unit 311 are “replay video C from a predetermined timing”, “reproduce video D from a predetermined timing”, and “reproduce video E from a predetermined timing”. If there is, the image related to the moving image C, the image related to the moving image E, and the image related to the moving image E are respectively cut out from the decoded image data stored in the video memory 33, and the display device is displayed at a designated timing. The drawing circuit 312 is controlled to display the image on the screen 4.

  In addition, since the image data regarding a some moving image is united as a slice and the compressed data after the uniting is processed by the same hardware decoder 32, the picture types of the image data to be united as a slice need to be the same. Therefore, the number of frames included in one GOP of each moving image that may be merged is the same, and as shown in FIG. 6, an arbitrary (first) GOP I of each moving image (moving image C, moving image D, moving image E) You have to align the pictures and merge the slices.

  On the other hand, as shown in FIG. 13, when the playback timings of the moving image C, the moving image D, and the moving image E are all different, it is impossible to decode the images used for each moving image and draw each image at the same time. For this reason, information on the playback timing (drawing start timing) of the moving image is generated by each moving image playback timing information generation unit 111 of the host CPU 1, and the drawing circuit 312 of the image processor 3 generates an image based on the information, It is necessary to control to display on the display device 4 at the timing.

  As described above, in a state where the picture types of the respective moving pictures are aligned, the inter-moving picture uniting unit 112 unites the moving pictures in units of frames, that is, the moving pictures are combined at the level of the slice that is the encoded data of each picture. Then, it is organized as if it were one picture of a plurality of slices. FIG. 7 is a diagram for explaining this. As described above, the bit stream of the encoded data of each moving image is normally composed of one slice per picture as shown on the left side of FIG. Are combined at the slice level. Here, SPS (Sequence Parameter Set), PPS (Picture Parameter Set), and SEI (Supplemental Enhancement Information) are set as one set, and at that time, they are rewritten. For example, the SPS includes information on the number of reference frames and the width and height of the frame, but the height information after being combined needs to be adjusted. The PPS includes information such as an encoding method and a quantization coefficient, but adjustment is necessary when the information is different in each image. In addition, the start macroblock coordinates included in the slice header described with reference to FIG. 12 are also rewritten in accordance with the merge in each slice.

  The pre-processing unit 11 organizes the encoded image data of each moving image at the slice level as described above, and supplies the encoded image data to the hardware decoder 32 while combining the moving images. Here, on the hardware decoder 32 side, when the encoded image data is received, it is recognized that one picture is composed of a plurality of slices (three slices in the example), but this is only a standard (H.264). (The same applies to MPEG-2 and 4), and the decoding process can be performed as it is. However, the hardware decoder 32 does not recognize that it is a combination of a plurality of moving images. In other words, this means that the hardware decoder 32 need not be modified. In other words, this means that pre-processing and post-processing described later are performed.

  Returning to FIG. 6, the playback time of each moving image varies, so that the shortest one ends first. At this time, another moving image can be inserted in the empty space (in terms of vertical resolution, display timing, and capacity of the image data buffer 3124), but at that time, matching with other moving images is performed in GOP units. FIG. 8 and FIG. 9 illustrate a case where the moving image E is switched to the moving image G in the middle as illustrated in FIG. FIG. 8A shows a case where the video E is not interrupted, and FIG. 8B shows a case where the video E is interrupted and the video G is reproduced instead. In this case as well, switching is performed in units of GOPs. In the example of FIG. 5B, GOP [1] of video G is incorporated after GOP [2] of video E instead of GOP [3] of video E. ing. FIG. 9 shows the boundary between the GOP [2] of the moving picture E and the GOP [1] of the moving picture G from the viewpoint of combining the pictures of the moving pictures. As shown in the figure, in the final frame before switching, the P pictures of the moving picture C, the moving picture D, and the moving picture E are merged, but in the first frame after the switching, the moving picture C, The I pictures of the moving picture D and the moving picture G are combined.

Next, the post-processing unit 31 will be described. FIG. 10 is a diagram for explaining the processing of the post-processing unit 31.
When the image data after decoding from the hardware decoder 32 is stored in the video memory 33, first, each moving image separation unit 3121, as shown on the left side of FIG. Each video is separated for each frame based on the start macroblock coordinates of the video. Next, as shown on the right side of FIG. 5A, the original image cutout unit 3122 performs <encoding processing (for each frame) based on the SEI of each moving image or the data size of the original image stored in the container. In step S2)>, the compensated compensation data is deleted. The moving image data in which the original image is reproduced for each frame in this way is accumulated in the video memory 33. Then, each moving image display timing control unit 3123, based on the information regarding the reproduction timing of each moving image from each moving image reproduction timing information generation unit 111 of the host CPU 1, as shown in FIG. Each is reproduced on the display unit 4 at the display timing.

  In the above description, the moving image C, the moving image D, the moving image E, and the moving image G illustrated in FIG. 13 have been described as examples. However, the moving image B and the moving image F illustrated in FIG. It is the same.

  As described above, according to the embodiment of the image data processing method of the present invention, since the moving images whose resolutions are small in the vertical direction are combined and collectively decoded, a plurality of video data processing methods are performed in the vertical direction. Even when parallel processing is performed by the decoder core, the function can be enjoyed. In addition, as the number is reduced by combining, the accompanying processing such as initialization of the decoder is reduced, and the processing time is remarkably shortened.

  Further, since the moving images are combined in a manner in which the slices are grouped into a plurality of slices in units of pictures of each moving image, it is not necessary to change the configuration or function of the hardware decoder 32 at all.

  The image data processing method of the present invention can be employed in gaming machines such as pachinko machines, for example.

1 Host CPU
11 Pre-processing unit 111 Each moving image reproduction timing information generating unit 112 Inter-moving image uniting unit 2 Image data ROM
3 Image Processor 31 Post-Processing Unit 311 Command Interpreting Unit 312 Drawing Circuit 3121 Each Moving Image Separating Unit 3122 Original Image Cutting Unit 3123 Each Moving Image Display Timing Control Unit 32 Hardware Decoder 33 Video Memory 4 Display Unit 100 Host CPU
200 Image data ROM
300 Image processor 400 Display unit

Claims (8)

An upper CPU that outputs encoded image data related to a moving image and issues a command related to reproduction of the moving image;
An image processor that has a hardware decoder that performs parallel processing by a plurality of data cores, and that decodes encoded image data related to the moving image based on the input command;
A display unit that reproduces the moving image based on the image data decoded by the image processing processor;
An image data processing method using an image processing apparatus comprising:
Among the plurality of moving images to be reproduced on the display unit, the vertical resolution is less than or equal to the number of pixels related to the parallel processing of the hardware decoder. A set of images having closeness within a predetermined range is set, and for each moving image, the encoded image data is generated by encoding at the same GOP interval,
When a plurality of video is played on the display unit based on the command, the higher the CPU, each video is the said set, coalescing align the picture type at the level of the slice is the picture coded image data Then, the information of the start macroblock coordinates included in the header area of each slice is rewritten, and the encoded image data is collectively configured to be regarded as one picture of a plurality of slices, and supplied to the image processor. The wear decoder decodes the batched encoded image data ,
The upper CPU, the request from the resulting decoded image data to the image processor so as to separate the video from the hardware decoder, characterized Rukoto each video separated is reproduced on the display unit An image data processing method. Image data processing method according to claim 1 Total vertical resolution of each video that is to the set, characterized in Rukoto exceeds the number of pixels parallel processing by the plurality of data cores. When the image processor reproduces each moving image on the display unit, the decoded image data obtained from the hardware decoder is used to ensure a desired display timing for each frame of each moving image. The image data processing method according to claim 1 , wherein the image data is stored in an image data storage unit.   2. The image according to claim 1, wherein among the encoded image data related to the moving image, each moving image related to the coalescence is encoded with horizontal resolution aligned by supplementing data in a horizontal direction. Data processing method. 5. The image data processing method according to claim 4 , wherein a plurality of reference values are provided for the horizontal resolution, and the horizontal resolution is made uniform by matching one of them. 5. The encoded image data according to claim 4 , wherein among the encoded image data related to the moving image, each moving image related to the coalescence is encoded with the vertical resolution aligned by supplementing the data in the vertical direction as well. Image data processing method. Size information of the original image for each video, the image data processing method according to any one of claims 4 to 6, wherein the storing in the SEI or container. When the encoding process related to any moving image is completed or the reproduction of any moving image should be interrupted in the decoding process by the hardware decoder of the encoded image data related to each moving image related to the merging the image data processing method according to claim 1, a new video, characterized in that it is incorporated in the GOP.

Images