JP5484276B2 - Data compression apparatus, data decoding apparatus, data compression method, data decoding method, and data structure of compressed video file - Google Patents

Description

  The present invention relates to an information processing apparatus and information processing method for encoding and decoding three-dimensional data such as moving images.

  Home entertainment systems have been proposed that not only execute game programs, but also play video. In this home entertainment system, the GPU generates a three-dimensional image using polygons (see, for example, Patent Document 1).

  Regardless of moving images or still images, how to display images efficiently is always an important issue. For this reason, various technologies such as image data compression technology, transmission technology, image processing technology, and display technology have been developed and put into practical use, and high-definition images can be enjoyed in various situations.

US Pat. No. 6,563,999

  There is always a demand to display a high-definition image with high responsiveness according to a user's request. For example, in order to realize an image display with a high degree of freedom with respect to the user's viewpoint, such as enlarging and displaying a region that the user wants to focus on or displaying another region of the displayed whole image. Therefore, it is necessary to enable random access while processing large-size image data in a short time, and further technical progress is required.

  The present invention has been made in view of such problems, and an object thereof is to provide an information processing technique capable of outputting three-dimensional data such as a moving image with high responsiveness to various requests.

  One embodiment of the present invention relates to a data compression apparatus. This data compression apparatus divides a data string in a three-dimensional space to be compressed in the three-dimensional direction to form a coding unit, and for each coding unit formed by the data dividing unit, Among them, a palette that holds two values as representative values and information that specifies any one of a plurality of intermediate values and representative values determined by linear interpolation of the representative values are held instead of the original data of the coding unit. And a compression encoding unit that generates an index and generates compressed data.

  Another aspect of the present invention relates to a data decoding apparatus. This moving image data decoding apparatus includes, for each encoding unit formed by dividing a data string in a three-dimensional space in the three-dimensional direction, a palette that holds two of the pixel values as a representative value, and the representative value. Compressed data that reads from the storage device compressed data that associates information that specifies any one of a plurality of intermediate values and representative values determined by linear interpolation, instead of the original data of the encoding unit. The reading unit and the intermediate value are generated by linearly interpolating the representative value held by the palette, and according to the information held by the index, the data included in each coding unit is determined as either the representative value or the intermediate value, And a decoding unit that reconstructs and generates an original data sequence based on an array of encoding units.

  Yet another embodiment of the present invention relates to a data compression method. This data compression method includes a step of reading a data string in a three-dimensional space to be compressed from a storage device, a step of dividing the data string in a three-dimensional direction to form an encoding unit, and a data unit for each encoding unit. A palette that holds two values as representative values, and information that specifies any of a plurality of intermediate values and representative values determined by linear interpolation of the representative values, instead of the original data of the coding unit Generating an index to be stored in a storage device as compressed data.

  Yet another embodiment of the present invention relates to a data decoding method. In this data decoding method, for each encoding unit formed by dividing a data string in a three-dimensional space in the three-dimensional direction, a palette that holds two of the pixel values as a representative value, and the representative value is linear A step of reading from the storage device compressed data in which information specifying any one of a plurality of intermediate values and representative values determined by interpolation is held instead of the original data of the encoding unit; and An intermediate value is generated by linearly interpolating the representative value held in the palette, and the data included in each coding unit is determined as one of the representative value and the intermediate value according to the information held in the index. The method includes a step of reconstructing and generating an original data sequence based on the arrangement, and a step of outputting the generated data sequence to an output device.

  Still another embodiment of the present invention relates to a data structure of a compressed moving image file. This data structure corresponds to an image frame sequence constituting a moving image, a Y image sequence having a luminance Y as a pixel value, a Cb image sequence having a color difference Cb as a pixel value, and a Cr image sequence having a color difference Cr as a pixel value. A palette that holds two of the pixel values as representative values generated for each encoding unit formed by space-time division, and any of a plurality of intermediate values and representative values determined by linear interpolation of the representative values An index for holding information for designating each pixel is associated with an image area of an image frame, and is arranged in association with the image area.

  It should be noted that any combination of the above-described constituent elements and a representation of the present invention converted between a method, an apparatus, a system, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, it is possible to output 3D data with high throughput and random access.

It is a figure which shows the use environment of the image processing system which can be applied to this Embodiment. It is a figure which shows the example of an external appearance structure of the input device applicable to the image processing system of FIG. It is a figure which shows notionally the hierarchy data of the moving image made into a process target in this Embodiment. It is a figure which shows the structure of the image processing apparatus in this Embodiment. In this Embodiment, it is a figure which shows the structure of the control part which has a function which displays a moving image using the moving image data which has a hierarchical structure in detail. It is a figure which shows the structural example of the moving image data used as the process target in this Embodiment. It is a figure which shows the structural example of the moving image data used as the process target in this Embodiment. It is a figure which shows the structural example of the moving image data used as the process target in this Embodiment. It is a figure which shows the structural example of the moving image data used as the process target in this Embodiment. It is a figure which shows typically the data structure of the moving image in the case of substituting the moving image stream of a part hierarchy with the moving image stream of another hierarchy in this Embodiment. In this Embodiment, it is a figure which shows the structure of the control part and hard disk drive which have a moving image data compression function in detail. It is a figure which shows typically the compression procedure of the moving image stream which the image processing apparatus containing the control part shown in FIG. 11 implements. It is a figure which shows typically the method of producing | generating the data of a palette and an index from the encoding unit of a Y image sequence in this Embodiment. It is a figure which shows typically the method of producing | generating the data of a palette and an index from the encoding unit of a CbCr image sequence in this Embodiment. It is a figure which shows the variation of the pattern which divides | segments one process unit in this Embodiment. It is a figure which shows the example of a data structure of the division | segmentation pattern map in this Embodiment. It is a figure for demonstrating the arrangement | sequence of the compressed data in the compressed data storage part of this Embodiment. It is a figure which shows typically the transition of data when a compression encoding process is performed to the whole moving image stream in this Embodiment. It is a figure for demonstrating the method of embedding the identification number of a division | segmentation pattern in two pallets in this Embodiment.

  In the present embodiment, it is possible to move the display area corresponding to the user's viewpoint movement request in moving image display. The viewpoint movement here includes moving the viewpoint closer to or away from the image plane, and the moving image is enlarged and reduced while being reproduced accordingly. Therefore, in the present embodiment, the processing target moving image data has a hierarchical structure in which a plurality of moving image streams each composed of an image frame sequence representing one moving image at different resolutions are hierarchized in the order of resolution. In response to a request for moving the viewpoint in the near and near direction, the moving image stream used for display is switched to a different layer, so that enlarged display and reduced display are quickly performed. Hereinafter, moving image data having such a hierarchical structure is also referred to as “hierarchical data”.

  First, a basic display mode of such hierarchical data will be described. FIG. 1 shows a use environment of an image processing system 1 to which this embodiment can be applied. The image processing system 1 includes an image processing device 10 that executes image processing software, and a display device 12 that outputs a processing result by the image processing device 10. The display device 12 may be a television having a display that outputs an image and a speaker that outputs sound.

  The display device 12 may be connected to the image processing device 10 by a wired cable, or may be wirelessly connected by a wireless local area network (LAN) or the like. In the image processing system 1, the image processing apparatus 10 may be connected to an external network such as the Internet via the cable 14 to download and acquire hierarchical data. The image processing apparatus 10 may be connected to an external network by wireless communication.

  The image processing device 10 may be, for example, a game device or a personal computer, and may implement an image processing function by loading an image processing application program. The image processing apparatus 10 performs enlargement / reduction processing of a moving image displayed on the display of the display device 12, scroll processing in the vertical and horizontal directions, and the like in response to a viewpoint movement request from the user. Hereinafter, the display area changing process including such enlargement / reduction is expressed as “movement of the display area”. When the user operates the input device while viewing the image displayed on the display, the input device transmits a display area movement request signal to the image processing device 10.

  FIG. 2 shows an external configuration example of the input device 20. The input device 20 includes a cross key 21, analog sticks 27a and 27b, and four types of operation buttons 26 as operation means that can be operated by the user. The four types of operation buttons 26 include a circle button 22, a x button 23, a square button 24, and a triangle button 25.

  In the image processing system 1, a function for inputting a display image enlargement / reduction request and a vertical / left / right scroll request is assigned to the operation unit of the input device 20. For example, the input function of the display image enlargement / reduction request is assigned to the right analog stick 27b. The user can input a display image reduction request by pulling the analog stick 27b forward, and can input a display image enlargement request by pressing the analog stick 27b from the front.

  The scroll request input function is assigned to the cross key 21. The user can input a scroll request in the direction in which the cross key 21 is pressed by pressing the cross key 21. Note that the display area movement request input function may be assigned to another operation means. For example, the scroll request input function may be assigned to the analog stick 27a.

  The input device 20 has a function of transmitting an input display area movement request signal to the image processing device 10 and is configured to be capable of wireless communication with the image processing device 10 in the present embodiment. The input device 20 and the image processing device 10 may establish a wireless connection using a Bluetooth (registered trademark) protocol, an IEEE802.11 protocol, or the like. The input device 20 may be connected to the image processing apparatus 10 via a cable and transmit a display area movement request signal to the image processing apparatus 10.

  FIG. 3 conceptually shows hierarchical data of moving images to be processed in the present embodiment. The hierarchical data has a hierarchical structure including a 0th hierarchy 30, a first hierarchy 32, a second hierarchy 34, and a third hierarchy 36 in the z-axis direction from the top to the bottom of the figure. Although only four layers are shown in the figure, the number of layers is not limited to this. As described above, each layer includes moving image data representing one moving image at different resolutions, that is, data in which a plurality of image frames are arranged in time series. In the figure, each layer is symbolically represented by four image frames, but the number of image frames naturally varies depending on the playback time and frame rate of the moving image.

  As will be described later, the present embodiment is excellent in random accessibility with respect to a three-dimensional space of an image plane and a time axis possessed by moving image data. Therefore, for example, by regarding the time axis as “depth”, three-dimensional volume data may be processed instead of moving image data. Similarly, the type of parameter is not particularly limited as long as the data can have redundancy in the three-dimensional direction.

  For example, the hierarchical data has a hierarchical structure of a quadtree, and when the image frames constituting each hierarchy are divided into “tile images” having the same size, the 0th hierarchy 30 is one tile image, and the first hierarchy 32 Is 2 × 2 tile images, the second layer 34 is 4 × 4 tile images, the third layer is 8 × 8 tile images, and the like. At this time, the resolution of the Nth layer (N is an integer of 0 or more) is ½ of the resolution of the (N + 1) th layer in both the left and right (x-axis) directions and the vertical (y-axis) direction on the image plane. Hierarchical data can be generated by reducing an image frame in a plurality of stages based on a moving image of the third hierarchy 36 having the highest resolution.

  As shown in FIG. 3, the viewpoint coordinates at the time of moving image display and the corresponding display area are a virtual three-dimensional space composed of an x-axis representing the horizontal direction of the image, a y-axis representing the vertical direction, and a z-axis representing the resolution. Can be expressed as As described above, in this embodiment, since moving image data including a plurality of image frames is prepared as a hierarchy, the actually displayed image depends on the time from the start of reproduction. The time axis t is represented.

  The image processing apparatus 10 basically draws image frames in any hierarchy sequentially at a predetermined frame rate along the time axis t. For example, a moving image having a resolution of the 0th hierarchy 30 is displayed as a reference image. If a display area movement request signal is supplied from the input device 20 in the process, the change amount of the display image is derived from the signal, and the coordinates of the four corners of the next frame in the virtual space (frame coordinates) are used by using the change amount. ) Is derived. Then, an image frame corresponding to the frame coordinates is drawn. At this time, by providing a layer switching boundary with respect to the z-axis, the layer of moving image data used for frame drawing is appropriately switched according to the value of z of the frame coordinates.

  Instead of the frame coordinates in the virtual space, the image processing apparatus 10 may derive information for specifying the hierarchy and texture coordinates (UV coordinates) in the hierarchy. Hereinafter, the combination of the hierarchy specifying information and the texture coordinates is also referred to as frame coordinates.

  In the image processing apparatus 10, the hierarchical data is held in the storage device in a compressed state in a predetermined compression format. Data necessary for frame drawing is read from the storage device and decoded. FIG. 3 conceptually represents hierarchical data, and does not limit the storage order or format of data stored in the storage device. For example, if the position of the hierarchical data in the virtual space is associated with the storage area for the actual moving image data, the moving image data can be stored in an arbitrary area. As will be described later, space division or time division may be applied to the image frame sequence constituting each layer, and compression coding may be performed in that unit.

  FIG. 4 shows the configuration of the image processing apparatus 10. The image processing apparatus 10 includes a wireless interface 40, a switch 42, a display processing unit 44, a hard disk drive 50, a recording medium mounting unit 52, a disk drive 54, a main memory 60, a buffer memory 70, and a control unit 100. . The display processing unit 44 has a frame memory that buffers data to be displayed on the display of the display device 12.

  The switch 42 is an Ethernet switch (Ethernet is a registered trademark), and is a device that transmits and receives data by connecting to an external device in a wired or wireless manner. The switch 42 is connected to an external network via the cable 14 and configured to receive hierarchical data from the image server. The switch 42 is connected to the wireless interface 40, and the wireless interface 40 is connected to the input device 20 using a predetermined wireless communication protocol. A display area movement request signal input from the user by the input device 20 is supplied to the control unit 100 via the wireless interface 40 and the switch 42.

  The hard disk drive 50 functions as a storage device that stores data. The hierarchical data may be stored in the hard disk drive 50. When a removable recording medium such as a memory card is mounted, the recording medium mounting unit 52 reads data from the removable recording medium. When a read-only ROM disk is loaded, the disk drive 54 drives and recognizes the ROM disk to read data. The ROM disk may be an optical disk or a magneto-optical disk. Hierarchical data may be stored in these recording media.

  The control unit 100 includes a multi-core CPU, and includes one general-purpose processor core and a plurality of simple processor cores in one CPU. The general-purpose processor core is called a PPU (Power Processing Unit), and the remaining processor cores are called a SPU (Synergistic-Processing Unit). The control unit 100 may further include a GPU (Graphics Processing Unit).

  The control unit 100 includes a memory controller connected to the main memory 60 and the buffer memory 70. The PPU has a register, has a main processor as an operation execution subject, and efficiently assigns a task as a basic processing unit in an application to be executed to each SPU. Note that the PPU itself may execute the task. The SPU has a register, and includes a sub-processor as an operation execution subject and a local memory as a local storage area. The local memory may be used as the buffer memory 70.

  The main memory 60 and the buffer memory 70 are storage devices and are configured as a RAM (Random Access Memory). The SPU has a dedicated DMA (Direct Memory Access) controller as a control unit, can transfer data between the main memory 60 and the buffer memory 70 at high speed, and the frame memory and the buffer memory 70 in the display processing unit 44 can be transferred. High-speed data transfer can be realized. The control unit 100 according to the present embodiment realizes a high-speed image processing function by operating a plurality of SPUs in parallel. The display processing unit 44 is connected to the display device 12 and outputs an image processing result according to a request from the user.

  The image processing apparatus 10 sequentially transfers moving image data spatially and temporally close to the currently displayed frame from the hard disk drive 50 to the main memory in order to smoothly perform enlargement / reduction processing and scroll processing of the display image. 60 is loaded. Further, a part of the moving image data loaded in the main memory 60 is decoded and stored in the buffer memory 70. As a result, the display area can be smoothly moved while moving picture reproduction is progressing. At this time, the data to be loaded or decoded may be determined by pre-reading the necessary area based on the movement direction of the display area so far.

  In the hierarchical data shown in FIG. 3, the position in the z-axis direction indicates the resolution. The position closer to the 0th hierarchy 30 has a lower resolution, and the position closer to the third hierarchy 36 has a higher resolution. When attention is paid to the size of the image displayed on the display, the position in the z-axis direction corresponds to the scale ratio. When the scale ratio of the display image of the third hierarchy 36 is 1, the scale ratio in the second hierarchy 34 is 1. / 4, the scale factor in the first hierarchy 32 is 1/16, and the scale factor in the 0th hierarchy 30 is 1/64.

  Therefore, in the z-axis direction, when the display image changes in the direction from the 0th layer 30 side to the third layer 36 side, the display image expands and the direction from the third layer 36 side to the 0th layer 30 side. In the case of changing to, the display image is reduced. For example, when the scale ratio of the display image is in the vicinity of the second hierarchy 34, the display image is created using the image data of the second hierarchy 34.

  Specifically, as described above, a switching boundary is provided at an intermediate scale ratio of each layer. For example, when the scale ratio of the image to be displayed is between the switching boundary between the first hierarchy 32 and the second hierarchy 34 and the switching boundary between the second hierarchy 34 and the third hierarchy 36, the second hierarchy 34. A frame is drawn using the image data. At the switching boundary between the first hierarchy 32 and the second hierarchy 34 and the scale ratio between the second hierarchy 34, the image frame of the second hierarchy 34 is scaled and displayed. At the switching boundary between the second layer 34 and the third layer 36 and the scale ratio between the second layer 34, the image frame of the second layer 34 is enlarged and displayed.

  On the other hand, when a future necessary region predicted from the display region movement request signal is identified and decoded, the scale ratio of each layer is set as a prefetch boundary. For example, at least a part of the image data of the first hierarchy 32 in the reduction direction is transferred from the hard disk drive 50 or the main memory 60 when the requested scale ratio by the display area movement request signal crosses the scale ratio of the second hierarchy 34. Read ahead, decode and write to buffer memory 70.

  The same applies to the prefetch processing in the vertical and horizontal directions of the image. Specifically, a prefetch boundary is set for the image data developed in the buffer memory 70, and the prefetch process is started when the display position by the image change request signal crosses the prefetch boundary. By doing so, it is possible to realize a mode in which the moving image reproduction proceeds while smoothly changing the resolution and the display position in response to the user's request for moving the display area.

  FIG. 5 shows in detail the configuration of the control unit 100a having a function of displaying moving images using moving image data having a hierarchical structure in the present embodiment. The control unit 100a includes an input information acquisition unit 102 that acquires information input by the user from the input device 20, a frame coordinate determination unit 110 that determines frame coordinates of a region to be newly displayed, and compression of a video stream to be newly loaded. A load stream determination unit 106 that determines data and a load unit 108 that loads a necessary moving image stream from the hard disk drive 50 are included. The control unit 100a further includes a decoding unit 112 that decodes the compressed data of the moving image stream, and a display image processing unit 114 that draws an image frame.

  In FIG. 5 and FIG. 10 to be described later, each element described as a functional block for performing various processes can be configured by a CPU (Central Processing Unit), a memory, and other LSIs in terms of hardware. Specifically, it is realized by a program loaded in a memory. As described above, the control unit 100 includes one PPU and a plurality of SPUs, and each functional block can be configured by the PPU and the SPU individually or in cooperation. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  The input information acquisition unit 102 acquires request contents such as moving image reproduction start / end and display area movement input by the user to the input device 20 and notifies the frame coordinate determination unit 110 of them. The frame coordinate determination unit 110 determines the frame coordinates of a region to be newly displayed in accordance with the frame coordinates of the current display region and the display region movement request signal input by the user, the load stream determination unit 106, the decoding unit 112, the display The image processing unit 114 is notified.

  Based on the frame coordinates notified from the frame coordinate determination unit 110, the load stream determination unit 106 identifies compressed video data to be newly loaded from the hard disk drive 50 to the main memory 60, and issues a load request to the load unit 108. Issue. As will be described later, the hierarchical data of the present embodiment individually holds a moving image stream for each tile image sequence obtained by spatially dividing the frame image sequence constituting each layer into the same size.

  Therefore, in addition to the correspondence between the scale ratio and the hierarchy used for display, the spatial coordinates in each hierarchy, the identification information of the moving picture stream including the image data corresponding to the coordinates, and the storage area thereof are associated in advance. Based on the information, the load stream determination unit 106 acquires necessary moving image stream identification information. If the compressed data of the corresponding video stream has not been loaded, a load request is issued to the load unit 108. Further, even if the frame coordinates do not change, it is requested that the compressed data of the necessary moving image stream is sequentially loaded according to the progress of the moving image.

  The load stream determination unit 106 identifies a video stream that is predicted to be necessary in addition to the video stream necessary for frame drawing at that time by the pre-read process described above, and issues a load request to the load unit 108. Good. The load stream determination unit 106 may make a load request at a predetermined timing in a state where the load unit 108 is not performing the load process, for example, at a predetermined time interval or when the user inputs a display area movement request. The load unit 108 performs load processing from the hard disk drive 50 in accordance with a request from the load stream determination unit 106. Specifically, the storage area is specified from the identification information of the moving picture stream to be loaded, and the data read from the storage area is stored in the main memory 60.

  Based on the frame coordinates at each time, the decoding unit 112 reads and decodes the necessary moving picture stream data from the main memory 60 and sequentially stores the data in the buffer memory 70. The decoding target may be a moving image stream unit, and when the frame coordinate area determined by the frame coordinate determining unit 110 extends over a plurality of moving image streams, the plurality of moving image streams are decoded. The display image processing unit 114 reads the data of the corresponding image frame from the buffer memory 70 based on the frame coordinates at each time, and draws it in the frame memory of the display processing unit 44.

  In a mode that allows movement of the display area including enlargement / reduction during playback of one movie, all layers share the time axis, and frame drawing is seamless regardless of whether the layer of the movie data to be used has been switched or not. It is desirable to progress. Therefore, as described above, hierarchical data is generated using an image frame as a moving image stream in units of tile images. Thereby, since an area necessary for one display and data predicted to be necessary thereafter can be preferentially loaded and decoded, it is possible to improve the efficiency of processing necessary until frame drawing. It is also desirable to prepare data in a state where random access is possible over time.

  Since the moving image data to be processed in this embodiment has four-dimensional parameters such as three-dimensional frame coordinates including the scale direction and time, the unit for generating the moving image stream and the overall configuration are It can be appropriately changed according to the compression method and the content of the moving image. 6 to 9 show examples of the structure of moving image data to be processed in the present embodiment.

  In these figures, a triangle represents moving image hierarchical data, and a rectangular parallelepiped represents one moving image stream. Each hierarchical data consists of three hierarchies of the zeroth hierarchy, the first hierarchy, and the second hierarchy, but the number of hierarchies is not limited to that. As described above, one moving image stream is generated for each tile image obtained by dividing the image frame of each layer into the same size. In these examples, the size of the 0th layer image is set as the size of the tile image.

  First, the moving image data structure 200 shown in FIG. 6 includes one layer data 201 in which each layer is one moving image stream for each tile image from the start to the end of the moving image. Here, since the tile images that are image frames of the respective moving image streams have the same size as described above, the 0th layer is one moving image stream 202a, the first layer is 4 moving image streams 202b, and the second layer is 16 The video stream 202c and the like.

  In the case of the moving image data structure 200 of FIG. 6, the length of the moving image stream in the time direction varies depending on the length of the original moving image, that is, the number of original image frames. Therefore, it is advantageous when the number of image frames is originally small, or when using a compression method capable of long-term data compression and random access, such as MPEG (Moving Picture Experts Group) with all frames as I pictures. It is.

  The moving image data structure 204 shown in FIG. 7 is composed of one hierarchical data 205 in which moving image data is divided by a predetermined number of image frames, and each layer is a plurality of moving image streams in the time axis direction for each tile image. That is, the moving image stream in FIG. 6 is obtained by dividing each moving image stream shown in FIG. 6 with respect to the time axis which is the vertical direction of the drawing. In this example, the moving picture stream of FIG. 6 is divided into six moving picture streams. Therefore, the 0th layer is composed of 1 × 6 moving image streams 206a, the first layer is composed of 4 × 6 moving image streams 206b, the second layer is composed of 16 × 6 moving image streams 206c, and the like. This structure is used when a compression method that performs compression in units of a fixed number of image frames is used.

  The moving image data structure 208 shown in FIG. 8 has a configuration in which moving image data is divided by a predetermined number of image frames, and different hierarchical data 210a, 210b, and 210c are generated for each moving image stream generated in that unit. That is, each hierarchical data 210a, 210b, 210c is composed of one moving image stream in the time axis direction for each layer as in FIG. 6, but each moving image stream has a fixed number of image frames. For example, in the hierarchical data 210a, the 0th layer is composed of one moving image stream 212a, the first layer is composed of 4 moving image streams 212b, and the second layer is composed of 16 moving image streams 212c.

  In the case of the moving picture data structure 208 in FIG. 8, since it is composed of a plurality of hierarchical data in the time axis direction, only a certain scene is replaced with another hierarchical data, or hierarchical data is inserted or deleted. Easy video editing in any direction. In addition, since the number of image frames in each video stream is fixed, the data size is easy to estimate. For example, if the compression method described later is applied, the data of each moving picture stream can have the same structure as the data of the tile image when the still picture has the same hierarchical structure. Coexistence with a still image such as an image display or a partial image as a still image is facilitated.

  The moving picture data structure 214 shown in FIG. 9 has a configuration in which the moving picture data is divided by a predetermined number of image frames, and the moving picture stream generated in that unit is further divided into a predetermined number of pieces as different hierarchical data 216a, 216b, and 216c. That is, each hierarchical data 216a, 216b, 216c is composed of a plurality of moving image streams in the time axis direction for each layer as in FIG. 7, but the number is fixed regardless of the length of the moving image. In this case, the hierarchy data is divided by using two.

  For example, in the hierarchical data 216a, the 0th layer includes 1 × 2 moving image streams 218a, the first layer includes 4 × 2 moving image streams 218b, and the second layer includes 16 × 2 moving image streams 218c. Also in this case, it is easy to estimate and adjust the data size of each layer constituting one layer data, and it is easy to edit a moving image in the time axis direction by replacing the layer data.

  The moving image data structures shown in FIGS. 6 to 9 all retain the moving image stream so as to cover the entire area of the image frame in each layer. However, some moving image streams depend on the redundancy of the moving image. May be omitted from the moving image data and replaced with a moving image stream of a different level. FIG. 10 schematically shows the data structure of a moving image when a moving image stream of a part of the hierarchy is replaced with a moving image stream of another layer. The way of representing the data structure is the same as in FIG. In the hierarchical data 222 shown in the figure, the moving picture stream corresponding to the area 224 is omitted.

  Compared with the hierarchical data 201 shown in FIG. 6, the number of moving image streams is smaller in the first hierarchy 228 and the second hierarchy 230. This is a video stream from which this difference is omitted. In this case, the area represented by the omitted moving image stream has no data in the hierarchy. Therefore, when displaying the corresponding area at a scale ratio that should use data of such a hierarchy, the hierarchy is held up to the hierarchy holding the data of the applicable area, in the example of FIG. And draw.

  Such an aspect can be applied to a case where there is a region that does not require detailed information in the image frame, for example, a region composed of almost a single color such as sky, sea, or lawn. Thus, the presence or absence of redundancy in the image frame can be detected by image analysis. For example, for each image frame at each time, a difference image between an image obtained by enlarging an image frame on the low resolution side and an image on the high resolution side is generated, and an area where the difference value is equal to or less than a predetermined threshold is detected. . Then, among the video streams included in the area, the high-resolution layer video stream is excluded from the video data.

  By doing so, the size of the moving image data can be kept small, and a part of the loading processing of the moving image stream can be omitted. In such a case, in the information associating the three-dimensional coordinates determined by the hierarchical data and the moving image stream, identification of the moving image stream of the upper layer used in an enlarged manner with respect to the coordinates of the area corresponding to the excluded moving image stream Drawing is possible by associating information and adding information such as an enlargement ratio.

  The example of FIG. 10 is an aspect realized by the present embodiment having both the feature that the moving image data has a hierarchical structure and the feature that the frame image is spatially divided and the moving image stream is individually generated. In other words, by dividing the frame image into tile images, the data retention format can be locally changed, and data with a lower resolution can be used instead, so some data is omitted. Data size can be reduced. With the same idea, only a part of moving image streams may be thinned out to reduce the number of image frames, thereby reducing the data size.

  In this way, the time resolution of the area handled by the moving picture stream is reduced, but it is effective for moving pictures including areas with little temporal change such as the background. Similar to the above-described spatial redundancy, the temporal redundancy at this time can be specified by, for example, detecting a region having a difference value equal to or less than a predetermined threshold among the difference images between adjacent image frames. Similarly, some moving picture streams can be replaced with still images.

  Also, the compression method may be different for each video stream. Furthermore, without sharing the time axis in the hierarchical data, various image representations can be realized by intentionally shifting the time axis in predetermined units such as for each hierarchy, for each moving picture stream, for each pixel column in the image. It may be.

  As described above, the hierarchical structure of the moving image data to be displayed in the present embodiment is not particularly limited with respect to the compression method of each moving image stream, such as JPEG (Joint Photographic Experts Group), MPEG, S3TC (S3 Texture Compression), etc. Any of the existing methods may be applied. However, in order to enable seamless movement of the display area, including layer switching, spatial and random access is possible, and both image quality and decoding throughput are maintained even for high-definition images. It is desirable to be able to do it.

  Next, a method for compressing a moving picture stream in units of a fixed number of image frames, which can be applied to the moving picture data structure shown in FIGS. The compression method can be applied not only to a plurality of moving image streams constituting hierarchical data but also to a single moving image stream. An apparatus that implements this compression technique can also be realized with the same configuration as the image processing apparatus 10 shown in FIG. Hereinafter, the description will be given focusing on the configuration of the control unit 100.

  FIG. 11 shows in detail the configuration of the control unit 100b having the moving image data compression function and the hard disk drive 50 in the present embodiment. The control unit 100b includes a YCbCr conversion unit 120 that converts a color space of an image frame constituting a moving image stream to be compressed into YCbCr, an image division unit 122 that generates a coding unit by space-time dividing the converted image sequence, and A compression encoding unit 124 that performs compression encoding processing by quantizing the image data for each of the divided encoding units is included.

  The hard disk drive 50 includes a moving image stream storage unit 126 that stores a moving image stream to be compressed including individual image frame sequences, a division pattern storage unit 128 that stores division patterns when the image dividing unit 122 divides the image sequence, and A compressed data storage unit 130 for storing compressed data generated by compression encoding by the compression encoding unit 124 is included.

  The YCbCr conversion unit 120 sequentially reads out the data of the image frames constituting the moving image stream to be compressed from the moving image stream storage unit 126. Then, by converting the RGB value, which is the pixel value of each image frame, into luminance Y and color differences Cb and Cr, a Y image, a Cb image, and a Cr image having the respective values as pixel values are generated. An existing method can be applied to the conversion of the color space from RGB to YCbCr. Since a Y image, a Cb image, and a Cr image are generated from one image frame, a Y image sequence, a Cb image sequence, and a Cr image sequence are generated for a plurality of image frames that form a moving image stream.

  The image dividing unit 122 first reduces each Cb image and Cr image at a predetermined ratio among the Y image sequence, Cb image sequence, and Cr image sequence generated by the YCbCr conversion unit 120. Then, the Y image sequence, the Cb image sequence, and the Cr image sequence are spatiotemporally divided by the division patterns stored in the division pattern storage unit 128. A unit generated by the division is referred to as “coding unit”.

  Although details will be described later, since the optimal division pattern differs depending on the content of the image, the image division unit 122 may perform a process of selecting an optimum pattern from a plurality of division patterns stored in the division pattern storage unit 128. . Note that the Cb image and the Cr image reduced in the subsequent processing are handled as a set for each corresponding frame. Hereinafter, such a set of Cb image and Cr image is simply referred to as “CbCr image”.

  The compression encoding unit 124 includes a palette representing two representative values for each encoding unit of the Y image and the CbCr image, and a plurality of intermediate values obtained by linearly interpolating the two representative values and the representative value. By generating an index that designates either one for each pixel, the image data is quantized and compression-coded. Thereby, a palette and an index are generated for each encoding unit of the Y image sequence and each encoding unit of the CbCr image sequence.

  FIG. 12 schematically illustrates a moving image stream compression procedure performed by the image processing apparatus 10 including the control unit 100b. The moving image stream 250 to be compressed may correspond to, for example, a moving image stream indicated by a rectangular parallelepiped in FIGS. The moving image stream 250 is composed of image frames of RGB images. In this compression method, the moving picture stream 250 is compressed every predetermined number of image frames, that is, every 8 frames in the example of FIG.

  First, the YCbCr converter 120 further divides the image frame for eight frames into a predetermined size and determines a processing unit in a three-dimensional space of the image plane (x, y) and the time axis t. In the illustrated example, data of 8 pixels × 8 pixels × 8 frames is used as a processing unit 252. Next, eight Y image sequences 254 and CbCr image sequences 256 are generated from the eight RGB images included in the processing unit 252 (S10).

  Here, as described above, the CbCr image sequence 256 is an image sequence obtained by reducing the Cb image and Cr image directly obtained from the original RGB image to ½ size in both the vertical and horizontal directions. Therefore, the Y image row 254 has 8 image frames of 8 pixels × 8 pixels, and the CbCr image row 256 has 8 frames of images obtained by connecting a Cb image of 4 pixels × 4 pixels and a Cr image of 4 pixels × 4 pixels. .

  Next, the image dividing unit 122 performs space-time division on the Y image sequence 254 and the CbCr image sequence 256 with any one of the division patterns stored in the division pattern storage unit 128 to form a coding unit (S12). ). In the example of the figure, an image block obtained by spatially dividing each image frame of the Y image sequence 254 and the CbCr image sequence 256 with the same size of 4 horizontal pixels × 2 vertical pixels is divided into two images adjacent in the time direction. Data of 4 pixels × 2 pixels × 2 pieces divided for each frame is used as an encoding unit.

  Since the Y image sequence 254 is 8 pixels × 8 pixels as described above, each image frame has “A”, “B”, “C”, “D”, “E”, “F”, “G”, “ The image block “A” is divided into eight image blocks “H”, and the image block “A” of the first frame and the image block “A” of the second frame form an encoding unit 258 (shaded area). The same applies to the other image blocks and image frames. As a result, for the Y image sequence 254, the number of spatial divisions 8 × the number of time divisions 4 = 32 encoding units are formed.

  On the other hand, since the CbCr image sequence 256 is 4 pixels × 4 pixels for both the Cb image and the Cr image, the former is divided into two image blocks of “I” and “J”, and the latter of “K” and “L”. The image blocks “I” and “K” of the first frame and the image blocks “I” and “K” of the second frame form an encoding unit 260 (shaded area). The same applies to the other image blocks and image frames. As a result, the coding unit of the space division number 2 × time division number 4 = 8 is formed for the CbCr image sequence 256.

  The compression encoding unit 124 generates palette and index data for each encoding unit. The palette and the index are basically the same as the palette and the index generated from the RGB image in the S3TC texture compression method. On the other hand, in the present embodiment, the number of parameter dimensions is different from that of general S3TC. FIG. 13 schematically shows a method for generating palette and index data from the encoding unit 258 of the Y image sequence 254.

  When the pattern shown in FIG. 12 is used for division, the encoding unit 258 includes 4 × 2 × 2 = 16 pixels. In the figure, the pixels are schematically shown as circles. When a sample value of luminance Y that each pixel has as a pixel value is represented on the axis of luminance Y, a distribution 262 is obtained. Of the 16 samples plotted with the distribution 262, two representative values are selected. For example, the minimum value (min) and the maximum value (max) are selected as representative values, and data holding the binary values is used as a palette. Furthermore, on the axis of luminance Y, the luminance Y value that internally divides the line segment between the minimum value and the maximum value by 1: 2 is the first intermediate value (mid1), and the luminance Y value that is internally divided by 2: 1. Assuming that the second intermediate value (mid2) is used, data that stores information specifying any one of the four values of the minimum value, the first intermediate value, the second intermediate value, and the maximum value is used as an index.

  That is, for one encoding unit 258 of the Y image sequence 254, the palette is 8 bits × 2 values = 2 bytes representing the luminance Y, and the index is information 2 bits × 16 pixels = 4 values representing identification numbers 0-3 = It becomes 4 bytes of data. As described above, the Y image sequence 254 that is one processing unit is composed of 32 encoding units. Therefore, in the entire Y image sequence 254, the palette is 32 × 2 bytes = 64 bytes, and the index is 32 × 4 bytes = The data is 128 bytes.

  FIG. 14 schematically shows a method of generating palette and index data from the encoding unit 260 of the CbCr image sequence 256. When divided in the pattern shown in FIG. 12, the coding unit 260 includes 4 × 2 × 2 = 16 pixels in each of the Cb image and the Cr image. Therefore, when a sample value of color difference having (color difference Cb, color difference Cr) as an element in the corresponding pixels of both images is represented on a two-dimensional plane having the axes of color difference Cb and color difference Cr, distribution 264 is obtained. become.

  Of the 16 samples plotted in this distribution 264, two representative values are selected. For example, when the distribution 264 is approximated by a straight line, the color difference at the left end and the right end of the straight line is set as a representative value as a minimum value (min) and a maximum value (max), respectively. Then, the data holding the binary values is set as a palette. At this time, each representative value is a two-dimensional parameter having (color difference Cb, color difference Cr) as an element. On the approximate straight line, the color difference that internally divides the line segment between the minimum value and the maximum value by 1: 2 is the first intermediate value (mid1), and the color difference that internally divides by 2: 1 is the second intermediate value (mid2). In this case, data that stores information specifying any one of the four values of the minimum value, the first intermediate value, the second intermediate value, and the maximum value is used as an index.

  That is, for one encoding unit 260 of the CbCr image sequence 256, the palette is 2 elements of color difference Cb and Cr × 8 bits representing each color difference × 2 values = 4 bytes, and the index is a 4-value identification number 0-3. Information to be expressed is 2 bits × 16 pixels = 4 bytes of data. As described above, the CbCr image sequence 256, which is one processing unit, is composed of 8 coding units. Therefore, in the entire CbCr image sequence 256, the palette is 8 × 4 bytes = 32 bytes and the index is 8 × 4 bytes = The data is 32 bytes.

  When compressed in this way, an RGB image of 8 pixels × 8 pixels × 8 frames in one processing unit is 256 in total, including a palette of 64 bytes and an index of 128 bytes for a Y image sequence, and a palette of 32 bytes and an index of 32 bytes for a CbCr image sequence. It becomes a byte. That is, the data is 0.5 bytes per pixel.

  When a RGB image of 4 pixels × 4 pixels is compressed using S3TC, the palette is 2 bytes × 2 values = 4 bytes representing the RGB value, and the index is information representing the identification number of 4 values among the RGB values as 0 to 3 Since 2 bits × 16 pixels = 4 bytes of data, the data after compression is 8 bytes / 16 pixels = 0.5 bytes per pixel, which is the same as the data size after compression by the compression method described above. . Therefore, by compressing moving image data in such processing units, still images and moving images are taken into consideration in terms of the unit of data loaded from the hard disk drive 50 to the main memory 60, the size of the cache line in the main memory 60, and the like. Can be treated equally.

  In the present embodiment, the RGB image is decomposed into a Y image holding a one-dimensional parameter and a CbCr image holding a two-dimensional parameter, and then a palette and an index are generated. Therefore, in the case of a one-dimensional Y image, all sample values are distributed on a straight line, and in a two-dimensional CbCr image, the sample deviating from the approximate line is only in the normal direction of the approximate line. Therefore, compared with a general S3TC method in which an RGB image holding three-dimensional parameters is approximated by a straight line and quantized, the quantization error can be suppressed small.

  In the division pattern of FIG. 12, the moving picture stream is divided into 4 horizontal pixels × 2 vertical pixels × 2 frames to be an encoding unit. As described above, this division pattern may be adaptively changed according to the content of the image. FIG. 15 shows a variation of a pattern for dividing one processing unit. Pattern (A), pattern (B), pattern (C), and pattern (D) are shown from the left end of the figure, and each of the upper Y image sequence and the lower CbCr image sequence indicates a space division delimiter with a straight line. One coding unit is represented by shading.

  Pattern (A) is a pattern divided into 4 horizontal pixels × 4 vertical pixels × one frame. The pattern (B) is the same as the pattern shown in FIG. Pattern (C) is a pattern divided into 2 horizontal pixels × vertical 2 pixels × 4 frames, and pattern (D) is a pattern divided into 2 horizontal pixels × 1 vertical pixel × 8 frames.

  In each of these patterns, since one processing unit is 16 pixels for the Y image sequence and 16 pixels × 2 for the CbCr image sequence, the number of samples at the time of quantization is as shown in FIG. 13 and FIG. The same. On the other hand, the more detailed time division is performed from the pattern (D) to the pattern (A), and the more detailed space division is performed from the pattern (A) to the pattern (D). Such a division pattern is prepared, and the division pattern is selected according to the characteristics of the image, such as whether it has redundancy in the spatial direction or redundancy in the temporal direction.

  Specifically, if the image has spatial redundancy, such as the sky and turf contain many areas close to a single color, the pixel values tend to be more uniform with respect to the space, and the number of space divisions can be reduced. Since an error due to quantization is difficult to be included, a divided pattern close to the pattern (A) is selected. On the other hand, if the image has temporal redundancy, such as when a fixed-point view of a scene with little motion is observed, the pixel values are likely to be uniform in the time direction, and even if the number of time divisions is reduced, errors due to quantization are not easily included. Therefore, a division pattern close to the pattern (D) is selected.

  For example, in the case of the pattern (D), one encoding unit has only two pixels in the spatial direction. If there is no time change for 8 frames included in the same encoding unit, the two representative values held in the palette represent the original pixel values as they are, and the quantization error is zero. When the RGB image is compressed by the S3TC method, the RGB data held in the palette is reduced from the original 24 bits to 16 bits, so that sufficient gradation cannot be obtained when decoding, such as image quality. A decrease may occur. In this embodiment, an 8-bit palette is prepared for each of the luminance Y and the color differences Cb and Cr, so that there is a high possibility that the original image quality can be maintained.

  The division pattern storage unit 128 stores four types of division patterns (A) to (D) and information for identifying them, for example, four identification numbers 0, 1, 2, and 3 in association with each other. Keep it. The image division unit 122 performs all the division patterns stored in the division pattern storage unit 128 for each image sequence generated by the YCbCr conversion unit 120, and selects a division pattern with the least error from the original image.

  In actuality, this processing causes the compression encoding unit 124 to perform compression encoding of the image sequence when divided by each division pattern, and compares the image obtained by decoding each compressed data with the image before compression for each image frame. To do. Then, a division pattern with a small difference may be selected. The image dividing unit 122 notifies the identification number of the selected division pattern to the compression encoding unit 124, and the compression encoding unit 124 includes the information of the identification number in the generated compressed data as final compressed data. And stored in the compressed data storage unit 130.

  The division pattern may be different for each region in the image. The procedure for selecting the division pattern for each region may be the same as described above. Then, the image division unit 122 generates a map in which the identification number of the selected division pattern is associated with the region, and includes the map in the final compressed data. FIG. 16 shows an example of the data structure of the division pattern map. The example in the figure shows a case where one moving picture stream is composed of image frames of 256 pixels × 256 pixels. When the four types of division patterns shown in FIG. 15 can be set, the minimum unit that can set the division pattern is 8 pixels × 8 pixels × 8 frames, which is one processing unit.

  If a division pattern is set for each minimum unit, as shown in FIG. 16, an identification number of the division pattern, that is, 0 to 3 for each region of 8 pixels × 8 pixels with respect to an image frame of 256 pixels × 256 pixels. Associate values. As a result, the division pattern map 270 is 32 × 32 × 2 bits = 256 bytes of information. If such a division pattern map 270 is added every 8 frames, the division pattern can be made different in the time direction.

  The example of FIG. 16 is a case where the division pattern is set for each minimum unit of 8 pixels × 8 pixels. Similarly, for example, every 16 pixels × 16 pixels, every 64 pixels × 32 pixels, etc. A division pattern may be set for each region obtained by connecting regions of 8 pixels × 8 pixels in the direction. Also, the setting unit itself can be set variously, such as setting one division pattern for all the areas. The division pattern map can be generated by decoding the data that was actually compressed and encoded as described above and with a small error from the original image. In addition, the division pattern map can be set by the test image having the same contents and the division pattern set there You may have prepared.

  Next, a procedure in which the compression encoding unit 124 stores the compression encoded data in the compressed data storage unit 130 will be described. The compressed data generated in the present embodiment is composed of palettes and indexes as in the S3TC texture compression method. Therefore, the decoding process can use a general GPU shading function included in the control unit 100 of the image processing apparatus 10 of FIG. 4 as it is.

  Therefore, the index and palette generated by quantizing the data of the Y image sequence and the index and palette generated by quantizing the data of the CbCr image sequence can be read and decoded in the same manner as a normal texture image. It is desirable to do. Therefore, when storing compressed data, the quantized data of the Y image sequence and the quantized data of the CbCr image sequence representing the same region are combined into one, so that the pixels can be restored with a small number of data accesses.

  FIG. 17 is a diagram for explaining the arrangement of compressed data in the compressed data storage unit 130. As described above, in order to treat the data quantized after being decomposed into the Y image sequence and the CbCr image sequence in the same manner as the compressed data of the RGB image, it is desirable to store the data representing the same region together. Therefore, in this embodiment, the compressed data 280 for the Y image sequence and the compressed data 282 for the CbCr image sequence representing the same area are collected as one storage unit.

  In the figure, among the compressed data 280 for the Y image sequence, the rectangular parallelepiped denoted as “I” is generated from one encoding unit, and the rectangular parallelepiped denoted as “P” is generated from one encoding unit. It is a pallet. The same applies to the compressed data 282 for the CbCr image sequence. As described above, the index and palette of the Y image sequence are 4 bytes and 2 bytes of data per encoding unit, respectively. Both the index and palette of the CbCr image sequence are 4 bytes of data per encoding unit.

  Therefore, as shown in FIG. 17, the data of 4 encoding units of the Y image sequence and 1 encoding unit of the CbCr image sequence representing the same area are arranged in a storage area of 4 bytes in depth. Here, in the compressed data 280 for the Y image sequence, each palette is 2-byte data, so by arranging two in the depth direction as shown in the figure, 2 × 4 × 4 bytes of data in the vertical direction. It becomes. Here, the Y image sequence and the CbCr image sequence representing the same area are, for example, the image blocks “A”, “B”, “C”, “D” of the Y image and the image block “I” of the Cb image in FIG. , The image block “K” of the Cr image.

  When the compressed data is collected in this way, it can be stored as it is in the storage area 284 for storing RGBA image data of 2 pixels in the vertical direction × 4 pixels in the horizontal direction. As described above, 32 encoding units are formed for the Y image sequence and 8 encoding units are formed for the CbCr image sequence per processing unit of 8 pixels × 8 pixels × 8 frames, so that there are 8 storage units per processing unit. Individually formed. Since one storage unit has the same data size as an RGBA image of 2 pixels in the vertical direction and 4 pixels in the horizontal direction, the data is equivalent to an RGBA image of 8 pixels × 8 pixels per processing unit. This feature is the same for any division pattern shown in FIG.

  FIG. 18 schematically shows the transition of data when the compression encoding process described so far is applied to the entire moving image stream. The moving image stream is composed of image frames of RGB images of 256 pixels × 256 pixels and is compressed in units of 8 frames. First, 8 image frames are divided into processing units of 8 pixels × 8 pixels (S20). As a result, 32 processing units are formed in the vertical and horizontal directions.

  Next, as shown in FIG. 12, YCbCr conversion is performed on each processing unit to generate a Y image and a reduced CbCr image, and each index is divided into coding units to generate an index and a palette. Collectively, eight storage units are generated per processing unit (S22). As a result, the RGB images for 8 frames are compressed into 1 frame of RGBA image having the same number of pixels.

  Here, a method of embedding the above-described division pattern map in the compressed data will be described. As shown in FIG. 17, four Y image sequence palettes are stored in one storage unit. Each palette stores binary values that are representative values of luminance Y. Therefore, 2-bit information for identifying the four division patterns is embedded using two palettes arranged side by side in the depth direction among the four palettes included in one storage unit.

  FIG. 19 is a diagram for explaining a method of embedding the identification numbers of the divided patterns in the two pallets. Among the two pallets, the binary values held by the first pallet 290 are “Pa0” and “Pa1” in order from the head address in the front of the figure, and the binary values held by the second pallet 292 are “Pb0” in the address order. ”And“ Pb1 ”. Here, information of 2 bits in total is represented by the magnitude relationship between “Pa0” and “Pa1” and the magnitude relationship between “Pb0” and “Pb1”.

  For example, 1-bit information is represented by “1” if “Pa0” of the first palette 290 is larger than “Pa1” and “0” otherwise. Similarly, if “Pb0” of the second pallet 292 is larger than “Pb1”, “1” is set, and “0” is expressed otherwise. The binary value held by the palette does not affect the decoding process, whichever is stored at the previous address. Therefore, the identification number of the divided pattern can be embedded in the pallet by changing the address in which the larger value is stored in each pallet according to the identification number of the divided pattern.

  In this way, the division pattern map can be included in the compressed data without being generated separately from the main body of the compressed data, and the data size can be suppressed as a whole. In addition, since it is embedded for each compressed data in the corresponding area, the efficiency in referring is good. As described above, since the division pattern is a minimum of one processing unit (= 8 storage units), it is only necessary to embed one division pattern in any pallet pair among the eight storage units. On the other hand, the same division pattern may be embedded in all 16 pairs of pallets included in the eight storage units.

  When decoding the compressed data in which the division pattern is embedded in this way, first, for each processing unit, the Y image sequence palette in which the division pattern is embedded is read, and the identification number of the division pattern set in the processing unit Is identified. As a result, the pixel can be associated with the storage location of the index and pallet containing the data necessary for drawing the pixel. Accordingly, the index and palette of the Y image sequence and the index and palette of the CbCr image sequence corresponding to the drawing target pixel may be read and decoded.

  The decoding process can be basically performed in the same manner as S3TC. That is, an intermediate value that interpolates the representative value held by each palette is generated, and the representative value or the intermediate value is set as the pixel value of each pixel according to the designation in the index. On the other hand, since the palette and the index are generated for each encoding unit in the present embodiment, the determined pixel value is based on the arrangement of the encoding units in the image sequence corresponding to the division pattern, and in the spatial direction and the temporal direction. The Y image sequence and the CbCr image sequence are restored by reconstructing the pixel arrangement. Then, by enlarging the CbCr image to generate a Cb image and a Cr image, a YCbCr image corresponding to the original image frame is obtained.

  According to the present embodiment described above, hierarchical data is generated by hierarchizing a plurality of video streams representing image frames constituting a video at different resolutions, and the display area is moved in response to a viewpoint movement request from the user. To display the video. By switching the hierarchy of data used for frame drawing according to the required scale ratio, even a general high-definition image or a moving image with a resolution higher than that can be enlarged to confirm details or have a bird's-eye view. Requests can be sequentially received and displayed with good responsiveness.

  The moving image stream constituting each layer of the layer data is configured with image frames of the same size in any layer. As a result, the higher the resolution, the greater the number of video streams that make up one hierarchy. By aligning the sizes of the image frames in this way, it is possible to equalize the processing such as loading and decoding at the time of display regardless of the hierarchy, and to perform efficient drawing processing suitable for the locality of the display target area.

  In addition, by configuring one image with a plurality of moving image streams, adjustments such as making the frame rate different for each moving image stream or setting some areas as still images can be performed in view of the spatial locality of the image. In addition, in an image of a certain resolution, when there is an area that can be used by enlarging the image of the lower resolution side, the moving picture stream itself responsible for the area can be omitted from the data.

  A moving image may be divided into a plurality of hierarchical data by being divided on the time axis, instead of being composed of a single hierarchical data throughout the entire story. In addition, the moving picture stream of each layer included in one hierarchical data may be one moving picture compressed data over the entire volume, or may be different moving picture compressed data for each predetermined number of image frames. In this way, the processing load when displaying moving images can be selected by appropriately selecting the number of hierarchical data, the data structure of the video stream in the hierarchical data, and the compression encoding format depending on the content of the moving image and the playback time. Therefore, an optimum display mode can be realized from various viewpoints such as required image quality.

  Furthermore, in this embodiment, the moving image stream is compression-coded for each predetermined number of image frames. At this time, an image in which the RGB image of the image frame constituting the original moving image stream is represented by luminance Y, color difference Cb, and Cr is generated. Then, after reducing the Cb image and the Cr image, each image sequence is divided into a predetermined size and a predetermined number of image frames to generate a coding unit. In this way, palette and index data are generated for each of the Y image sequence and the CbCr image sequence. The palette is binary data representing the representative value of each image, and the index is data for designating one of the intermediate value and representative value obtained by linear interpolation of the representative value for each pixel.

  The concept of palette and index is introduced in the S3TC compression method for texture RGB images, but in this embodiment, the binary value of the palette is 8 bits for all of luminance Y, color difference Cb, and color difference Cr. Image quality is unlikely to deteriorate due to retention. In addition, since the quantization is separately performed on the Y image sequence and the CbCr image sequence, the number of parameter dimensions is smaller and the quantization error is smaller than when the RGB three-dimensional parameters are quantized. In addition, by changing the combination of the number of space divisions and the number of time divisions when forming the coding unit, it is possible to flexibly provide a data structure adapted to the redundancy in the spatial direction and the redundancy in the time direction of the image.

  If the above compression method is used, the rendering process can be performed in the same manner as the texture mapping process by the GPU. Therefore, the video stream in the display area is read while the hierarchy is switched, and the image is rendered at a predetermined frame rate. High throughput that can be applied to hierarchically structured video data can be expected. Compared with the existing compression encoding method, for example, when decoding for each image frame using JPEG, the processing load of decoding tends to increase depending on the content of the image. In addition, MPEG requires decoding of I pictures for each of a plurality of moving picture streams. As a result, the processing load tends to increase, and if I pictures are reduced, there is a problem that latency tends to occur for random access in the time direction. .

  The compression coding technique in the present embodiment realizes high-speed drawing as compared with the above-described existing technique by realizing decoding with a GPU. As a result, a high-definition moving image can be displayed while reducing the processing load on the CPU. For this reason, it is possible to perform additional processing in the CPU, and the risk of dropping frames is reduced even in a device such as a portable terminal having inferior CPU processing performance. This feature can be said to be suitable for future technological trends in which data reading from a storage device is accelerated with the spread of SSD (Solid Sate Drive) and decoding processing is likely to become a bottleneck.

  As a result, this compression coding technology can achieve high-throughput rendering while maintaining image quality, and also enables temporal and spatial random access with low latency, so high-definition video can be displayed while changing the display area. By applying to the moving image data having a hierarchical structure used for the purpose, a more effective moving image display technique can be realized.

  The present invention has been described based on the embodiments. Those skilled in the art will understand that the above-described embodiment is an exemplification, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  DESCRIPTION OF SYMBOLS 1 Image processing system, 10 Image processing apparatus, 12 Display apparatus, 20 Input apparatus, 30 0th hierarchy, 32 1st hierarchy, 34 2nd hierarchy, 36 3rd hierarchy, 44 Display processing part, 50 Hard disk drive, 60 Main memory , 70 buffer memory, 100 control unit, 102 input information acquisition unit, 106 load stream determination unit, 108 load unit, 110 frame coordinate determination unit, 112 decoding unit, 114 display image processing unit, 120 YCbCr conversion unit, 122 image division unit , 124 compression encoding unit, 126 moving image stream storage unit, 128 division pattern storage unit, 130 compressed data storage unit.

Claims (22)

A data dividing unit that divides a data string in a three-dimensional parameter space to be compressed in the three-dimensional direction to form a coding unit;
For each coding unit formed by the data dividing unit, a palette that holds binary data as representative values, and a plurality of intermediate values determined by linear interpolation of the representative values and one of the representative values are designated. An index that holds information instead of the original data of the encoding unit, and a compression encoding unit that generates compressed data,
A data compression apparatus comprising: As the data string, the pixel value in each image frame of the moving image data having the pixel value with respect to the image frame plane and the time axis is generated for each image frame by converting the luminance Y and two color differences Cb and Cr. Y image having luminance Y as a pixel value, Cb image having color difference Cb as a pixel value, Y image sequence in which Cr images having a color difference Cr as a pixel value are arranged in chronological order, Cb image sequence, and YCbCr for generating a Cr image sequence A conversion unit;
The data division unit according to claim 1, wherein the data division unit divides the Y image sequence, the Cb image sequence, and the Cr image sequence generated by the YCbCr conversion unit, respectively, into space-time divisions to form an encoding unit. Data compression device.   The data dividing unit forms a pattern of a plurality of encoding units by dividing the data string with a plurality of division patterns prepared in advance, and causes the compression encoding unit to perform data compression with each pattern, 3. The data compression apparatus according to claim 1, wherein a division pattern with the least error is selected. The data division unit selects a division pattern for each predetermined unit area of the three-dimensional parameter space, and generates a division pattern map in which identification information of the selected division pattern is associated with the predetermined unit area.
The compression encoding unit generates the palette and the index for each encoding unit formed by a selected division pattern, and includes the division pattern map in the compressed data. 4. The data compression apparatus according to any one of 3.   The compression encoding unit generates a palette and an index for the luminance Y for each encoding unit of the Y image sequence, and calculates (color difference Cb, color difference Cr) for each encoding unit of the Cb image sequence and the Cr image sequence. The data compression apparatus according to claim 2, wherein a palette and an index are generated for the parameters of the parameter.   The data dividing unit reduces the Cb image and the Cr image at a predetermined magnification in the image plane direction before dividing, and the number of pixels included in the encoding unit is the Y image sequence, the Cb image sequence, and the Cr image sequence. 6. The data compression apparatus according to claim 2, wherein the space-time division is performed so as to be equal to each other.   The compression encoding unit generates a storage unit in which palettes and indexes generated for each encoding unit are grouped in a predetermined area unit of the original three-dimensional parameter space, and stores the compressed unit for each storage unit. The data compression apparatus according to claim 1, wherein the data compression apparatus is stored in the apparatus.   The compression encoding unit is configured to generate the division pattern map for each encoding unit by expressing the identification information of the division pattern by a binary magnitude relationship held in the palette and a storage order thereof. The data compression apparatus according to claim 4, wherein the data compression apparatus is embedded in the data compression apparatus.   The moving image data includes hierarchical moving image data formed by hierarchizing a plurality of image frame sequences representing image frames constituting one moving image at different resolutions in the order of resolution. 7. The data compression device according to claim 2, wherein the data compression device is a moving image stream in units of tile images divided into sizes. For each coding unit formed by dividing a data string in the three-dimensional parameter space in the three-dimensional direction, a palette that holds two of the pixel values as representative values, and a plurality of values determined by linear interpolation of the representative values A compressed data reading unit that reads compressed data associated with an index that holds information specifying either the intermediate value or the representative value instead of the original data of the encoding unit from the storage device;
The intermediate value is generated by linearly interpolating the representative value held by the palette, and the data included in each coding unit is determined as one of the representative value and the intermediate value according to the information held by the index. A decoding unit that reconstructs and generates an original data sequence based on the arrangement of the encoding units;
An output unit for outputting the generated data string;
A data decoding apparatus comprising: The compressed data reading unit corresponds to an image frame sequence constituting a moving image, a Y image sequence having a luminance Y as a pixel value, a Cb image sequence having a color difference Cb as a pixel value, and a Cr image sequence having a color difference Cr as a pixel value. As the data string,
The decoding unit generates data of the Y image sequence, the Cb image sequence, and the Cr image sequence by reconstructing an array of pixels based on the array of coding units,
11. The data according to claim 10, wherein the output unit outputs data of a YCbCr image sequence representing the image frame sequence based on the data of the Y image sequence, Cb image sequence, and Cr image sequence to a display device. Decoding device. The compressed data further includes a division pattern map that holds information for identifying a division pattern in the three-dimensional direction in a predetermined region unit of the three-dimensional parameter space,
The data decoding according to claim 10 or 11, wherein the decoding unit specifies the arrangement of the encoding units based on the division pattern indicated by the division pattern map, and reconstructs the original data arrangement based on the arrangement. apparatus. The compressed data is stored in a storage device in a storage unit in which a palette and an index for each encoding unit are grouped in a predetermined area unit of the original three-dimensional parameter space,
The data decoding according to any one of claims 10 to 12, wherein the compressed data reading unit specifies the storage unit for each region to be decoded and reads a palette and an index included in the storage unit. apparatus. Reading a data string in a three-dimensional parameter space to be compressed from a storage device;
Dividing the data string in the three-dimensional direction to form a coding unit;
For each encoding unit, a palette that holds two values of data as representative values, and a plurality of intermediate values determined by linear interpolation of the representative values and information specifying any one of the representative values are encoded. Generating an index to be held instead of the original data of the unit, and storing it in the storage device as compressed data;
A data compression method comprising: For each coding unit formed by dividing a data string in the three-dimensional parameter space in the three-dimensional direction, a palette that holds two of the pixel values as representative values, and a plurality of values determined by linear interpolation of the representative values Reading the compressed data associated with the index that holds the information specifying either the intermediate value or the representative value instead of the original data of the encoding unit from the storage device;
The intermediate value is generated by linearly interpolating the representative value held by the palette, and the data included in each coding unit is determined as one of the representative value and the intermediate value according to the information held by the index. Reconstructing and generating the original data sequence based on the arrangement of the encoding units;
Outputting the generated data string to an output device;
A data decoding method comprising: A function for reading a data string in a three-dimensional parameter space to be compressed from a storage device;
A function of dividing the data string in the three-dimensional direction to form a coding unit;
For each encoding unit, a palette that holds two values of data as representative values, and a plurality of intermediate values determined by linear interpolation of the representative values and information specifying any one of the representative values are encoded. A function of generating an index to be held instead of the original data of the unit and storing it in the storage device as compressed data;
A computer program for causing a computer to realize the above. For each coding unit formed by dividing a data string in the three-dimensional parameter space in the three-dimensional direction, a palette that holds two of the pixel values as representative values, and a plurality of values determined by linear interpolation of the representative values A function of reading compressed data in association with an index that holds information designating either the intermediate value or the representative value instead of the original data of the encoding unit from the storage device,
The intermediate value is generated by linearly interpolating the representative value held by the palette, and the data included in each coding unit is determined as one of the representative value and the intermediate value according to the information held by the index. A function of reconstructing and generating an original data sequence based on the arrangement of the encoding units;
A function of outputting the generated data string to an output device;
A computer program for causing a computer to realize the above. A function for reading a data string in a three-dimensional parameter space to be compressed from a storage device;
A function of dividing the data string in the three-dimensional direction to form a coding unit;
For each encoding unit, a palette that holds two values of data as representative values, and a plurality of intermediate values determined by linear interpolation of the representative values and information specifying any one of the representative values are encoded. A function of generating an index to be held instead of the original data of the unit and storing it in the storage device as compressed data;
The recording medium which recorded the computer program characterized by making a computer implement | achieve. For each coding unit formed by dividing a data string in the three-dimensional parameter space in the three-dimensional direction, a palette that holds two of the pixel values as representative values, and a plurality of values determined by linear interpolation of the representative values A function of reading compressed data in association with an index that holds information designating either the intermediate value or the representative value instead of the original data of the encoding unit from the storage device,
The intermediate value is generated by linearly interpolating the representative value held by the palette, and the data included in each coding unit is determined as one of the representative value and the intermediate value according to the information held by the index. A function of reconstructing and generating an original data sequence based on the arrangement of the encoding units;
A function of outputting the generated data string to an output device;
The recording medium which recorded the computer program characterized by making a computer implement | achieve. A data structure of a compressed video file,
The Y image sequence having luminance Y as a pixel value, the Cb image sequence having chrominance Cb as the pixel value, and the Cr image sequence having chrominance Cr as the pixel value corresponding to the image frame sequence constituting the moving image are respectively spatiotemporally divided. A palette that holds two of the pixel values as representative values generated for each encoding unit formed in this way, and a plurality of intermediate values determined by linear interpolation of the representative values and one of the representative values are designated. A data structure of a compressed moving image file, characterized in that an index for storing information for each pixel is associated and arranged in correspondence with an image area of the image frame.   21. The compressed moving image file according to claim 20, further comprising a division pattern map that holds information for identifying a division pattern at the time-space division for each predetermined image area of the image frame. data structure.   The recording medium which recorded the compressed moving image file which has the data structure of Claim 20 or Claim 21.

Images