RU2576518C2 - Device to generate data of moving image, device to display moving image, method to generate data of moving image, method to display moving image, moving image file data structure - Google Patents

Abstract

FIELD: physics, video.

SUBSTANCE: invention relates to a device for generation and reproduction of moving image data. It is proposed to configure frames of the moving image in the form of a hierarchical structure, where each frame is presented with multiple resolution levels, at the same time the resolution becomes higher sequentially of the zero layer 30, first layer 32, second layer 34, third layer 36, the zero layer 30 and the second layer 34 are set as the layer of the initial image, and the first layer 32 and the third layer 36 are set as the layer of a differential image in hierarchical data presenting the frame at the moment of time t1. In the case when the area 124a, which must be reproduced with resolution of the third layer 36, to the values of appropriate pixels of the differential image of the area 124a, stored by the third layer 36, they add values of appropriate pixels of the image of the appropriate area 126a, stored by the second layer 34, to produce an image increased to resolution of the third layer 36. A unit of compressed data generation compresses and codes hierarchical data of the moving image and records compressed and coded hierarchical data of the moving image in the memory.

EFFECT: provision of image processing, making it possible to reproduce moving images of high accuracy on the screen with high efficiency of changes whenever a user inputs an operation of image area reproduction, limiting the volume of processed data.

19 cl, 17 dwg

Description

Technical field

The present invention relates to a moving image data generating apparatus that generates moving image data, a moving image reproducing apparatus that reproduces a moving image on a screen, and also to a method for generating moving image data and a moving image reproducing method implemented by these devices.

Prior art

Home entertainment systems are proposed that can not only execute game programs, but also play moving images on the screen. In these home entertainment systems, the graphics processor generates 3D images using polygons (see, for example, Patent Document 1).

Regardless of whether the image is a moving image or a still image, increasing the reproducing efficiency of this image remains a constant and important problem. In this context, various technologies have been developed and put into practice in many areas, such as compression technology, transmission technology, image processing technology and on-screen image reproduction technology, which provide aesthetic perception of high-definition images in various scenes and in a known manner.

Related Art

Patent documents

Patent Document 1: US Patent 6,563,999

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

There is always a need to reproduce high-definition images with high-speed changes to these images in accordance with the requirements of the user. For example, the implementation of image playback on the screen in which the user could view the image at an arbitrary angle, or when the user from the entire reproduced image could enlarge the region of interest to him, and move the region displayed on the screen to another region with high speed changes will become possible if it will be possible to process large volumes of data in a short time, while ensuring random access to data. Thus, inevitably there is a need for further technological progress.

Based on such circumstances, the aim of the present invention is to provide image processing technology that allows you to play on the screen moving images of high definition with high speed changes when the user enters the operation to reproduce the image area, while limiting the amount of processed data.

Means of solving this problem

One embodiment of the present invention relates to a moving image data generating apparatus. The device for generating data of moving images generates hierarchical data of a moving image, where many sequences of images, which are representations of image frames forming one moving image at different resolution levels, are hierarchically organized in order of resolution to form a reproduced image on the screen, while switching the layer used to another layer in accordance with the level of resolution required in the image processing device that produces a moving image, and contains: a hierarchical data generation unit that generates image data of the corresponding layers of each image frame by stepwise reduction of each image frame, and the difference image data representing the difference between the enlarged image and the image on another layer, which is a representation of the same image frame with a resolution level lower than the resolution of said one of the layer and thereby forms a hierarchical moving image data; and a compressed data generating unit that compresses and encodes the hierarchical moving image data formed by said hierarchical data generating unit and records compressed and encoded hierarchical moving image data in a storage device.

Another embodiment of the present invention relates to a moving image reproducing apparatus. A moving image reproducing device comprises: a moving image data memory unit that stores hierarchical moving image data formed by a hierarchical organization in order of resolution levels of many image sequences that are representations of image frames forming one moving image with different resolution levels; an input information obtaining unit that sequentially receives demand signals for moving a reproduction area in a moving image that is displayed on a screen; and an image reproduction processing unit that provides reproduction of an image in accordance with a movement demand signal of each image frame, while switching the layer used to another layer of many layers included in the hierarchical data of the moving image, in accordance with the required resolution determined by the demand signal on the movement transmitted from the input information obtaining unit, wherein, image data of at least one layer of many layers forming hierarchically e the moving image data contains differential image data representing the difference between the enlarged image and the image on another layer, which is a representation of the same image frame with a resolution level lower than the resolution of the above-mentioned one layer, and the image processing processing unit restores the image enlarging and adding an image on said other layer in the case when the data used in the reproduction is differential data Images.

Another embodiment of the present invention relates to a method for generating moving image data. The method of generating moving image data provides the formation of hierarchical moving image data, which are hierarchically organized in the order of resolution levels of many image sequences, which are representations of image frames forming one moving image at different resolution levels, to form a reproduced image, switching the layer used to another layer in accordance with the level of resolution required in the processing device weave images that reproduces the moving image. This method of generating moving image data comprises: a stage at which moving image data comprising sequences of image frames presented with one resolution level is read from a storage device; a step in which image data of corresponding layers of each image frame is generated by several stepwise decreases of each image frame to several sizes; a step in which differential image data is included in the image data of at least one layer, which is the difference between the enlarged image and the image on another layer, which is a representation of the same image frame with a resolution level lower than the resolution of said one layer and thereby forms hierarchical data of a moving image; and a step in which the hierarchical data of the moving image is compressed and encoded, and said data is recorded in the storage device.

Another embodiment of the present invention relates to a method for reproducing moving images. A method for reproducing moving images comprises: a stage in which at least a part of the hierarchical data of a moving image is read out, which are hierarchically organized in order of resolution levels, many sequences of images that are representations of image frames forming one moving image at different levels of resolution and starting playback of the moving image on the display device using at least a portion erarhicheskih moving image data; the stage at which the signal is received requirements to move the playback area in the moving image that is displayed on the screen; a step in which a reproduced image is formed in accordance with a movement demand signal of each image frame, switching the used layer to another layer of many layers included in the hierarchical data of the moving image in accordance with the required resolution determined by the movement demand signal; and the stage at which the generated image is reproduced on the screen of the display device, wherein the image data of at least one layer of many layers forming the hierarchical data of the moving image, contains differential image data, which is the difference between the enlarged image and the image on another layer, which is a representation of the same image frame with a resolution lower than the resolution of the aforementioned single layer, and thereby forms hierarchical data moving image; where the stage at which the reproduced image is formed comprises image restoration by enlarging and adding an image on said other layer in the case where the data used in the reproduction is differential image data.

Another embodiment of the present invention relates to a data structure of a moving image file to be displayed on a display device. In this data structure, a correspondence is established between: a range of resolution levels determined by a user-entered operation concerning a playback area; and hierarchical data of the moving image, which are created by the hierarchical organization in the order of resolution of many sequences of images, which are representations of image frames forming one moving image at different resolution levels, and which are used by switching in accordance with the range of resolution levels, wherein the image data is at least one layer include difference image data representing the difference between the enlarged image and the image image on another layer, which is a representation of the same image frame with a resolution lower than the resolution of said one layer, and wherein the image on said other layer is enlarged and added, thereby restoring the image when it is displayed on the screen.

Arbitrary combinations of the structural components mentioned above, as well as the implementation of the invention in the form of methods, devices, systems, computer programs and other things, can also be used in practice in the form of additional forms of the present invention.

Advantageous Effect of the Invention

In accordance with the present invention, smooth reproduction of moving images with the possibility of high-speed execution of operations with the reproduced area entered by the user can be realized.

Brief Description of the Drawings

FIG. 1 is a drawing illustrating an environment in which an image processing system in accordance with the present embodiment can be used;

FIG. 2 shows an example of the appearance of an input device that can be used for the image processing system shown in FIG. 1;

FIG. 3 shows, at a conceptual level, the hierarchical data of a moving image that is an object of processing in accordance with this embodiment;

FIG. 4 shows a configuration of an image processing apparatus in accordance with the present embodiment;

FIG. 5 shows a detailed configuration of a control unit having a function for reproducing a moving image using moving image data that has a hierarchical structure in accordance with the present embodiment;

FIG. 6 schematically shows a state where some layers of moving image data having a hierarchical structure are represented by difference images, in accordance with the present embodiment;

FIG. 7 is a drawing illustrating an embodiment where the distribution of the source image layer and the differential image layer switches between several distribution configurations, in accordance with the present embodiment;

FIG. 8 shows an example of the distribution order of the layers of the original image and the layers of the differential image, in accordance with the present embodiment;

FIG. 9 shows an example of the distribution order of the layers of the original image and the layers of the differential image, in accordance with the present embodiment;

FIG. 10 shows another example of an order of distribution of layers of a source image and layers of a difference image, in accordance with the present embodiment;

FIG. 11 shows another example of an order of distribution of layers of a source image and layers of a difference image, in accordance with the present embodiment;

FIG. 12 shows a configuration of a moving image stream in the case where the frame order of the moving image is received exactly the same as the order of the moving image stream data in accordance with the present embodiment;

FIG. 13 shows a configuration of a moving image stream in the case where the same types of frame data are extracted, grouped and compressed from a sequence of frames, in accordance with the present embodiment;

FIG. 14 shows a configuration of a moving image stream in a case where mosaic image data that represents the same area on many layers is grouped and compressed in accordance with the present embodiment;

FIG. 15 shows a detailed configuration of a control unit having a function of generating compressed moving image data and a hard disk drive in accordance with the present embodiment;

FIG. 16 is a flowchart describing a processing procedure where the compressed moving image data generating apparatus generates compressed moving image data in accordance with the present embodiment; and

FIG. 17 is a flowchart describing a processing procedure where the reproducing apparatus reproduces a moving image in accordance with the present embodiment.

The method of carrying out the invention

According to the present embodiment, the display area of the image on the screen can be moved in accordance with the user's requirement during the reproduction of the moving image to move the viewpoint. Moving the viewpoint involves moving the viewpoint closer or further from the image plane, and, accordingly, the moving image increases or decreases during its playback on the screen. In such an embodiment, the more the varied range of resolution levels expands and the larger the image size becomes, the more difficult it becomes to smoothly display on the screen a moving image of the requested area with high speed of execution of the operation entered by the user.

Therefore, in accordance with an embodiment, the reproduced moving image data is configured as a hierarchical structure in which a moving image frame is presented with several resolution levels and hierarchically organized in order of resolution levels. In addition, a stream of a moving image is formed from units of blocks obtained by spatial decomposition of the frame in each layer of the hierarchy. High performance of the operation is achieved by switching the layer to be used on the screen to another layer, and the moving image stream to another moving image stream in accordance with the movement of the playback area. Hereinafter, moving image data having such a hierarchical structure is also referred to as "hierarchical data".

First, an explanation will be given of a basic display mode of such hierarchical data. In FIG. 1 illustrates an environment in which an image processing system 1 is used and to which the present embodiment can be applied. The image processing system 1 comprises an image processing device 10 that executes image processing programs, and a display device 12 that provides the processing result obtained by the image processing device 10. The display device 12 may be a television receiver having a screen for outputting an image and a speaker for outputting sound .

The display device 12 may be connected to the image processing device 10 via a cable or wirelessly via a wireless local area network (LAN) or the like. The image processing device 10 in the image processing system 1 can be connected to an external network, such as the Internet or the like, cable 14, through which the data is received and downloaded moving images. The image processing device 10 may be connected to an external network via a wireless channel.

The image processing device 10 may be, for example, a gaming device or a personal computer, and the image processing function may be implemented by downloading an image processing application. The image processing device 10 enlarges or reduces the moving image displayed on the screen of the display device 12, or scrolls the moving image up, down, left or right, in accordance with the request from the user to move the viewpoint. Hereinafter, such a change in the playback area, including an increase or decrease, is referred to as "moving the playback area". When the user works with the input device, while looking at the image displayed on the screen, the input device transmits a signal to the image processing device 10, which requests the movement of the playback area.

In FIG. 2, an example of the external configuration of an input device 20 is shown. Input device 20 comprises direction keys 21, analog joysticks 27a and 27b, and four types of control buttons 26, which are operating means that the user can manipulate. These four types of control buttons 26 include a round button 22, a cross-shaped button 23, a square button 24, and a triangular button 25.

The working means of the input device 20 in the image processing system 1 is assigned a function of inputting a request for increasing or decreasing an image on the screen and a function for inputting a requirement for scrolling an image up, down, left or right. For example, a function for inputting a request to enlarge or reduce an image on a screen is assigned to the right analog joystick 27b. The user can enter a request to reduce the image on the screen by pulling the analog joystick 27b towards the user, and can enter the requirement to increase the area on the screen by pulling the analog joystick 27b away from the user.

The function to enter the scroll request is assigned to the direction keys 21. By pressing the direction keys 21, the user can enter the scroll demand in the direction in which the corresponding direction key 21 is pressed. The function to enter the requirement to move an area on the screen can be assigned to an alternative control tool. For example, the function of entering a scroll demand may be assigned to the analog joystick 27a.

The input device 20 has the function of transmitting to the image processing device 10 a signal requesting the movement of an area on the screen that has been input. In the present embodiment, the input device 20 is configured so that the input device 20 can communicate wirelessly with the image processing device 10. The input device 20 and the image processing device 10 can establish a wireless connection with each other using the Bluetooth protocol (registered trademark), IEEE protocol 802.11 or similar protocols. The input device 20 can be connected to the image processing device 10 via a cable through which signals are sent to the image processing device 10, requesting the movement of the area on the screen.

Input device 20 is not limited to the device shown in FIG. 2. The input device 20 may be a keyboard, touch panel, button device, or any other means that provides for the manipulation of a user, a camera that captures an image of a specified object, a microphone that senses sound, etc., and may be of any type and / or any kind of interface, if only this interface would allow you to perceive the user's intention, moving the specified object, or the like, as information in electrical form.

In FIG. 3 conceptually illustrates hierarchical moving image data to be processed in accordance with the present embodiment. This data has a hierarchical structure containing in the direction of the ζ axis, that is, in the figure - from top to bottom, the zero layer 30, the first layer 32, the second layer 34 and the third layer 36. Although only four layers are shown in the figure, the number of layers is not limited to. As described above, each layer is composed of data from one moving image, presented with different levels of resolution, that is, data from several image frames organized in order of their time. In this figure, each layer is symbolically represented by four image frames. Obviously, the number of image frames varies depending on the duration of the playback and the frame rate of the moving image.

Hierarchical data have, for example, a hierarchical structure of a quad tree; when the image frames that form the layers are divided into “mosaic images” having the same size, then the null layer 30 is composed of one mosaic image, the first layer 32 is composed of 2 × 2 mosaic images, the second layer 34 is composed of 4 × 4 mosaic images, and the third layer is composed of 8 × 8 mosaic images and so on. In this case, permission to Ν-th layer (Ν is an integer equal to or greater than 0) is _^half resolution (N + 1) th layer in both the horizontal direction (X-axis) and vertical direction (Y axis) on the image plane. Hierarchical data can be generated, for example, by reducing the image frame in several stages (steps), based on the moving image in the third layer 36 having the highest level of resolution.

Both coordinates of the viewpoint during playback on the screen of a moving image and both coordinates of the corresponding playback area can be, as shown in Figure 3, represented in virtual three-dimensional space, which is formed by the x axis passing along the horizontal direction of the image, the y axis passing along the vertical the direction of the image, and the z axis representing the level of resolution. As mentioned above, moving image data in which several image frames are sequentially displayed are formed in the present embodiment as a layer. Thus, the image, which is actually reproduced on the screen, also depends on the interval of time that has elapsed after the start of playback. Therefore, for each layer, this figure also shows the time axis t.

At its core, the image processing device 10 sequentially reproduces image frames of any of the layers along the time axis t at a certain frame frequency. For example, the image processing apparatus 10 reproduces on the screen a moving image with a resolution level of the null layer 30 as a reference image. If during this process a signal is received from the input device 20 requesting the movement of the playback area on the screen, the image processing device 10 determines from this signal the magnitude of the change in the reproduced image and outputs the coordinates of the four corners of the subsequent frame (frame coordinates) in virtual space, using the magnitude of said specific change. The image processing device 10 then reproduces an image frame that corresponds to such frame coordinates. In this case, the image processing device 10 by setting the layer switching boundary along the z axis, appropriately switches the moving image data layers used to render the frame in accordance with the z coordinate value of the frame.

Instead of the coordinates of the frame in virtual space, the image processing device 10 can output both information identifying the layer and texture coordinates (UV coordinates) in this layer. Subsequently, a combination of the information identifying the layer and texture coordinates will also be referred to as frame coordinates.

Hierarchical data is stored in a compressed state in the memory of the image processing device 10 in the form of units of mosaic images. Then, the data necessary for rendering the frame is read from the memory of the image processing apparatus 10 and decoded. In FIG. 3, hierarchical data is conceptually shown, and the storage order or format of the data stored in the storage device is not limited at all. For example, if a correspondence is established between the position of the hierarchical data in the virtual space and the frame number with the actual storage area of the moving image data, then the moving image data can be stored in an arbitrary area.

In FIG. 4 illustrates the configuration of the image processing device 10. The image processing device 10 includes a wireless interface 40, a switch 42, a playback processor 44, a hard disk drive 50, a recording medium loading unit 52, a drive 54, a random access memory 60, a buffer memory 70, and a control unit 100. The playback processor 44 comprises a frame memory for buffering data that is displayed on the screen of the display device 12.

Switch 42 is an Ethernet switch (Ethernet is a registered trademark) and is a device that connects via cable or wireless to external devices and transmits data. The switch 42 is connected by cable 14 to an external network and configured so that it can receive hierarchical data from the image server. Next, the switch 42 is connected to the wireless interface 40, and the wireless interface 40 is connected to the input device 20 using a predetermined wireless communication protocol. The signal requesting the movement of the playback area, which is input by the user through the input device 20, is sent to the control unit 100 via the wireless interface 40 and the switch 42.

The hard disk drive 50 functions as a storage device for storing data. The drive of the hard disk 50 may also store moving image data. If a removable recording medium, such as a memory card, is inserted, the loading block of the recording medium 52 reads data from this removable recording medium. When a read-only read only disk is inserted, the drive 54 recognizes and initiates a ROM disk, then reading data from it. The ROM disk may be an optical disk, a magneto-optical disk, and other recording media. These recording media can also store moving image data.

The control unit 100 comprises a multi-core CPU. One CPU contains one core of a universal processor and several simple processor cores. The core of a universal processor will be referred to as a “powerful processor unit (BCH),” and the other processor cores will be referred to as “synergetic processor units (SPB)." The control unit 100 may also be equipped with a graphics processor (GPU).

The control unit 100 includes a memory controller connected to the RAM 60 and the buffer memory 70. The MBP contains registers and a central processor as a component that performs the calculations. The IBE effectively distributes tasks, as the basic processing units in applied tasks, to the corresponding SPB. The IBE itself can perform the task. SPB contains registers, a sub-processor, as a component that performs calculations, and local memory used as a local storage area. Local memory can be used as buffer memory 70.

The random access memory 60 and the buffer memory 70 are storage devices and are organized in the form of random access memory (RAM). SPB contains a specialized controller for direct memory access (DMA) and provides high-speed data exchange between the RAM 60 and the buffer memory 70. High-speed data exchange is also between the frame memory in the playback processor 44 and the buffer memory 70. The control unit 100, in accordance with an example implementation, implements high-speed image processing in parallel operation of several SPB. The playback processor 44 is connected to the display device 12 and provides an image processing result in accordance with a user requirement.

The image processing device 10 sequentially downloads moving image data that is directly linked in space and time to a frame that is currently being loaded from the hard disk drive 50 into the RAM 60 to smoothly perform the process of increasing or decreasing or scrolling the image displayed on the screen. Further, the image processing apparatus 10 decodes a part of the moving image data loaded into the RAM 60 and stores the decoded data in the buffer memory 70. This makes it possible to gradually move the moving image and thereby smoothly move the playback area on the screen. In this case, the data to be downloaded or decoded can be determined in advance by predicting the area that will be needed subsequently, based on the earlier direction of movement of the playback area on the screen.

In the hierarchical data shown in FIG. 3, the position in the z-direction shows the resolution level, and the closer the position to the zero layer 30, the lower the resolution level, and the closer the position to the third layer 36, the higher the resolution level. In terms of the size of the image displayed on the display device, the position in the z direction represents the scale. Assuming that the scaling factor of the reproduced image in the third layer 36 is 1, the scaling factor in the second layer 34 will be _^1/4, a scaling factor in the first layer 32 would be 1/16, and the scale factor in a zero layer 30 will be 1/64 from the image on the third layer 36.

Therefore, if the image on the display changes in the direction of the z axis from the zero layer 30 to the third layer 36, then the reproduced image is enlarged. If the reproduced image changes in the direction from the third layer 36 towards the null layer 30, then the reproduced image is reduced. For example, when the zoom ratio of the reproduced image is close to that of the second layer 34, the reproduced image is generated using image data on the second layer 34.

More specifically, as described above, the switching boundaries are set for the corresponding intermediate scaling factor of each layer. For example, in the case where the scaling factor of the image displayed on the screen is between the switching boundary between the first layer 32 and the second layer 34 and the switching boundary between the second layer 34 and the third layer 36, the frame is reproduced using image data on the second layer 34. In this when the scaling factor is between the second layer 34 and the switching boundary between the first layer 32 and the second layer 34, the image frame of the second layer 34 is reproduced on the screen with a decrease. When the scaling factor is between the second layer 34 and the switching boundary between the second layer 34 and the third layer 36, the image frame of the second layer 34 is reproduced on the screen with magnification.

On the other hand, in the case of identification and decoding of the area that will become necessary in the future and which is predicted from the signal requesting the movement of the playback area on the screen, the scaling factor of each layer is predefined as the border of the data prefetch. For example, when the required scaling factor determined by the signal of the requirement to move the playback area exceeds the scaling factor of the second layer 34, the image processing device 10 preliminarily reads at least part of the image data of the first layer 32 located in the decreasing direction, decodes the previously read image data, and writes the decoded image data to the buffer memory 70.

The same is true for the process of pre-reading the image in the direction of up, down, right or left. More specifically, a prefetch boundary of image data distributed in the buffer memory 70 is predetermined, and while the position of the reproduction area determined by the demand signal to move the reproduction area exceeds the prefetch boundary, the prefetch process is started. Thus, a mode can be realized when a moving image is gradually reproduced with a smooth change in this resolution level and the position of the playback area on the screen in accordance with the requirement to move the playback area received from the user.

In FIG. 5 illustrates in detail the configuration of a control unit 100a having a function for reproducing a moving image using moving image data that has a hierarchical structure in accordance with the present embodiment. The control unit 100a comprises an input information obtaining unit 102, which receives information input by the user through the input device 20, a software frame coordinate determination unit that determines the frame coordinates of the newly reproduced area, a loaded area determination unit 106, which determines compressed data of the moving image of the newly loaded area , and a loading unit 108, which loads from the drive of the hard disk 50, moving image data of a desired area. The control unit 100a also comprises a decoding unit 112, which decodes the compressed moving image data, and an image reproducing processing unit 114, which renders the image frame.

In FIG. 5 and FIG. 15, described later, each of the presented elements, which are functional blocks that perform the most diverse processing, are implemented in hardware, which may consist of a central processor (CPU), memory, and various other LSIs, and software using programs loaded into memory. As was previously established, the control unit 100 has one MPB and several SPBs, and the functional blocks can be formed only from the MPB, only from the SPB or by a combination of both. Therefore, it will be apparent to those skilled in the art that functional blocks can be implemented in a wide variety of forms, only in hardware, only in software, or combinations thereof, and there can be no limitations.

The input information obtaining unit 102 receives the requirement, entered by the user through the input device 20, to start or end the playback of the moving image, move the playback area on the screen, etc., and sends a message about this requirement to the unit for determining the coordinates of the software frame. The frame coordinate determination unit PO determines the frame coordinates of the area to be replayed in accordance with the frame coordinates of the current playback area and the signal requesting the movement of the playback area that is entered by the user and sends a message about the specified specific frame coordinates to the load area determination unit 106 , a decoding unit 112 and an image reproduction processing unit 114.

The unit for determining the loading area 106, based on the frame coordinates about which a message has been received from the unit for determining the frame coordinates of the software, specifies the moving image data re-downloaded from the hard disk drive 50, and issues a download request to the loading unit 108. As described above, in accordance with an embodiment, a sequence of frames of each layer is formed into one or more independent streams of a moving image from mosaic image units. As a result of which, basically, streams of moving images are formed, the number of which is equal to the number of mosaic images in each layer. Or, as will be described later, in a single stream of a moving image, there may coexist data that points to the same area of many layers, or many mosaic images.

Information that combines the identification number indicating the layer and position of the mosaic image and the identification number of the moving image stream data is pre-attached to the moving image data, and this information is loaded into the RAM 60 when the reproduction process of this moving image is started. Based on the coordinates of the frame, the unit for determining the loaded area 106 accesses this information and obtains the identification number of the moving image stream of the required area. If the data of the corresponding flow of the moving image has not yet been downloaded, the determination unit of the loaded area 106 issues a download requirement to the loading unit 108. In addition, even in the case where the frame coordinates do not change, the determination unit of the loaded area 106 is sequentially in accordance with the passage of the moving image, gives the requirement to download the necessary data stream of a moving image. A moving image stream can be pre-divided by time into some units, and the stream can be loaded by units formed after separation.

In addition to the moving image stream that is currently needed to render the frame, the loading area determination unit 106 may specify a moving image stream, which is expected to be requested later by the prefetch process described earlier, and may issue a download request to the loading unit 108 In accordance with the requirement issued by the unit for determining the downloadable area 106, the download unit 108 performs the process of downloading a stream of a moving image from the drive of the hard disk 50. More specifically but, boot unit 108 of the moving image stream identification number which is to be loaded, determines the storage area in the hard disk drive 50 and stores the data read from this memory area in RAM 60.

Based on the coordinates of the frame at each point in time, the decoding unit 112 reads the data of the required moving image stream from the RAM 60, decodes the data that has been read, and sequentially writes the decoded data to the buffer memory 70. The decoding object can be represented in units of moving image streams . When the area specified by the frame coordinates, which were determined by the frame coordinate determining unit 110, is located on several streams of the moving image, then the decoding unit 112 decodes several streams of the moving image.

In accordance with an embodiment, as will be described later, the data volume of the hierarchical layer is reduced due to the fact that part of the data of the hierarchical layer from all data of this hierarchical layer is stored as a differential image, which is the difference between the said part and the enlarged image of the image on a layer higher than the layer of this part. Therefore, in the case where the decoded image is a differential image, the decoding unit 112 also decodes the image of the hierarchical layer of a higher level than the layer that was used to generate the differential image, enlarges the image of the higher level layer and adds the image of the higher layer to the decoded differential image, thereby restoring the data of the original image, and writes the data of the original image in the buffer memory 70. Based on the coordinates of the frame at each time, the image reproduction processing unit 114 reads the corresponding data from the buffer memory 70 and sends this data to the frame memory 44 playback processor.

In an embodiment where, during the reproduction of one moving image, the playback area can be moved, including increasing or decreasing, it is desirable that all flows of the moving image pass synchronously in time and that the image frame is rendered smoothly, regardless of whether the used stream is switched or not switched moving image to another stream. Therefore, as described above, the region requested for playback during and / or after the motion picture stream is expected to be requested, loaded and decoded with high priority, so that the processing efficiency needed during frame rendering is improved . Further, as will be described later, by properly configuring the moving image stream, the playback of the frame on the screen starts with a low delay, regardless of the point in time when the moving image stream used for playback on the screen switches to another stream.

With the configuration described above, the whole image can be easily and smoothly viewed, part of the image, etc., can be enlarged, even with a large size of a moving image, for example, with a moving image, the frame size of which is on the order of several gigapixels. Further, due to the preliminary organization of the moving image in the form of a hierarchical structure, the moving image can be reproduced in the same form, regardless of the display device, by selecting the appropriate layer in accordance with, for example, characteristics such as display resolution level, screen size and processing performance.

As for images at the same moment in time on different layers contained in the hierarchical data of a moving image, in this case the content on these layers is the same, although the image resolution levels are different, therefore the hierarchical data is characterized by the property that between the layers there is always redundancy. Therefore, in accordance with an exemplary embodiment, as described above, one or more layers of hierarchical data is configured as a difference image between the given layer and the enlarged image of the image on a layer of a higher hierarchy level than this layer. Given the properties of the hierarchical data described above, the size of the data can be significantly reduced by using this difference between the layers.

6 schematically shows a state where some layers of the data hierarchy of a moving image having a hierarchical structure are shown by difference images. In the example of FIG. 6, the null layer 30 and the second layer 34 store the original image data, and the first layer 32 and the third layer 36 store the differential image data. The original data is shown in white, not shaded parts, and the difference image data is shown in shaded parts. In the following explanation, the layer that stores the original image data is referred to as the "original image layer", and the layer that stores the differential image data is referred to as the "differential image layer".

Here, in the case where the region 120 is to be reproduced with a resolution of the third layer 36, the value of each pixel of the image corresponding to the region 122, which is located on the second layer 34, after increasing this image to the resolution level of the third layer 36 is added to the difference image of the region 120, which is located on the third layer 36. In this case, shown in this figure, the resolution of the third layer 36 is 2 × 2 times greater than the resolution of the second layer 34, so the image of the area 122 is enlarged 2 × 2 times. To increase, well-known interpolation algorithms can be used, such as the nearest neighbor interpolation algorithm, bilinear algorithm, bicubic algorithm, etc.

By configuring the data in this way, the total amount of data can be reduced. In addition, the necessary bandwidth of the transmission path through which data passes during playback (i.e., the internal bus in the image processing device 10, the network connected to the image server, etc.) can be limited. This is because the amount of the sum of the differential image data with the desired resolution and the source image data on the layer where the resolution is lower than the desired resolution will be less than the amount of the original image data with the desired resolution at which it is intended to reproduce the original image.

In the example shown in FIG. 6, the distribution of the layers of the original image and the layers of the differential image is fixed. In this case, the bit rate when reproducing an image with a resolution level on the differential image layer will be lower, therefore, the bandwidth required for data transmission can be reduced. On the other hand, during the period when the image is reproduced with a resolution level on the layer of the original image, the bit rate becomes the same as the speed when the layer of the differential image is not created, and thus the decrease in the bandwidth cannot have an effect. In order for the image to be reproduced smoothly, without jerks, at any resolution level and no matter what the layer is, it will be necessary to reserve the bandwidth in accordance with the maximum bit rate, therefore, in such an embodiment it will be difficult to obtain a decrease in the bandwidth, which required to reserve.

Therefore, by switching the distribution of the source image layer and the differential image layer to another distribution configuration in accordance with a predetermined rule, the bit rate is averaged, the amount of transmitted data is limited to a range longer than the switching period, and the bandwidth required for transmission can be reduced. 7 is a drawing illustrating an embodiment where the distribution of the source image layer and the differential image layer is switched between some distribution configurations. In this figure, the horizontal axis shows the time axis of the moving image. The figure shows the configuration of hierarchical data composed only of the corresponding frames at time instants t1, t2 and t3, respectively.

First, in the hierarchical data that represents frames at time t1, the null layer 30 and the second layer 34 are defined as layers of the original image, and the first layer 32 and the third layer 36 are defined as layers of the differential image. In this case, an image with a resolution level of the first layer 32 can be obtained by summing the difference image of the first layer 32 and the original image of the zero layer 30. Further, an image with a resolution level of the third layer 36 can be obtained by summing the difference image of the third layer 36 and the original image of the second layer 34.

In the hierarchical data that represent frames at time t2, which occurs later than time t1, the zero layer 30, the first layer 32 and the third layer 36 are defined as layers of the original image, and the second layer 34 is set as the difference image layer. In this case, an image with a resolution level of the second layer 34 can be obtained by summing the differential image of the second layer 34 and the original image of the first layer 32. For hierarchical data that represents frames at time t3, which comes even later, the distribution is the same as and the distribution configuration at time t1.

For example, in the case when the region 124a with the resolution level of the third layer 36 is to be reproduced from among all the frames at time t1, then the differential image data of the region 124a and the image data of the corresponding region 126a included in the original image of the second layer 34 are required. It is assumed that the playback area moves simultaneously with how the frames advance over time, and from among all the frames, at the time t2, the region 124b with the resolution level of the third layer 36 should be reproduced. In this case, require only the data of one region 124b is obtained, since the data of the third layer 36 are the original image data. As time passes further and in the case where, from among all the frames, at the time t3, the region 124c with the resolution level of the third layer 36 is to be reproduced, the differential image data of the region 124c and the data of the corresponding region 126c, which is included in the original image of the second layer 34, are required .

As a result, as shown at the bottom of this figure, to reproduce region 124a, region 124b, and region 124c with a resolution level of the third layer 36 from all frames at time t1, t2, and t3, data 128a, data 128b, and data 128 s are sequentially read, accordingly, in this case, the data 128a consists of a combination of the difference image of the third layer 36 and the original image of the second layer 36, the data 128b is the data of the original image of the third layer 36, and the data 128c consists of the combination of the difference image of the third layer 36 and image of the second layer 34.

Thus, by the configuration in which the source image layer and the differential image layer are switched over time, the possibility that large data such as the original image data 128b will be continuously transmitted and the amount of data to be transmitted per unit time is reduced ( yes, bit rate) can be reduced. To improve understanding, in the corresponding figures, the reproduced object is fixed with a resolution of the third layer 36, however, even if the layer of the reproduced object is changed to another layer in the middle, a similar effect can be obtained.

In FIG. 8 - FIG. 11 shows examples of distribution orders of hierarchical layers of the original image and layers of the differential image. In these figures, the horizontal axis represents time, and the change in time of the distribution of hierarchical data, in which the zero layer, the first layer, the second layer are indicated as "Lv0", "Lv1", "Lv2", ... is represented by white rectangles - for the time intervals of the layers of the original image, and shaded rectangles for the time intervals of the layers of the differential image, respectively. In the time interval represented by one rectangle, one or more frames are reproduced.

In FIG. Figure 8 shows the case when all the entire hierarchical data uses the common moment of distribution switching time. The zero layer Lv0 is usually represented as the layer of the original image 152, since there are no other layers above the zero layer. The same applies to the following examples. In the example shown in FIG. 8, the distribution of layers switches to one or more layers at time points T0, Τ1, T2, T3, T4, T5 and T6. Namely, during the period from the time point T0 to the time point T1, the zero layer Lv0 and the second layer Lv2 are the layers of the original image 152 and 156, and the first layer Lv1, the third layer Lv3 and the fourth layer Lv4 are the layers of the difference image 154, 160 and 162 of which, at time T1, the second layer Lv2 switches to the differential image layer 164, and the third layer Lv3 switches to the layer of the original image 166.

At time T2, the first layer Lv1 and the fourth layer Lv4 switches to the layers of the original image 168 and 170, respectively, and the third layer Lv3 switches to the layer of the differential image 172. At time T3, the first layer Lv1 and the fourth layer Lv4 switch to layers of the differential image 174 and 176, respectively, and the second layer Lv2 switches to the layer of the original image 178. Such switching is repeated also after that. Then, the layer of the original image, with the exception of the zero layer Lv0, is switched from the first layer Lv1 → the second layer Lv2 → the third layer Lv3 → the fourth layer Lv4, and layers other than these layers are set as the difference image layer.

In the embodiment shown in FIG. 8, the number and arrangement of layers, which are simultaneously set as layers of the original image, and the number and arrangement of layers, which are simultaneously set as layers of a difference image, are not limited by the number and arrangement shown in this figure, but any number and any arrangement can be taken, since only the combination of the layer that is defined as the layer of the original image and the layer that is defined as the layer of the differential image change at the same time instant, common to all layers. In the example shown in FIG. 8, difference image layers, whatever the time period, are defined as layers following one after another in the vertical direction. For example, during the period from time T2 to time T3, the second layer Lv2 and the third layer Lv3, which follow each other in the vertical direction, are the layers of the difference image 179 and 172.

In the case of such an arrangement, when more than two successive layers simultaneously form layers of the difference image, the difference image can be generated by subtracting from the layer immediately above it, or it can be generated by searching for the layer of the original image and subtracting from the original image. In the first case, the difference image obtained from the difference image is stored as long as the layer that is directly above is the layer of the original image, so the lower the layer is relative to the layer of the original image, the greater the number of layers and a greater number of summation processes to restore the image, but the data volume becomes smaller.

In the latter case, the number of layers and the number of summation processes required to restore the image remains constant, however, the lower the layer relative to the layer of the original image, the larger the difference becomes, and therefore the data volume will be larger compared to the data volume of the previous case. Which of the differential images should be used is determined based on the contents of the moving image, the processing capabilities of the display device, the bandwidth that can be used in transmission, and so on. In addition, the processing order can be determined so that the layers of the differential image will not be located on successive layers.

Switching times T0 - T6 can follow at regular intervals. In this case, for each certain number of frames (for example, for every eight frames), switching is performed on at least one of the layers, and in this case, a difference image layer can be predefined for each frame. Therefore, when reproducing a moving image, the layer that is required for rendering can be easily determined and / or it can easily be determined whether or not an addition process is required or not, thereby simplifying the control process.

Alternatively, at least one of the switching times may be determined based on the characteristics of the moving image. For example, referring to the scene change information contained in the moving image, the point in time when the scene changes is identified, and switching is performed at the point in time between the frame before changing the scene and the frame after changing the scene. Instead of information about a scene change, switching can be carried out at a time between frames that are very different from each other. As will be described later, in the case when the frames are configured so that they are dependent on each other created by compression encoding, such as, for example, inter-frame prediction encoding, it is convenient to switch between frames having low temporal redundancy.

Alternatively, the point in time when the layers are switched can be determined in accordance with the accumulated amount of data volume. That is, on one of the layers other than the zero layer Lv0, the next switching time is defined as the time when the cumulative amount of the data frame after the current switching reaches a predetermined threshold value. Although the size of the image differs depending on the layers, the amount of actually transmitted data is proportional to the number of mosaic images, regardless of the layers, therefore it is assumed here that the amount of data used to calculate the accumulated value is a quantity reduced to a single surface, for example, to the surface of a mosaic image . In this case, since the amount of data of the differential image is small, the switching times are determined essentially by the amount of data of the original image stored by one or more layers of the original image. Having similarly determined switching times based on the actual amount of data, it is possible to efficiently distribute data with a high bit rate.

In other words, the determination of the switching times in a similar manner will exactly correspond to setting a higher switching frequency as soon as the bit rate of the original image increases. Since the continuous transmission of data with a high bit rate leads to a long state of busyness when the free range of frequencies is almost not available, a wide range of frequencies is required in order to result in a continuous smooth transmission of images. More frequent switching to images with a higher bit rate, as described above, ensures a constant bandwidth margin, and using this bandwidth margin allows data to be transmitted at a high bit rate.

For an image whose bit rate is not too high, and for which the bandwidth does not become a limitation even when the data of its original image is transmitted, the switching frequency of the processes during image playback is reduced by reducing the switching frequency, which leads to easier control . Thus, the bandwidth required for data transmission can be directly and more effectively reduced.

In FIG. Figure 9 shows the case when each group formed by the decomposition of hierarchical data layers shares the same switching time. In the example shown in this figure, two layers successive in the vertical direction (i.e., the first layer Lv1 and the second layer Lv2, the third layer Lv3 and the fourth layer Lv4, and the fifth layer Lv5 and the sixth layer Lv6) are assembled as one group (groups 180a, 180b and 180c), respectively. The layers in this group share the switching times. The source image layer and the difference image layer are distributed over two layers that alternately belong to one group.

At each moment of switching time, the layer of the original image and the layer of the differential image are switched. For example, in group 180a, during the interval between the time T0 and the time T1, the first layer Lv1 is the layer of the original image 182, and the second layer Lv2 is the layer of the difference image 184. In contrast, during the interval between the time T1 and the time T2, the first layer Lv1 is the difference image layer 186, and the second layer Lv2 is the layer of the original image 188. Further, during the interval between the time T2 and the time T3, the first layer Lv1 is the layer of the original image 190, and the second oh Lv2 is a layer differential image 192. The same holds true for other groups 180b and 180 c. However, the switching times for each group may vary. Thus, the layer of the original image, which is required to restore the original image from the differential image, exists in that group, which at any time is certainly directly above.

In the example shown in this figure, two layers form one group, therefore, the layer of the original image, which is required to restore the original image from the differential image, exists, as a maximum, on the second upper layer. Similarly, by limiting the number of layers from the differential image layer to the original image layer, the load resulting from the data sampling process required to restore the original image and / or the image adding process can be reduced.

The switching times in each group can be determined for each group by selecting one of the rules, as explained in FIG. 8. For example, switching times for each group can be determined in accordance with the accumulated amount of data volume. In this case, since the amount of data for each unit area has the tendency that the smaller the image becomes, the larger the amount of data, the higher the group in the hierarchy, the earlier the accumulated amount of data will reach a threshold value.

Therefore, as shown in FIG. 9, the switching frequency of group 180a, consisting of the first layer Lv1 and the second layer Lv2, is the highest, followed by group 180b, consisting of the third layer Lv3 and the fourth layer Lv4, and then the group 180c, consisting of the fifth layer Lv5 and the sixth layer Lv6, that is, the switching frequency of the lower layer group is getting lower. The careful setting of the switching times of the respective layers, while taking into account the characteristics of the respective layers, can be carried out according to the same principle as was explained with reference to FIG. 8, the necessary bandwidth is effectively reduced.

Further, the threshold value for the accumulated value of the data volume may vary specifically for each group. For example, sometimes the bit rate of a certain layer is set higher than the speed of other layers, depending on the contents of the moving image at the time of encoding with compression. Such a specific layer is, for example, a layer that is expected to be used for reproduction at a higher frequency than other layers, a user-defined layer, etc. In this case, if the threshold value of the group containing such a specific layer is set lower value, in accordance with the set bit rate, the switching frequency can be adjusted in accordance with the actual bit rate.

In FIG. 10 shows another example of a case where switching times are common to each group created by layering hierarchical data. In the example shown in this figure, the first layer Lv1 and the second layer Lv2, the third layer Lv3, the fourth layer Lv4 and the fifth layer Lv5, and the sixth layer Lv6 and the seventh layer Lv7 form the corresponding groups (groups 194a, 194b and 194c, respectively). The number of layers belonging to one group is different. In this case, the switching times of each group are determined in a similar way, as was shown in FIG. 9.

Further, the distribution order is appropriately planned so that the number of layers from the differential image layer (e.g., the differential image layer 198) to the layers of the original image (e.g., layers of the original image 196, 199) can be limited even if more than two layers belong to the group. More specifically, first within a group, the number of layers following one after another at the same time and which are layers of a difference image is limited to a maximum of 2N layers. Further, if in each group the number of layers calculated from the boundary of the group following one after another at the same time and being layers of the difference image is limited so that it is a maximum of N, then the number of layers following one after another in the same at the same time being the layers of the difference image, it becomes equal to a maximum of 2N, even taking into account the two groups between which the boundary is located.

For example, in the case shown in FIG. 10, it doesn’t happen at all that within a group several several successive layers are at the same time layers of a difference image. Further, it also does not happen that several successive layers, counted from the group boundary, are at the same time layers of a difference image. In other words, N = 1, and this implies that the number of successive layers, which are layers of the difference image, is a maximum of 2N = 2, even considering all the hierarchical data. The maximum value 2N of the number of layers from the differential image layer to the layer of the original image has, as mentioned above, an effect on the computational load in the display device. Therefore, 2N is determined in accordance with the processing performance of the display device, and in accordance with this, the layer distribution order is also determined.

In FIG. 11 shows the case where groups, each of which contains several layers and which were formed in accordance with the embodiments shown in FIG. 9 and FIG. 10 are grouped further for each area in the image. In other words, images of several layers are divided at the same positions on the images, and for each area representing the same part of the image, a group is formed. In the example shown in the figure, the image of the third layer Lv3 is divided into areas Lv3_0, Lv3_1, etc., and the image of the fourth layer Lv4 is divided into areas Lv4_0, Lv4_1, ..., etc.

Further, if the region Lv3_0 of the third layer Lv3 and the region Lv_4 of the fourth layer Lv4 representing the same parts in the images are taken as the zero region, and the region Lv3_1 of the third layer Lv3 and the region Lv4_1 of the fourth layer Lv4 representing the same parts in the images are taken as the first region, then groups are formed for each region (groups 200b and 200c), that is, zero regions in the third layer Lv3 and in the fourth layer Lv4, the first regions in the third layer Lv3 and in the fourth layer Lv4, etc., and the switching times in each group will be the same.

Similarly, for the whole image, groups are formed in which several layers fall on each region. Although the figure shows only the third layer Lv3 and the fourth layer Lv4 as a decomposition object, more than two layers can belong to the same group. The switching times in each group can be determined by choosing for each group the rules in the same manner as shown in FIG. 9. Further, at the switching time determined for each group, the region of at least one layer within the group switches between the layer of the original image and the layer of the differential image (for example, from the layer of the original image 202 to the layer of the differential image 204).

For example, the switching times of each group can be determined in accordance with the accumulated amount of data volume. Even for a single image, if the complexity of the image differs depending on the areas in that image, its bit rate will also be different. For example, an area of blue sky, which is almost uniform in color, and an area of the road that cars follow in both directions, will differ in bit rate. As described above, for a region having a high bit rate, it is preferable to switch the distribution at a higher frequency. As a result, switching frequencies will accordingly vary depending on the region. Therefore, by determining the switching times of each region, it will be possible to adjust these moments at a more detailed level, and the used bandwidth can be effectively reduced in accordance with the image content.

Similarly as explained in FIG. 9, a threshold value that is set in accordance with the accumulated data volume value can be determined so that it will be different for each group. From the point of view of regulating the switching times, taking into account the difference in bit rates between regions due to the difference between these regions, the layer for which the grouping of the corresponding regions is performed is defined as a relatively high resolution layer in which bit rates differ due to the difference between mosaic images. Namely, for a low resolution image, one group 200a can be formed without decomposition, as shown in FIG. 11, in areas such as the zero layer Lv0, the first layer Lv1, the second layer Lv2, etc. In this case, the zero region and the first region, which are defined as separate groups 200b and 200c in the third layer Lv3 and the fourth layer Lv4, respectively, are combined into one group in the second layer Lv2. Alternatively, depending on the content of the image, grouping for each area can be performed for all layers, with the exception of the zero layer Lv0.

Also in this embodiment, as explained in FIG. 10, the permissible upper limit 2N of the number of consecutive layers defined as layers of the difference image at the same time is determined based on the processing capabilities of the display device. Further, the distribution order is set such that the number of consecutive layers that are defined as layers of the difference image at the same moment in time, where this number is calculated from the boundary of the group having up and down links in the hierarchical structure (for example, the boundary where the divided regions combined, as described above), takes a maximum value of N.

As described above, according to an embodiment, compression coding is performed by units of regions, for example, tiles, and independent streams of a moving image are generated. During playback, only the moving image stream that includes the playback area on the screen is separately decoded, and each frame of the moving image is visualized by connecting the reproduced mosaic images. By making it possible to randomly select a time stream of a moving image, that is, by making it possible that playback of any of the streams of moving images will begin at any time, it will be possible to arbitrarily move the playback area of the moving image.

In such an embodiment, in the case where the original image data and the difference image data, as described above, exist together in the same moving image stream, by configuring the moving image stream, taking into account the switching times, the processing efficiency during playback can be improved. In FIG. 12 - FIG. Figure 14 shows examples of the configuration of a moving image stream. In FIG. 12 illustrates a case where the frame order of a moving image is used without change as the order of the data of a moving image stream. In this figure, the upper rectangles show the sequence of mosaic images obtained by highlighting the area of some mosaic image in the sequence of frames before encoding with compression, and the time axis of the moving image is shown along the horizontal axis. The bottom of the picture shows the flow of a moving image after compression encoding, where the left end is the header of the stream. Parts corresponding to the original image data are shown both before and after coding with white rectangles, and parts corresponding to the difference image data are shown with shaded rectangles.

In the following explanation, the “tiled image” contained in the frame is also sometimes referred to as the “frame”, so that the meaning of the images arranged in chronological order on the moving image can be easily understood. In the case of this figure, first, before encoding with compression, the sequence of frames of the original image i′1, the sequence of frames of the differential image d′1, the sequence of frames of the original image i′2, the sequence of frames of the differential image d′2, ... are alternately arranged. The compressed data i′1, d′1, i′2, d′2, ..., which are compressed and encoded by the corresponding sequences of frames, are generated in the same order, without change. In the case when compression encoding is performed independently for each frame, that is, in the case when all frames are considered as an internal frame, only a simple connection of the compressed and encoded data in the frame order will suffice.

On the other hand, in the case of increasing the compression ratio due to the use of temporal redundancy of images, for example, coding with prediction between frames, it is preferable that the relationship between the data does not affect different types of frame sequences (i.e., the original image and the difference image) . In other words, the data is configured such that decoding the compressed data d′1 of the differential image does not require the use of compressed data i′1 of the original image before the compressed data d′1. Similarly, the data is configured so that decoding the compressed data i′2 of the original image does not require the use of compressed data d′1 of the differential image before the compressed data i′2.

Under such circumstances, the frame decoding processing can be carried out within the same data type, at whatever point in time the data is accessed, and it becomes possible to suppress processing delay. Therefore, the frame immediately following the type switching (that is, the header frame data of the corresponding compressed data i′1, d′1, i′2, d′2, ...) is defined as an internal frame that can be decoded independently . Similarly, the dependence of data between frames is restored and therefore the compressed data i′1, d′1, i′2, d′2, ..., which are divided depending on the type of image, can be formed into a separate stream of a moving image. Further, since the characteristics of the images, such as, for example, the frequency response, are different for the original image and the difference image, various compression algorithms can be used.

In FIG. 13, a case is shown where frame data of the same type is extracted from a sequence of frames, and a single unit of compressed data is formed by grouping and compressing them. This pattern is presented mainly in the same form as the pattern in FIG. 12, however, the frame sequence of the original image before compression coding and the sequence of frames of the differential image before compression coding are shown in FIG. 13 with a shift in the vertical direction. The time axis for these two sequences of frames is common. Of these sequences of frames, before encoding with compression, several (five in the figure) blocks of sequences of frames of the original image following one after another are selected, starting from the header, and the selected blocks are combined into a sequence of frames i3, which has a new order in time. Then, the same number of blocks of sequences of the frame of the differential image are allocated, respectively, immediately after the selected blocks of the sequence of frames of the original image, and the selected blocks are combined into a sequence of frames d3 having a new order in time.

Similarly, the blocks are combined into a sequence of frames i4 of the original image, a sequence of frames d4 of the differential image, ... Each of the compressed data i′3, d′3, i′4, d′4, ..., which are compressed and encoded sequences of frames i3, d3, i4, d4, ..., selected and combined for each type of image, are considered as one unit of compressed data. The generated compressed data may be formed into a separate moving image stream of each unit, or may be combined in the generation order and formed into a moving image stream. The boundary to which the sequences of frames are combined can be set at the moment when the accumulated number of frames or the accumulated value of the data volume exceeds a predetermined threshold. Alternatively, for example, header frames included in blocks of sequences of frames that, when selected, are arranged sequentially and the boundary can be specified at the moment when the difference between the header frames exceeds a predetermined threshold can be compared. This can happen, for example, at the time the scene changes.

As explained in FIG. 12, when decoding a frame to which the call is arbitrary, it is preferable that the header frame data of various types of data in the moving image stream after compression and encoding is specified as an internal frame so that the data needed for decoding does not go into a sequence of frames of another type . However, in the case of switching between the original image and the difference image at a high frequency, arising, for example, due to the high bit rate of the image, the number of internal frames will increase and the compression ratio will decrease if all frames immediately after switching are set as an internal frame.

Also, in the case when the time redundancy lasts long enough (for example, in the case when the object displayed on the screen does not move), the compression rate can be significantly reduced by introducing an internal frame at the moment when the original image and the difference image are switched. In such a case, as described above, by combining sequences of frames of the same type that do not follow continuously in time and forming one unit of compressed data from them, thereby reducing the number of frames specified as an internal frame, it will be possible to combine the prevention of propagation dependence on another type of frame sequence and improved compression ratio.

Further, in this embodiment, since the order of the data after they are compressed and encoded differs from the order of the frames in the original moving image, information is added to the data after compression and encoding, which relates the order of the original frames and the order in which the data appears in the data after compression and coding. During playback of a moving image, the decoded frames are reorganized and restored in the original order using the link to this information.

In FIG. 14, a case is shown where mosaic image data, which represent the same area on many layers, are grouped and compressed, thereby forming one unit of compressed data. Although this figure is presented basically in the same form as FIG. 12, the frame sequences of five layers (i.e., the Lv0 zero layer, the first Lv1 layer, the second Lv2 layer, the third Lv3 layer and the fourth Lv4 layer) are shown as frame sequences before compression encoding with a common time axis. The rectangle in each layer symbolically represents one or more sequences of mosaic images. Further, the data after compression coding is shown by rectangles with a different type of hatching than the hatching of the difference image data, since the data after compression coding, in accordance with an embodiment, contains both the original image data and the difference image data.

If we take the top layer of all many layers, which are combined as one unit of compressed data, as the layer of the original image, then there will be no need to read another stream of the moving image to obtain the data required to restore the original image from the differential image, which ensures high efficiency processing necessary to reproduce the image on the screen. As a result of this, whichever layer is used for reproduction, the delay time before reproduction is reduced, and thus, this embodiment is particularly effective in the case where the resolution level in the display device is selected, or in the case where the resolution level is set variable.

The dimensions of the image, which is represented by the same area on several layers, are different due to the difference in resolution levels of these layers. Therefore, in the case where hierarchical data has layers where the data of each layer is 2 × 2 times larger than the data of the layer immediately adjacent to it, when combining the data of two layers, as shown in the figure, one mosaic image element of the upper layer and four mosaic image elements of the lower layer are combined into one unit of compressed data. When the data of three layers is combined, then one mosaic image element of the upper layer, four mosaic image elements of the middle layer and sixteen mosaic image elements of the lower layer are combined into one unit of compressed data. The same applies for more than three layers.

Although frames that are in the same unit of compressed data belong to different layers, these frames belong to the same period of the moving image. As shown in this figure, in combined layers, all frames that are contained in a period during which there is no switching between the layer of the original image and the layer of the differential image can be combined into one unit of compressed data. Alternatively, frames included in a period shorter than in a period when switching does not occur can be combined into one unit of compressed data. In the latter case, the frames can be grouped in the presence of a threshold value of the accumulated number of frames and / or the accumulated value of the data volume. The sequence of data in the data after compression and encoding is not specifically limited, if only the data before encoding with compression would be associated with the frame number, layer and region.

As described above, the embodiment shown in FIG. 14 has the advantage that, if it is necessary to restore the original image from the differential image, easy access to the data of the layer of the original image is provided. On the other hand, data of a differential image, which is not necessary, is also transmitted along with this, even in the case of reproducing an image with a resolution level of a layer of the original image. Therefore, to prevent this situation from continuing for a long time, the number of frames included in one unit of compressed data is controlled, for example, by increasing the switching frequency between the layer of the original image and the layer of the differential image.

Next, an explanation will be presented of a device that generates the compressed moving image data discussed above. This device may also be implemented in a configuration similar to that of the image processing apparatus 10 shown in FIG. 4. This explanation will now be given with emphasis on the configuration of the control unit 100. In FIG. 15 shows a detailed configuration of a control unit 100b providing a function of generating compressed moving image data and a hard disk drive 50, in accordance with the present embodiment.

The control unit 100b comprises a planning unit 130, a hierarchical data generating unit 132, and a compressed data generating unit 134. The planning unit 130 determines the distribution order of the source image layer and the differential image layer. The hierarchical data generation unit 132 generates data of each layer in accordance with a certain distribution order. The compressed data generating unit 134 compresses and encodes the hierarchical data in accordance with a specific algorithm and generates a moving image stream. The hard disk drive 50 comprises a moving image data storage unit 136 that stores moving image data to be processed, and a compressed data memory unit 140 that stores compressed and encoded moving image data.

The moving image data being processed and stored in the moving image data storage unit 136 may be ordinary moving image data consisting of a sequence of frames, where frames presented with the same resolution level at the corresponding time points are arranged in chronological order. The planning unit 130 determines the distribution order of the source image layer and the differential image layer based on one of the distribution policies, which are explained with reference to FIG. 8 - FIG. 11. The number of layers is determined based on the resolution level of the frame of the original moving image. Which policy should be selected from the various distribution policies described above can be predefined in the control device, or the device can be configured so that the user can select a policy through input device 20. Alternatively, the choice of policy can be determined based on on the image characteristic and / or type of the moving image attached to the moving image data as metadata.

The hierarchical data generation unit 132 reads the data of the moving image to be processed from the moving image data memory unit 136 and compresses each frame in accordance with a predetermined set of discrete resolution levels, thereby forming hierarchical data consisting of the original image. In accordance with the distribution order defined by the planning unit 130, the hierarchical data generation unit 132 specifies for each frame a layer to be set as a differential image layer, and converts the specified layer data to the difference image data by subtracting it from the original image on the original image layer or from a differential image immediately above. Further, the hierarchical data generation unit 132 divides the image of each layer into images of a given size, thereby forming mosaic images.

The compressed data generating unit 134 performs compression encoding with one of the algorithms that were mentioned above with reference to FIG. 12 - FIG. 14, thereby generating a flow of a moving image. The generated moving image streams are stored in the compressed data storage unit 140. In this process, information regarding the configuration of the moving image stream (for example, information that relates the positions of the data in the moving image stream and the frame order of the original moving image) is attached, for example, to moving image header. Further, the correspondence relation between the mosaic image area on the image plane of each layer and the moving image stream, and information regarding the distribution order of the layers of the original image and the layers of the difference image are also attached.

Next, an explanation will be given of the operation of the respective devices implementing the configuration described above. In FIG. 16 shows a processing procedure where a compressed motion image generating apparatus generates compressed moving image data. First, the user selects the processed moving image data that is stored in the moving image data storage unit 136 of the hard disk drive 50 (S10), and the planning unit 130 determines the initial conditions, such as, for example, the number of layers to be generated, the planning policy, and determines, respectively, the distribution order of the layers of the original image and the layers of the difference image, based on these initial states, (S12 and S14).

Further, the hierarchical data generation unit 132 reads the moving image data to be processed from the moving image data memory unit 136 and generates hierarchical data, stepwise reducing each frame to several discrete sizes. Then, the hierarchical data generation unit 132 generates differential images on the layers of the differential image, updates the hierarchical data and divides the images on all hierarchical layers into mosaic images. By generating such hierarchical data for each frame at each instant of time, hierarchical data of a four-dimensional structure are created in which, as shown in FIG. 6, a time axis is added to the three virtual dimensions x, y, and z (S16).

Further, the compressed data generating unit 134 compresses and encodes the image data in the frame order shown in FIG. 12 - FIG. 14, and forms a moving image stream (S18). In this process, all the mosaic images can be represented as inner frames, but can also be represented as a mixture of the inner frame and the predicted frame or the inner frame and the previous and subsequent frames, on which other frames can depend. As a result, in the latter case, the differential image layer will store the data obtained as the time difference between the images in the resolution direction. As described above, the frame that follows immediately after the image type is switched in the sequence of frames is set as the internal frame.

Further, the compressed data generating unit 134 generates: a correspondence relation between the mosaic image region on the image plane of each layer and the moving image stream; information related to the distribution order of the source image layer and the differential image layer; and information related to the configuration of the moving image stream, and then attaches this information to the group of moving image streams to form the final compressed moving image data and stores this data in the hard disk drive 50 (S20).

In FIG. 17, a processing procedure is shown allowing the image reproducing apparatus to display a moving image on a screen. First, the user through the input device 20 enters a command to start playing back the moving image, and then the playback of the moving image on the display device 12 is started by the joint execution unit loading 108, decoding unit 112, image processing processing unit 14 and playback processor 44 (S30). The compressed moving image data may be data stored in the drive of the hard disk 50, or it may be data received through the network from the moving image server.

If the user requests the movement of the playback area by inputting the zooming operation through the input device 20 at a specific position of the reproduced moving image, the vertical or horizontal movement of the viewpoint or the like (Υ in S32), the software frame coordinate determining unit calculates the movement of the playback region velocity vector in virtual space from a signal requesting the movement of the playback area, and sequentially determines the coordinates of the frame at the corresponding time and frame playback (S34).

Regardless of whether the playback area moves or does not move (S34 or N in S32), the decoding unit 112, based on the z coordinate of the coordinates of the next frame, from all the layers included in the hierarchical data of the moving image, determines the layer that should be used for playback, and then the decoding unit 112 of the x coordinate and y coordinate specifies the flow of the moving image from the mosaic images corresponding to the playback area of the layer defined before, reads the stream from the operational memory 60, decodes this stream and stores the decoded data in buffer memory 70 (S36). The moving image stream is loaded into the RAM 60 by the joint operations of the loading area determination unit 106 and the loading unit 108. A similar decoding procedure is used regardless of whether the data to be decoded is original image data or differential image data.

Namely, if the frame is an internal frame, then the frame is decoded independently, and if the frame is a frame different from the internal frame, then the frame is decoded using the reference frame. In the case where the order of the data of the moving image stream is different from the sequence of frames of the original moving image, the data that is the processing object is specified based on the correspondence information attached to the moving image stream.

Next, the decoding unit checks whether or not the decoded image is a difference image (S38). Whether or not the decoded image is a differential image is determined by accessing information about the distribution order attached to the compressed data of the moving image, as described above. In an embodiment where switching is performed regularly (for example, switching is performed for each predetermined number of frames), the difference image layer may be specified based on, for example, the number of frames, and it can be immediately determined whether or not the decoded image is a difference image.

In the case where the decoded image is a difference image, the image is restored by decoding the image of the upper layer that describes the same area and adding the corresponding pixels, after which the data in the buffer memory 70 is updated (S40). In the case where the layer immediately above is also a difference image, a layer-by-layer search is performed up to the original image. In this process, the question of whether to sum the difference images sequentially or to add only the image directly on the layer of the original image can be solved specifically for each piece of moving image data.

Then, the image at the corresponding time is updated by the fact that the image processing processing unit 114, using mosaic image data stored in the buffer memory 70, writes them to render the image of the playback area in the frame buffer of the playback processor 44, and the playback processor 44 outputs the image to the display device 12 (S42). Repeating the processes described above with respect to each frame, the reproduction of a moving image is sequentially implemented, while ensuring the ability to move the playback area (Ν in S44, S32 - S42). If the playback of the moving image ends or if the user stops the playback, the process stops (Υ on S44).

According to the embodiment described above, configuring the hierarchical data, where each frame of the moving image is presented with several resolution levels, and forming a flow of the moving image from each mosaic image of the hierarchical data, the layer used for reproduction is switched to another layer, and the stream used the moving image switches to another stream in accordance with the movement of the playback area, including changes in resolution levels. In this process, at least one layer of hierarchical data is formed as a differential image from an image on a layer higher than this layer. Thus, an increase in the amount of data can be prevented even if the moving image is presented as hierarchically organized data.

Further, the distribution of the layers of the original image that store the data of the original image and the layers of the differential image that store the data of the differential image is switched between the layers forming the hierarchical data. Thus, even in the embodiment where only a portion of the moving image data is transmitted from the hard disk drive or the image server, the bit rate required for transmission can be averaged, and the bandwidth required for transmission can be reduced. As a result, even with a narrow bandwidth, data transfer can be carried out without affecting the speed of movement of the playback area, and even a huge image, in the order of gigapixels, can be reproduced with a small memory capacity.

Further, the distribution order of the layers of the original image and the layers of the differential image and / or the configuration of the flow of the moving image can be optimized in accordance with, for example, the characteristics of the moving image, processing performance, and therefore the data of the moving image, in accordance with an example implementation, can be used in most diverse application areas, from mobile terminals to universal computers.

The above explanation is based on exemplary embodiments. These embodiments are only considered illustrative, and it will be apparent to one skilled in the art that various modifications of components and processes can be developed, and that such modifications are also within the scope of the present invention.

For example, in accordance with an embodiment, the image data of a certain layer is stored mainly as an image of the difference between the enlarged image and the image on the layer with a resolution level lower than the resolution level of this certain layer. On the other hand, an image of the same layer (i.e., a unit resolution image) can be divided into regions and image data of a certain region can also be stored as a difference image with an image of another region. The “image of another region” referred to herein may be a source image or may be a difference image in which the image of another layer or another region is taken as the original image. Also in this case, by switching over time the area specified as the differential image and the area specified as the original image, the bit rate can be averaged and the bandwidth required for transmission can be reduced. This embodiment is particularly effective for an image containing a region that is almost monochrome and / or a region with repeating structures.

Description of Reference Numbers

1 - image processing system, 10 - image processing device, 12 - display device, 20 - input device, 30 - zero layer, 32 - first layer, 34 - second layer, 36 - third layer, 44 - playback processor, 50 - drive hard disk, 60 random access memory, 70 - buffer memory, 100 - control unit, 100b - control unit, 102 - input information receiving unit, 106 - loading area determination unit, 108 - loading unit, 110 - frame coordinate determination unit, 112 - decoding unit, 114 - image processing processing unit, 130 - planning block Vanya 132 - a block for generating hierarchical data, 134 - a block for generating compressed data, 136 - a block for storing data of moving images, and 140 - a block for storing compressed data.

Industrial applicability

As described above, the present invention can be applied to an information processing apparatus and information processing systems such as a computer, a gaming device, an image forming apparatus, an image rendering apparatus, and so forth.

Claims (19)

1. A device for generating data of moving images, forming hierarchical data of a moving image, where hierarchically arranged in order of resolution of a plurality of images, which are representations of image frames forming one moving image at different levels of resolution, to form a reproduced image on the screen, while switching used layer to another layer in accordance with the level of resolution required in the image processing device, which e reproduces a moving image, comprising:
a hierarchical data generating unit that generates image data of the respective layers of each image frame by reducing the size of each image frame, and difference image data representing the difference between the enlarged image and the image on another layer, which is a representation of that layer, is included in the image data of at least one layer the same image frame with a resolution level lower than the resolution of the aforementioned single layer, and thereby forms hierarchical data moving image;
a compressed data generating unit that compresses and encodes the hierarchical data of the moving image formed by said hierarchical data generating unit and writes compressed and encoded hierarchical data of the moving image in the storage device; and
a planning unit that determines the switching order of the layer containing the differential image data in the hierarchical data of the moving image, as time passes, to another layer,
wherein said hierarchical data generation unit determines a layer and an area defined as a differential image for each image frame, in accordance with the switching order determined by said planning unit, and forms hierarchical data of the moving image. 2. The device according to claim 1, characterized in that said scheduling unit switches for each region and does not save or does not save the difference image data of each layer, and switches the combination of the layer storing data of the said difference image and a layer different from this layer to another combination at a point in time that is common to all layers, with the exception of the layer with the lowest level of resolution among the layers forming the above-mentioned hierarchical data of a moving image. 3. The device according to claim 1, characterized in that the scheduling unit switches for each area and does not save the difference image data of each layer, and switches the combination of the layer storing the data of the difference image and a layer different from this layer to another combination at a time instant common to each of the many groups formed by partitioning the many layers that form the hierarchical data of the moving image, with the exception of the layer with the lowest resolution among the layers, the image Those mentioned hierarchical data of a moving image. 4. The device according to claim 1, characterized in that the planning unit switches for each region within the layer and saves or does not save the differential image data of each region in at least one of the layers, and switches the combination of the layer that stores the differential image data, and a layer different from the said layer to another combination at a point in time that is common for areas representing the same position in many layers for which the units switch areas. 5. The device according to any one of paragraphs. 1-4, characterized in that the planning unit switches the combination of the layer that stores the differential image data and the layer other than the layer to a different combination for each given number of image frames. 6. The device according to any one of paragraphs. 1-4, characterized in that the planning unit receives information about the moments of time of the scene change in the moving image, and switches the combination of the layer that stores the difference image data and the layer other than the mentioned layer to another combination at the time of the scene change. 7. The device according to any one of paragraphs. 1-4, characterized in that the planning unit switches the combination of the layer that stores the difference image data and the layer different from the layer to another combination at the time when the accumulated value of the data volume of the switching units per unit area reaches a predetermined threshold value. 8. The device according to any one of paragraphs. 1-4, characterized in that the unit for generating compressed data:
classifies chronological data of mosaic images formed by dividing with a certain image size of each layer in the hierarchical data of the moving image formed by the hierarchical data generation unit based on whether or not these data are differential image data;
generates new chronological data sequences by combining in accordance with a certain rule of chronological data of each classification group; and
compresses and encodes new chronological data sequences in order of data. 9. The device according to any one of paragraphs. 1-4, characterized in that the compressed data generation unit compresses and encodes the hierarchical data of the moving image, which are formed by the hierarchical data generation unit, for the respective chronological sequences of mosaic image data, which are formed by separation with a certain image size of each layer, are independently or are not this data is differential image data. 10. The device according to claim 8, characterized in that the compressed data generating unit sets at least the very first mosaic image data after switching to / from the differential image, and the very first mosaic image data contained in compressible chronological data sequences is taken as internal a frame that can be decoded independently. 11. The device according to any one of paragraphs. 1-4, characterized in that the compressed data generation unit combines, compresses and encodes chronological sequences of mosaic image data, which are formed by dividing with a predetermined image size of each layer in the hierarchical data of the moving image, formed by the hierarchical data generation unit for each same area on the moving image in many layers, and thereby forms a unit of compressed data;
in this case, one unit of compressed data is formed so that the uppermost layer of the many layers included in it stores the data of the necessary source image to restore the differential image, which stores a different layer from it. 12. A device for reproducing moving images, comprising:
a moving image data memory unit that stores hierarchical moving image data formed by a hierarchical organization in the order of resolution levels of many image sequences, which are representations of image frames forming one moving image with different resolution levels;
an input information obtaining unit that sequentially receives request signals for moving a reproduction area in a moving image that is displayed on a screen; and
an image reproduction processing unit that reproduces an image in accordance with a motion request signal of each image frame, while switching the layer used to another layer of many layers included in the hierarchical data of the moving image, in accordance with the required resolution determined by the motion request signal transmitted from the input information obtaining unit,
wherein, the image data of at least one layer of the many layers forming the hierarchical data of the moving image contains difference image data, which is the difference between the enlarged image and the image on another layer, which is a representation of the same image frame with a resolution level lower, than the resolution of the aforementioned one layer, and the layer containing the differential image data, as time passes, switches to another layer in the hierarchical data moving Gosia image
the image processing processing unit obtains information on the order of switching the layer containing the differential image data to another layer, and determines, based on these data, the data used for reproduction are or not the difference image data, and restores the image by enlarging and adding images on said another layer in the case where the data used in the reproduction is differential image data. 13. The device according to p. 12, characterized in that the layer in the hierarchical data of the moving image containing the differential image data is switched to another layer for each specified number of frames, and
the image reproducing processing unit, based on said predetermined number of frames, displays whether or not the data used for reproduction is differential image data, a definition of a layer containing differential image data for each image frame. 14. The device according to p. 12, characterized in that the data block of the moving image data also stores information about the switching order, while linking the above information with the hierarchical data of the moving image, and
wherein the image processing processing unit determines whether or not the data used for reproduction is differential image data, referring to the switching order. 15. The device according to any one of paragraphs. 12-14, characterized in that the hierarchical data of the moving image contains several units of compressed data, which are compressed and encoded data in a chronological order different from the chronological order of frames in a moving image, and which are classified depending on whether or not mentioned differential image data
where the moving image data memory unit also stores correspondence information that interconnects the order of the image frames in the moving image and the data order in the unit of compressed data, while linking the above information with the hierarchical data of the moving image, and
wherein the image reproduction processing unit reorders the decoded data in the frame order of the original image based on the correspondence information. 16. A method of generating moving image data forming hierarchical moving image data that is hierarchically organized in order of resolution levels of a plurality of image sequences, which are representations of image frames forming one moving image, at different resolution levels, to form a reproduced image, while switching used layer to another layer in accordance with the level of resolution required in the processing device expressions, which reproduces a moving picture, comprising:
a step in which moving image data comprising sequences of image frames presented with one resolution level is read from the storage device;
a step in which image data of corresponding layers of each image frame is formed by several reductions in the size of each image frame to several sizes;
a step in which differential image data is included in the image data of at least one layer, which is the difference between the enlarged image and the image on another layer, which is a representation of the same image frame with a resolution level lower than the resolution of said one layer, and thereby forms hierarchical data of a moving image; and
a step in which the hierarchical data of the moving image is compressed and encoded, and said data is recorded in a storage device,
however, the method of generating moving image data also includes the step of determining the switching order of the layer containing the differential image data in the hierarchical data of the moving image, as time passes, to another layer,
and a step of including differential image data, on which a layer and an area defined as a differential image are determined for each image frame, in accordance with the switching order, and hierarchical data of the moving image are generated. 17. A method for reproducing moving images, comprising:
the stage at which at least a part of the hierarchical data of the moving image is read out from the storage device, which are hierarchically arranged in order of resolution levels of a plurality of image sequences, which are representations of image frames forming one moving image, at different resolution levels, and starting playback of the moving image on a display device using at least a portion of the hierarchical data of the moving image;
the step of receiving a request signal to move the playback area in the moving image that is displayed on the screen;
a step in which a reproduced image is formed in accordance with the motion request signal of each image frame, while switching the used layer to another layer of many layers included in the hierarchical data of the moving image, in accordance with the required resolution determined by the motion request signal; and
the stage at which the generated image is displayed on the screen of the display device;
wherein, the image data of at least one layer of many layers forming the hierarchical data of the moving image contains difference image data, which is the difference between the enlarged image and the image on another layer, which is a representation of the same image frame with a resolution level lower than resolution of said one layer, and the layer containing the difference image data, as time passes, switches to another layer in the hierarchical data moving gosya images
where the stage at which the reproduced image is formed includes obtaining information about the order of switching the layer containing the differential image data to another layer, and determining, based on this data, the data used for reproduction are or are not the difference image data, and restoration images by enlarging and adding an image on said other layer in the case where the data used in the reproduction is differential image data. 18. A computer-readable recording medium on which a computer program is recorded that provides the computer with the function of generating hierarchical data of moving images, where hierarchically arranged in the order of resolution of a plurality of images that are representations of image frames forming one moving image at different levels of resolution, to form the image displayed on the screen, while switching the used layer to another layer in accordance a video player with a resolution level required in an image processing apparatus that reproduces a moving image, including:
a function that reads from a storage device moving image data consisting of sequences of image frames presented with one resolution level;
a function that generates image data of the respective layers of each image frame by reducing the size of each image frame in many sizes;
a function that includes in the image data of at least one layer a difference image data representing the difference between the enlarged image and the image on another layer, which is a representation of the same image frame with a resolution level lower than the resolution of said one layer, and thereby forms hierarchical data of a moving image; and
a function that compresses and encodes the hierarchical data of the moving image, and writes compressed and encoded hierarchical data of the moving image in the storage device, while
the computer program also includes a function that determines the order in which the layer containing the differential image data in the hierarchical data of the moving image is switched over as time passes to another layer.
and a function including differential image data that contains a function that defines a layer and an area defined as a differential image for each image frame, in accordance with the switching order determined by said scheduling unit, and forms hierarchical data of the moving image. 19. A computer-readable recording medium on which a computer program is recorded that enables the computer to execute:
a function that reads from the storage device the hierarchical data of a moving image formed by a hierarchical organization in order of resolution levels of many image sequences that are representations of image frames forming one moving image with different levels of resolution, and starts playback of the moving image on the display device using at least parts of the hierarchical data of the moving image;
a function that sequentially receives request signals to move a reproduction area in a moving image that is displayed on a screen;
a function that reproduces the image in accordance with the motion request signal of each image frame, while switching the used layer to another layer of many layers included in the hierarchical data of the moving image, in accordance with the required resolution determined by the motion request signal; and
a function that reproduces the generated reproduced image on the screen of the display device,
wherein, the image data of at least one layer of the many layers forming the hierarchical data of the moving image contains difference image data, which is the difference between the enlarged image and the image on another layer, which is a representation of the same image frame with a resolution level lower, than the resolution of the aforementioned one layer, and the layer containing the differential image data, as time passes, switches to another layer in the hierarchical data moving Gosia image
the function that generates the image reproduction includes functions that receive information about the order of switching the layer containing the difference image data to another layer, and determine, based on this data, whether the data used for reproduction is or not a differential image, and restore the image by enlarging and adding an image on said other layer in the case when the data used in the playback are differential image data tions.

Images