JP5520890B2 - Image processing apparatus, image data generation apparatus, image processing method, image data generation method, and data structure of image file - Google Patents

Description

  The present invention relates to an image display technique for displaying an image by changing a display area in response to a display area movement request.

  Home entertainment systems have been proposed that not only execute game programs, but also play video. In this home entertainment system, the GPU generates a three-dimensional image using polygons (see, for example, Patent Document 1).

  Regardless of moving images or still images, how to display images efficiently is always an important issue. For this reason, various technologies such as image data compression technology, transmission technology, image processing technology, and display technology have been developed and put into practical use, and high-definition images can be enjoyed in various situations.

US Pat. No. 6,563,999

  There is always a demand to display a high-definition image with high responsiveness according to a user's request. For example, in order to realize an image display with a high degree of freedom with respect to the user's viewpoint, such as enlarging and displaying a region that the user wants to focus on or displaying another region of the displayed whole image. Therefore, it is necessary to enable random access while processing large-size image data in a short time, and further technical progress is required.

  The present invention has been made in view of such problems, and an object thereof is to provide an image processing technique capable of displaying a high-definition image with high responsiveness to an operation input related to a display area by a user. is there.

  One embodiment of the present invention relates to an image processing apparatus. This image processing apparatus is an image processing apparatus that displays an image by changing a display area in response to an input display area movement request, and data of a plurality of component images that are displayed as a display image by being displayed in a superimposed manner An image data storage unit for storing compressed data compressed for each tile image divided by a predetermined size, and compression of a tile image corresponding to a display area among the compressed data stored in the image data storage unit Display image processing that decodes and decodes data for each component image and overlays and combines tile images of multiple component images decoded by the decoder to generate a display area image and display it on the display device And a section.

  Another embodiment of the present invention relates to an image data generation apparatus. This image data generation device accepts designation of a plurality of component images that become display images by displaying them in a superimposed manner, and compresses the data of each component image for each tile image divided by a predetermined size. Of the compressed data generated by the data generation unit and the tile data generation unit, the composite data is generated by connecting the compressed data of the tile images of the component images that represent the same area on the display image, and the position on the display image And a composite data generation unit that outputs the image data of the display image to the storage device by associating with each other.

  Yet another embodiment of the present invention relates to an image processing method. This image processing method is an image processing method in which an image processing apparatus displays an image by changing a display area in response to an input display area movement request. Reading compressed data of tile images corresponding to the display area for each of the constituent images from a storage device storing compressed data obtained by compressing the constituent image data of each of the constituent images by a predetermined size for each tile image And decoding the compressed data of each read component image, and superimposing and synthesizing the decoded tile images of the component images to generate an image of the display area and displaying it on the display device It is characterized by that.

  Still another embodiment of the present invention relates to an image data generation method. This image data generation method is a method in which the image data generation device generates image data, the step of accepting designation of a plurality of component images to be displayed by superimposing and displaying each component image from a memory A step of reading data and compressing each tile image divided by a predetermined size and connecting compressed data of tile images of each component image representing the same area on the display image among the generated compressed data Generating composite data and associating it with a position on the display image, and outputting it as image data of the display image to a storage device.

  Still another embodiment of the present invention relates to a data structure of an image file. This data structure is a data structure of an image file that is displayed by changing the display area in response to a display area movement request input in the image processing apparatus. Of the compressed data obtained by compressing the component image data for each tile image divided by a predetermined size, the composite data connecting the tile image compressed data of each component image representing the same area on the display image and In the image processing apparatus, the composite data corresponding to the display area is specified by associating the position on the display image, and the compressed data included in the composite data is decoded, synthesized, and displayed. And

  It should be noted that any combination of the above-described constituent elements and a representation of the present invention converted between a method, an apparatus, a system, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, it is possible to display an image with high responsiveness to a user instruction while maintaining a required image quality.

It is a figure which shows the use environment of the image processing system which can be applied to this Embodiment. It is a figure which shows the example of an external appearance structure of the input device applicable to the image processing system of FIG. It is a figure which shows the structure of the image processing apparatus in this Embodiment. It is a conceptual diagram of the hierarchical structure of the image data used in this Embodiment. In this Embodiment, it is a figure for demonstrating the relationship between the image made into a subhierarchy, and a final display image. It is a figure which shows the structure of the image data generation apparatus in this Embodiment in detail. It is a figure which shows typically the procedure of the image data generation process by the image data generation apparatus in this Embodiment. It is a figure which shows typically the structure of the composite data in this Embodiment. It is a figure which shows the structure of the control part of the image processing apparatus in this Embodiment in detail. 4 is a flowchart illustrating a processing procedure in which an image processing apparatus displays an image in response to a display area movement request from a user in the present embodiment.

  The image data to be processed in the present embodiment has a hierarchical structure composed of images with different resolutions generated by reducing the original image in a plurality of stages. Each layer image is divided into one or a plurality of tile images. For example, the image with the lowest resolution is composed of one tile image, and the original image with the highest resolution is composed of the largest number of tile images. At the time of image display, the tile image used for drawing is switched to a tile image of a different hierarchy when the display image has a predetermined resolution, so that enlarged display or reduced display is quickly performed.

  First, a basic display mode of an image having such a hierarchical structure will be described. FIG. 1 shows a use environment of an image processing system to which an embodiment of the present invention can be applied. The image processing system 1 includes an image processing device 10 that executes an application program including image processing, and a display device 12 that outputs a processing result by the image processing device 10. The display device 12 may be a television having a display that outputs an image and a speaker that outputs sound.

  The display device 12 may be connected to the image processing device 10 by a wired cable, or may be wirelessly connected by a wireless local area network (LAN) or the like. In the image processing system 1, the image processing apparatus 10 may be connected to an external network such as the Internet via the cable 14 to download and acquire content including hierarchical compressed image data. The image processing apparatus 10 may be connected to an external network by wireless communication.

  In response to a request from the user, the image processing apparatus 10 performs processing for changing the display area, such as enlargement / reduction processing of an image displayed on the display of the display device 12 and scroll processing in the vertical and horizontal directions. When the user operates the input device while viewing the image displayed on the display, the input device transmits a display area movement request signal to the image processing device 10.

  FIG. 2 shows an external configuration example of the input device 20. The input device 20 includes a cross key 21, analog sticks 27a and 27b, and four types of operation buttons 26 as operation means that can be operated by the user. The four types of operation buttons 26 include a circle button 22, a x button 23, a square button 24, and a triangle button 25.

  In the image processing system 1, a function for inputting a display image enlargement / reduction request and a vertical / left / right scroll request is assigned to the operation unit of the input device 20. For example, the input function of the display image enlargement / reduction request is assigned to the right analog stick 27b. The user can input a display image reduction request by pulling the analog stick 27b forward, and can input a display image enlargement request by pressing the analog stick 27b from the front.

  Also, the input function of the scroll request in the vertical and horizontal directions is assigned to the cross key 21. The user can input a movement request in the direction in which the cross key 21 is pressed by pressing the cross key 21. These request input functions may be assigned to another operation means. For example, a scroll request input function may be assigned to the analog stick 27a. Hereinafter, the enlargement / reduction of the display image and the scrolling in the vertical and horizontal directions may be collectively referred to as “movement of the display area”.

  The input device 20 has a function of transmitting an input display area movement request signal or the like to the image processing device 10 and is configured to be capable of wireless communication with the image processing device 10 in the present embodiment. The input device 20 and the image processing device 10 may establish a wireless connection using a Bluetooth (registered trademark) protocol, an IEEE802.11 protocol, or the like. The input device 20 may be connected to the image processing device 10 via a cable and transmit a display area movement request signal or the like to the image processing device 10.

  FIG. 3 shows the configuration of the image processing apparatus 10. The image processing apparatus 10 includes a wireless interface 40, a switch 42, a display processing unit 44, a hard disk drive 50, a recording medium mounting unit 52, a disk drive 54, a main memory 60, a buffer memory 70, and a control unit 100. . The display processing unit 44 has a frame memory (not shown) that buffers data to be displayed on the display of the display device 12.

  The switch 42 is an Ethernet switch (Ethernet is a registered trademark), and is a device that transmits and receives data by connecting to an external device in a wired or wireless manner. The switch 42 is configured to connect to an external network via the cable 14 and receive image data from the server. The switch 42 is connected to the wireless interface 40, and the wireless interface 40 is connected to the input device 20 using a predetermined wireless communication protocol. A signal input from the user in the input device 20 is supplied to the control unit 100 via the wireless interface 40 and the switch 42.

  The hard disk drive 50 functions as a storage device that stores data. Image data received via the switch 42 is stored in the hard disk drive 50. When a removable recording medium such as a memory card is mounted, the recording medium mounting unit 52 reads data from the removable recording medium. When a read-only ROM disk is loaded, the disk drive 54 drives and recognizes the ROM disk to read data. The ROM disk may be an optical disk or a magneto-optical disk. Image data may be stored in these recording media.

  The control unit 100 includes a multi-core CPU, and includes one general-purpose processor core and a plurality of simple processor cores in one CPU. The general-purpose processor core is called PPU (PowerPC Processor Unit), and the remaining processor cores are called SPU (Synergistic Processor Unit).

  The control unit 100 includes a memory controller connected to the main memory 60 and the buffer memory 70. The PPU has a register, has a main processor as an operation execution subject, and efficiently assigns a task as a basic processing unit in an application to be executed to each SPU. Note that the PPU itself may execute the task. The SPU has a register, and includes a sub-processor as an operation execution subject and a local memory as a local storage area. The local memory may be used as the buffer memory 70.

  The main memory 60 and the buffer memory 70 are storage devices and are configured as a RAM (Random Access Memory). The SPU has a dedicated DMA (Direct Memory Access) controller as a control unit, can transfer data between the main memory 60 and the buffer memory 70 at high speed, and the frame memory and the buffer memory 70 in the display processing unit 44 can be transferred. High-speed data transfer can be realized. The control unit 100 according to the present embodiment realizes a high-speed image processing function by operating a plurality of SPUs in parallel. The display processing unit 44 is connected to the display device 12 and outputs an image processing result according to a request from the user.

  The image processing apparatus 10 according to the present embodiment transfers a part of the compressed image data from the hard disk drive 50 to the main memory in order to smoothly change the display image when the display image enlargement / reduction processing or scroll processing is performed. 60 is loaded. Further, a part of the compressed image data loaded in the main memory 60 is decoded and stored in the buffer memory 70. This makes it possible to instantaneously switch an image used for generating a display image at a necessary later timing.

  FIG. 4 shows a conceptual diagram of a hierarchical structure of image data used in the present embodiment. The image data has a hierarchical structure including a 0th hierarchy 30, a first hierarchy 32, a second hierarchy 34, and a third hierarchy 36 in the depth (Z-axis) direction. Although only four layers are shown in the figure, the number of layers is not limited to this. Hereinafter, image data having such a hierarchical structure is referred to as “hierarchical data”.

  The hierarchical data shown in FIG. 4 has a quadtree hierarchical structure, and each hierarchy is composed of one or more tile images 38. All the tile images 38 are formed in the same size having the same number of pixels, and have, for example, 256 × 256 pixels. The image data of each layer expresses one image at different resolutions, and the original image of the third layer 36 having the highest resolution is reduced in a plurality of stages to obtain the second layer 34, the first layer 32, the 0th layer. Image data of the hierarchy 30 is generated. For example, the resolution of the Nth layer (N is an integer greater than or equal to 0) may be ½ of the resolution of the (N + 1) th layer in both the left and right (X axis) direction and the up and down (Y axis) direction.

  In the image processing apparatus 10, the hierarchical data is held in the storage device for each tile image 38 in a state compressed in a predetermined compression format, and is read from the storage device and decoded before being displayed on the display. . The image processing apparatus 10 according to the present embodiment has a decoding function corresponding to a plurality of types of compression formats, and can decode, for example, compressed data in JPEG format, JPEG2000 format, PNG format, and S3TC format.

  As shown in FIG. 4, the hierarchical structure of the hierarchical data is set with the horizontal direction as the X axis, the vertical direction as the Y axis, and the depth direction as the Z axis, thereby constructing a virtual three-dimensional space. When the image processing apparatus 10 derives the movement amount of the display area from the display area movement request signal supplied from the input device 20, the image processing apparatus 10 derives the coordinates (frame coordinates) of the four corners of the frame in the virtual space using the movement amount. . The frame coordinates in the virtual space are used for loading compressed data into the main memory and generating a display image.

  For example, for changing the resolution of the display image, a switching boundary is set between the layers on the Z axis. Then, when the Z coordinate of the frame exceeds the switching boundary by the display area movement request signal, the hierarchy used for drawing the display image is switched. Then, loading and decoding are performed as necessary, and a display image is generated by enlarging / reducing in accordance with the requested resolution. Instead of the frame coordinates in the virtual space, the image processing apparatus 10 may derive information for specifying the hierarchy and texture coordinates (UV coordinates) in the hierarchy. Hereinafter, the combination of the hierarchy specifying information and the texture coordinates is also referred to as frame coordinates.

  In the present embodiment, the entire image of at least one of the hierarchical data as described above or a partial area thereof is further hierarchized into individual compressed data. Hereinafter, the hierarchy thus formed is referred to as a sub-hierarchy. The sub-hierarchy images constituting the same hierarchy have the same resolution, and are created so as to be the final image by being superimposed and synthesized. That is, the image processing apparatus 10 individually decodes the compressed data of a plurality of sub-layers, and combines the decoded images to generate a display image. By doing in this way, the compression format suitable for every subhierarchy can be selected.

  FIG. 4 shows that the entire second hierarchy 34 is composed of a first sub-hierarchy 35a and a second sub-hierarchy 35b, and the entire third hierarchy 36 is composed of a first sub-hierarchy 37a and a second sub-hierarchy 37b. ing. The first subhierarchy 35a of the second hierarchy 34 is an image obtained by reducing the first subhierarchy 37a of the third hierarchy 36. The second subhierarchy 35b of the second hierarchy 34 is the second subhierarchy 37b of the third hierarchy 36. This is a reduced image. The images of the 0th hierarchy 30 and the 1st hierarchy 32 are reduced images after combining the subhierarchies of the 3rd hierarchy 36, and are compressed by one compression format. In the figure, there are two hierarchies, the second hierarchy 34 and the third hierarchy 36, but the number of sub hierarchies is not limited. Further, the number of sub-hierarchies constituting the same hierarchy may be three or more.

  FIG. 5 is a diagram for explaining the relationship between an image as a sub-hierarchy and a final display image. For example, it is assumed that the image creator creates a final image 84 by performing lettering as shown in the image 82 on the photographic image 80. In such a case, the final image 84 is generally compressed in one compression format. When JPEG is adopted here, JPEG is irreversible compression, and therefore, the higher the compression rate, the more data is lost, and as a result, there is a tendency for image quality to deteriorate, for example, block noise occurs in the display image.

  The deterioration of the image quality becomes more conspicuous as the image is enlarged and characters and illustrations are included in the image. For this reason, it is necessary to lower the compression rate in order to maintain the image quality of characters and illustrations even at a high enlargement rate. In this case, however, the data size naturally increases and the time required for data transfer and decoding increases. As a result, the responsiveness from the display area movement request to the image display deteriorates.

  On the other hand, when PNG is used, there is no deterioration in image quality because of lossless compression, but the compression rate decreases as the complexity of images such as natural images increases. Thus, the compression format suitable for achieving a good balance between the compression rate and the image quality differs depending on the nature of the image. Therefore, the image to be created is classified into a plurality of images according to the difference in properties and compressed in a compression format suitable for each, and the above-described sub-layer is formed.

  In the example of FIG. 5, even if the compression rate is increased with JPEG, an image consisting only of lettered characters is compressed with JPEG at a relatively high compression rate for a photographic image 80 that does not affect the appearance at the time of display. 82 is compressed by PNG. Since an image consisting only of characters and illustrations such as the image 82 is generally local and has low complexity, it is unlikely that the size of the compressed data will increase. In this example, the alpha value is set so that the background of the image 82 other than the character string is transparent.

  The configuration of the sub-layer as described above is particularly effective in a high-resolution layer used for image display at a high magnification ratio among the layer data. On the other hand, when the enlargement ratio of the display image is low, the character string is also displayed in a small size, so that deterioration is not easily recognized. In addition, since an image with a low resolution has a higher complexity per unit area than an image with a high resolution, compression with PNG tends to lower the compression rate. Therefore, the lower resolution layer is not provided with a sub-layer, and the combined image 84 is compressed with JPEG at a high compression rate to reduce the data size.

  Note that the compression rate when compressing with JPEG may be appropriately set by the image creator in consideration of the image quality at the time of display. Thus, by changing the presence or absence of the sub-hierarchy depending on the hierarchy in the hierarchy data and compressing in a compression format suitable for each, it is possible to maintain an image quality that can withstand zooming while suppressing the data size. As a result, a high-quality image can be displayed with high responsiveness regardless of the resolution.

  Next, an aspect of generating the hierarchical data as described above will be described. An image data generation apparatus that realizes this aspect can be realized by the same configuration as the image processing apparatus 10 shown in FIG. Hereinafter, the description will focus on the configuration related to image data generation, and illustration and description of other configurations will be omitted as appropriate. Also, the same reference numerals are given to the same components as those shown in FIGS.

  FIG. 6 shows the configuration of the image data generation apparatus in detail. The control unit 100a of the image data generation device 90 includes an input information acquisition unit 91 that acquires an instruction input performed by the image creator, an image processing support unit 92 that performs image processing and image display according to the instruction input, and supports image generation. The tile data generation unit 94 that generates image data representing the created image at a plurality of resolutions and further generates tile image data by dividing the data into sub-hierarchical data for each tile image area. A composite data generation unit 96 for data of one tile image is included.

  In FIG. 6 and FIG. 9 to be described later, each element described as a functional block for performing various processes can be configured by a CPU (Central Processing Unit), a memory, and other LSIs in terms of hardware. Specifically, it is realized by a program loaded in a memory. As described above, the control units 100 and 100a have one PPU and a plurality of SPUs, and each of the PPUs and SPUs can be configured individually or in cooperation to constitute each functional block. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  The input information acquisition unit 91 acquires information on an instruction input performed by the image creator via the input device 20. Specifically, the instruction input includes start and end of image creation, reading of an image file, drawing and editing of an image, confirmation, designation of a compression format, and the like. The input device 20 may be any of commonly used input devices such as a keyboard, a mouse, a touch pen, a touch panel, and a button in addition to the one having the appearance shown in FIG.

  The image creation support unit 92 displays an image creation screen on the display device 12 and changes the image being created displayed on the screen in response to an instruction input from the user acquired by the input information acquisition unit 91. The screens and functions displayed by the image creation support unit 92 can be those provided by a general image creation apparatus. When the image creator inputs an instruction to confirm the image, the image data at that time is stored in the main memory 60. When the image at this time is composed of a plurality of layers and an instruction is input with the plurality of layers as sub-hierarchies, the image data of each layer is stored. At this time, designation of the compression format of each layer is received from the image creator and recorded in association with the data of each image.

  The tile data generation unit 94 generates image data having a plurality of resolutions corresponding to each layer of the layer data based on the image data stored in the main memory 60. At this time, for the hierarchy in which the sub-hierarchy is provided, that is, in the example of FIG. 4, the second hierarchy 34 and the third hierarchy 36, image data with a plurality of resolutions is generated for each image of the sub-hierarchy. For the other hierarchies, that is, in the example of FIG. 4, the 0th hierarchy 30 and the first hierarchy 32, image data with a plurality of resolutions is generated from an image obtained by combining the images of the subhierarchy. Which hierarchy is to be provided with the sub-hierarchy may be set in advance within the apparatus, or may be designated by the image creator.

  The tile data generation unit 94 divides each image into predetermined tile sizes on the image plane to form tile images, compresses them in a compression format designated by the user, and stores them in the main memory 60. At this time, the number representing the hierarchy and the position in the hierarchy in the hierarchy data is used as the identification number of the tile image, and data that associates the identification number with the storage area of the compressed data of the tile image is generated as an index. For an area having a sub-hierarchy, compressed data of a plurality of tile images corresponding to the sub-hierarchy is associated with the same identification number.

  The composite data generation unit 96 collects a plurality of compressed data associated with the same identification number in the index as continuous data. At this time, a tag indicating that the data is composite data including a plurality of image data corresponding to the sub-hierarchy is added to the data. Then, the index is updated using the composite data as compressed data of a new tile image. The set of compressed tile image data and index thus generated is stored in the compressed data storage unit 98 in the hard disk drive 50 as final hierarchical data.

  FIG. 7 schematically shows a procedure of image data generation processing by the image data generation device 90. First, when the image creator uses the image data generation device 90 to perform operations such as reading, drawing, and editing of an image file as a material, the input information acquisition unit 91 and the image creation support unit 92 The image to be created is changed in a general procedure according to the instruction input. When the image creator inputs a confirmation instruction, the image creation support unit 92 stores the image data in the main memory 60.

  The image 220 in FIG. 7 is created as described above, and includes two layers, that is, the configuration image 221a and the configuration image 221b. For example, the image creation support unit 92 displays the first and second drawing areas (layout frames) on the display device 12, and the image creator creates the configuration images 221a and 221b in the respective drawing areas. These component images are images that are created on the assumption that they are superimposed and combined, and the order of overlapping, the alpha value, and the like are also set.

  The second drawing area may be overlaid on the first drawing area, and the contents of the configuration image 221b may be drawn directly on the configuration image 221a. Thus, the method of creating one image by a plurality of layers is the same as that provided by a general image creation apparatus. When the image creator determines the image, the image creator also designates a compression format for each of the component images 221a and 221b as a sub-hierarchy.

  Based on the image data stored in the main memory 60 in this way, the tile data generation unit 94 creates image data of a plurality of resolutions corresponding to the hierarchy of the hierarchy data, and divides each into tile images of a predetermined size. And compress. In the example of FIG. 7, the original image 220 is used as the lowest layer of the hierarchical data, and hierarchical data consisting of four layers is generated. In addition, it is assumed that the lowest layer and the layer above it are set by sub-hierarchies.

  At this time, the tile data generation unit 94 first divides the constituent images 221a and 221b into tile images, and compresses the tile images for each tile image in the compression format designated by the image creator (S12). The positional relationship of each compressed data on the image is clarified by creating an index as described above. Next, the tile data generation unit 94 reduces the configuration images 221a and 221b, respectively, and generates data of the image 222 of the next higher level (S14). Therefore, the image 222 also includes two component images corresponding to the component images 221a and 221b. Then, each is divided into tile images and compressed for each tile image in the designated compression format (S16).

  Next, the tile data generation unit 94 generates an image 224 that is one level higher. However, since the hierarchy is set to have no sub-hierarchy, the two component images of the image 222 are combined. Reduce (S18). Then, the combined image is divided into tile images and compressed for each tile image in a predetermined compression format (S20). Further, the image 224 is reduced to generate an uppermost image 226 (S22), which is divided into tile images and compressed in the same manner (S24).

  Of the hierarchical data generated in this way, for the hierarchy of the data group 228 having the sub-hierarchy and the hierarchy of the data group 230, the composite data generation unit 96 has a plurality of compressed data (for example, data 236) of tile images representing the same area. ) Are combined into one tile image data. In the example of FIG. 7, a sub-hierarchy is provided for the entire area of the image. However, since image data is handled in units of tile images in this way, only some tile images may be configured with a sub-hierarchy as described above. .

  FIG. 8 schematically shows the structure of compressed data of a tile image having a sub-hierarchy, that is, composite data. Although this example shows a case where one hierarchy is composed of two sub-layers as shown in FIG. 7, the number of sub-layers is not limited. The composite data 140 includes a tag unit 142, a first offset unit 144, a second offset unit 146, a first image data unit 148, and a second image data unit 150. The tag portion 142 stores information indicating that this data is composite data. In the example of the figure, a character string “combine” is stored. A number representing the number of compressed data included may be stored.

  The first offset unit 144 stores the first address of the storage area of the first compressed data when the first address of the composite data 140 is 0, “offset1” in the example of FIG. Similarly, the second offset unit 146 stores the start address of the storage area of the second compressed data, “offset2” in the example of FIG. The first image data unit 148 and the second image data unit 150 store compressed data bodies (illustrated as “first image data” and “second image data”, respectively) of tile images that are sub-hierarchies. For example, “first image data” and “second image data” are compressed data of partial areas of the constituent images 221a and 221b in FIG. 7, respectively. The structure of the compressed data body is standardized for each compression format.

  Next, an aspect in which an image is displayed using the hierarchical data as described above will be described. This aspect is realized by the image processing apparatus 10 shown in FIGS. FIG. 9 shows the configuration of the control unit 100 in the image processing apparatus 10 in detail. The control unit 100 newly loads an input information acquisition unit 102 that acquires information input by the user, that is, an image viewer, from the input device 20, a frame coordinate determination unit 110 that determines the frame coordinates of the frame to be displayed next, and the like. A load unit 108 for determining a tile image to be loaded and loading compressed data from the hard disk drive 50 is included. The control unit 100 further includes a decoding unit 112 that decodes compressed data, and a display image processing unit 114 that draws an image of a frame.

  The input information acquisition unit 102 acquires instruction contents such as start / end of image display, selection of an image file, movement of a display area, and the like input by the user to the input device 20. The frame coordinate determination unit 110 determines the frame coordinates of each time determined by the frame rate according to the frame coordinates of the current display area and the display area movement request information input by the user. Information on the frame coordinates is notified to the loading unit 108, the decoding unit 112, and the display image processing unit 114, respectively.

  The load unit 108 determines a tile image to be loaded into the main memory 60 based on information from the frame coordinate determination unit 110, and if there is a tile image that has not yet been loaded, the load unit 108 sets the tile image data. Loading from the hard disk drive 50 into the main memory 60. The tile image to be loaded into the main memory 60 is the tile image included in the frame notified from the frame coordinate determination unit 110, the tile image within a predetermined range around it, and the transition of the past frame coordinates The tile image of the area predicted to be necessary from now on. Data other than the data necessary for drawing the display image immediately after the image may be loaded constantly at a predetermined time interval, for example.

  The decoding unit 112 includes an analysis unit 120 and a plurality of decoding means such as a first decoding unit 122a, a second decoding unit 122b,. The decoding unit 112 basically reads and decodes tile image data from the main memory 60 based on the frame coordinate information acquired from the frame coordinate determination unit 110 and stores the data in the buffer memory 70. At this time, among the data of the frame to be displayed immediately after that, the data not stored in the buffer memory 70 is preferentially decoded, and the surrounding area and the area predicted to be displayed in the future are decoded according to a predetermined rule. .

  The analysis unit 120 of the decoding unit 112 determines whether the data read from the main memory 60 is composite data based on the presence / absence of information in the tag unit 142. A plurality of decoding means such as the first decoding unit 122a, the second decoding unit 122b,... Decode the compressed data in parallel for each tile image.

  When the data read from the main memory 60 is composite data, the analysis unit 120 uses the first image data unit 148 and the second image data based on the information stored in the first offset unit 144 and the second offset unit 146. The data stored in the unit 150 is separated. Each of them is assigned to different decoding means as individual compressed data. As a result, the separated compressed data is decoded in parallel.

  Thus, when one tile image is composed of sub-hierarchies and each compressed data is decoded in parallel, the decoding time can be shortened as compared to the case where one long compressed data is sequentially decoded without providing the sub-hierarchy. For example, when the latter data is JPEG compressed at a low compression rate in order to maintain image quality, the difference becomes significant. In the data read from the main memory 60, when there is no information indicating that the data is composite data in the area of the tag part 142, that is, the header part of the data, the analysis part 120 assigns the data to one of a plurality of decoding means. Just do it.

  Regardless of whether or not the original data is composite data, the first decoding unit 122a, the second decoding unit 122b,... Compress the allocated data in the same way as normal compressed data. Decode according to format. However, if the original data is composite data, the data of the sub-hierarchy of each tile image is associated by managing with the same identification number used in the above-described index.

  The display image processing unit 114 includes a combining unit 126 and a drawing unit 128. The drawing unit 128 reads the corresponding image data from the buffer memory 70 based on the frame coordinates of the next frame to be displayed, acquired from the frame coordinate determination unit 110, and draws the frame image in the frame memory of the display processing unit 44. To do. At this time, if an area composed of sub-hierarchies is included in the frame, the composition unit 126 superimposes these images to compose one image. A general alpha blending technique may be used for the image composition process.

  Next, the operation of the image processing apparatus 10 will be described. FIG. 10 is a flowchart illustrating a processing procedure in which the image processing apparatus displays an image in response to a display area movement request from the user. First, when the user inputs an instruction to start image display, the image processing apparatus 10 displays an initial image (S30). Specifically, the load unit 108 loads initial image data based on the display start information from the input information acquisition unit 102, the decode unit 112 decodes, and the display image processing unit 114 draws the image in the frame memory. 44 is displayed on the display device 12.

  In this state, when the user inputs a display area movement request via the input device 20 (S32), the frame coordinate determination unit 110 determines the frame coordinates of the movement destination and notifies each block (S34). The load unit 108 identifies a tile image to be loaded based on the notified frame coordinates, and loads compressed data of the tile image from the hard disk drive 50 to the main memory 60 (S36). Specifically, first, a range necessary for future image drawing is determined in the virtual space, and compressed data of a tile image within the range is identified and read with reference to an index.

  Based on the notified frame coordinates, the decoding unit 112 reads out necessary compressed data from the compressed data of the tile image stored in the main memory 60, and determines whether the data is composite data (S38). If it is composite data (Y in S38), a plurality of compressed data included in it is separated and decoded in parallel by separate decoding means (S40). If it is not composite data (N in S38), it is decoded by one of the decoding means (S44).

  Based on the notified frame coordinates, the display image processing unit 114 draws a frame image using the decoded data (S42, S46). At this time, if the decoded data is sub-hierarchical data, the final image is drawn after combining them (S42). The display processing unit 44 displays the image of the frame drawn in this way on the display device 12, whereby the image of the destination area is displayed (S48).

  According to the present embodiment described above, in the image display technology that accepts a display area movement request from a user, one image is composed of a plurality of layers and is individually compressed on the premise that they are combined at the time of display. . For example, by separating the layers according to the nature of the image, such as a natural image such as a photograph and a lettering image, it is possible to select a compression format suitable for each property, maintaining the image quality required at the time of display, It is possible to reduce the decoding processing cost.

  In the case of an image composed of a natural image and a character string or illustration arranged on the natural image, in order to maintain the image quality of the character or illustration, it may be possible to draw only that portion in real time at the time of display. However, in this case, since it is necessary to prepare a font or provide a shader function on the display device side, it is difficult to realize it with a device that does not have sufficient resources such as a portable terminal. Further, it may be necessary to use a substitute font, and the image intended by the image creator may not be reproduced.

  In this embodiment, natural images that are not easily deteriorated are compressed by JPG at a relatively high compression rate, and character strings and illustrations that are easily deteriorated are compressed by PNG. By individually compressing and synthesizing at the time of display, it is possible to display an image in which image quality deterioration is hardly recognized even with relatively poor resources with high responsiveness. Further, in hierarchical data in which one image is represented by a plurality of resolutions and hierarchized, a hierarchy that is compressed by being separated into a plurality of layers (sub-hierarchies) and a hierarchy that is compressed without being separated are mixed as described above.

  In particular, the image is zoomed in, and the high-resolution layer image used to display large text strings and illustrations is composed of sub-layers, and the low-resolution image that is difficult to recognize deterioration is composed of the combined image. When compression is performed at a compression rate, the change in image quality is small even when the resolution is greatly changed, and the overall data size can be suppressed.

  Furthermore, when a tile image is composed of sub-hierarchies, they are individually compressed and connected to form one tile image data. Thereby, at the time of image display, data constituting the sub-hierarchy can be easily identified and separated, and each can be decoded independently and in parallel. When displaying a high-resolution image, the area that is displayed at one time is local, so by generating compressed data for each tile image and decoding in parallel, only the data necessary for more efficient display is displayed. Can be decoded. As a result, the time required for the decoding process can be shortened and the image quality can be maintained.

  The present invention has been described based on the embodiments. Those skilled in the art will understand that the above-described embodiment is an exemplification, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  For example, in the embodiment, the sub-hierarchy is compressed with different compression formats depending on the properties of the image, but may be compressed with the same compression format. For example, the sub-layer consisting of natural images may have a high compression ratio JPEG, the sub-layer consisting of character strings, etc., the low compression ratio JPEG, etc., and the compression format may be the same, and the compression ratio may be different, and other compression conditions may be different. It may be allowed. When compressing between JPEGs, the composite data may be provided with an area for storing an alpha value in addition to the information shown in FIG.

  Further, in the embodiment, hierarchical data composed of still images is a processing target, but not limited to hierarchical data, one image may be composed of a plurality of sub hierarchies and may be individual compressed data. Alternatively, a moving image composed of a plurality of frame images and character information such as captions superimposed on the moving image may be individually compressed and combined for each frame at the time of display. In these cases as well, the image quality can be maintained and the display processing cost can be reduced by performing compression under conditions suitable for each image.

  DESCRIPTION OF SYMBOLS 1 Image processing system, 10 Image processing apparatus, 12 Display apparatus, 20 Input device, 30 0th hierarchy, 32 1st hierarchy, 34 2nd hierarchy, 35a 1st subhierarchy, 35b 2nd subhierarchy, 36 3rd hierarchy, 37a 1st sub-hierarchy, 37b 2nd sub-hierarchy, 44 display processing unit, 50 hard disk drive, 60 main memory, 70 buffer memory, 90 image data generation device, 91 input information acquisition unit, 92 image creation support unit, 94 tile data Generation unit, 96 composite data generation unit, 98 compressed data storage unit, 100 control unit, 102 input information acquisition unit, 108 load unit, 110 frame coordinate determination unit, 112 decoding unit, 114 display image processing unit, 120 analysis unit, 122a 1st decoding part, 122b 2nd decoding , 126 combining unit 128 drawing unit, 142 a tag portion, 144 first offset portion, 146 a second offset section, 148 first image data unit, 150 second image data unit.

Claims (14)

An image processing apparatus that displays an image by changing a display area in response to an input display area movement request,
Hierarchical data obtained by hierarchizing a plurality of pieces of image data representing images to be displayed at different resolutions in the order of resolution, and a hierarchy constituted by data of a plurality of constituent images that are displayed as superimposed images Image data for storing compressed data obtained by compressing each tile image obtained by dividing an image of each layer of layer data including a layer composed of image data obtained by combining the plurality of component images with a predetermined size A storage unit;
Of the compressed data stored in the image data storage unit, a decoding unit that reads and decodes compressed data of a tile image corresponding to a display area in a hierarchy corresponding to a requested resolution ;
Depending on whether is constituted by data of the requested resolution in the corresponding hierarchy said plurality of component images, after which the decoding unit is switched whether synthesized by superimposing tile image decoding, a display area A display image processing unit for generating an image and displaying it on a display device;
An image processing apparatus comprising: The image data storage unit stores composite data obtained by connecting compressed data of tile images of each component image, which represents the same area on a display image, in association with the area,
The image processing apparatus according to claim 1 , wherein the decoding unit reads composite data corresponding to a display area from the image data storage unit, and separates and decodes each component image. The compressed data stored in the image data storage unit is data compressed in a different compression format for each constituent image,
The decoding unit in accordance with the compression format of each constituent image, the image processing apparatus according to claim 1 or 2, characterized in decoding the respective compressed data. An input information acquisition unit for accepting an designation of a plurality of constituent images to be displayed image by displaying superimposed,
Hierarchical data obtained by hierarchizing a plurality of pieces of image data representing the display image at different resolutions in the order of resolution, and a hierarchy composed of data of the plurality of component images and an image obtained by combining the plurality of component images A tile data generation unit that generates hierarchical data including data and compresses each tile image obtained by dividing an image of each hierarchy by a predetermined size;
Of the compressed data of the data by the configured hierarchy of said plurality of component images represent the same region on the display image, to generate the composite data by connecting the compressed data of tile images of each constituent image rice planting, each compressed data by associating a position on the display image, a composite data generator outputting to the storage device as the image data of the display image,
An image data generation device comprising: The image data generation apparatus according to claim 4 , wherein the tile data generation unit compresses the tile image in a different compression format for each component image. The hierarchical data, the area on the display image, the composite data, the image data generating apparatus according to claim 4 or 5, characterized in that it comprises a hierarchy of the compressed data of tile images of a single image . An image processing apparatus is an image processing method for displaying an image by changing a display area in response to an input display area movement request,
Hierarchical data obtained by hierarchizing a plurality of pieces of image data representing images to be displayed at different resolutions in the order of resolution, and a hierarchy constituted by data of a plurality of constituent images that are displayed as superimposed images A storage device that stores compressed data that is compressed for each tile image obtained by dividing an image of each layer of hierarchical data including a layer composed of image data obtained by combining the plurality of component images with a predetermined size Reading compressed data of the tile image corresponding to the display area in the hierarchy corresponding to the requested resolution ;
Decoding the read compressed data ;
Depending on whether or not the hierarchy corresponding to the requested resolution is composed of the data of the plurality of component images, whether or not to superimpose the decoded tile images is switched, and an image of the display area is generated. Displaying on the display device,
An image processing method comprising: An image data generation device generates image data,
Receiving a designation of a plurality of component images to be displayed by overlapping and displaying;
Read out data of each constituent image from the memory, hierarchical data obtained by hierarchizing a plurality of pieces of image data representing the display image at different resolutions in the order of resolution, and a hierarchy constituted by the data of the plurality of constituent images, Generating hierarchical data including image data obtained by synthesizing the plurality of component images, and compressing each tile image obtained by dividing each hierarchical image by a predetermined size;
Of the compressed data of the data by the configured hierarchy of said plurality of component images represent the same region on the display image, to generate the composite data by connecting the compressed data of tile images of each constituent image rice planting, each compressed data by associating a position on the display image, and outputting to the storage device as the image data of the display image,
An image data generation method comprising: A computer program for causing a computer to realize a function of displaying an image by changing a display area in response to an input display area movement request,
Hierarchical data obtained by hierarchizing a plurality of pieces of image data representing images to be displayed at different resolutions in the order of resolution, and a hierarchy constituted by data of a plurality of constituent images that are displayed as superimposed images A storage device that stores compressed data that is compressed for each tile image obtained by dividing an image of each layer of hierarchical data including a layer composed of image data obtained by combining the plurality of component images with a predetermined size From the hierarchy corresponding to the requested resolution, the function of reading the compressed data of the tile image corresponding to the display area,
A function for decoding the read compressed data ;
Depending on whether or not the hierarchy corresponding to the requested resolution is composed of the data of the plurality of component images, whether or not to superimpose the decoded tile images is switched, and an image of the display area is generated. Function to display on the display device,
A computer program for causing a computer to realize the above. A function of accepting designation of a plurality of component images to be displayed by overlapping and displaying;
Read out data of each constituent image from the memory, hierarchical data obtained by hierarchizing a plurality of pieces of image data representing the display image at different resolutions in the order of resolution, and a hierarchy constituted by the data of the plurality of constituent images, A hierarchy composed of image data obtained by synthesizing the plurality of component images, and a function of compressing each tile image obtained by dividing an image of each hierarchy by a predetermined size;
Of the compressed data of the data by the configured hierarchy of said plurality of component images represent the same region on the display image, to generate the composite data by connecting the compressed data of tile images of each constituent image rice planting, each compressed data by associating a position on the display image, and a function of outputting to the storage device as the image data of the display image,
A computer program for causing a computer to realize the above. A recording medium recording a computer program for causing a computer to realize a function of displaying an image by changing a display area in response to an input display area movement request,
Hierarchical data obtained by hierarchizing a plurality of pieces of image data representing images to be displayed at different resolutions in the order of resolution, and a hierarchy constituted by data of a plurality of constituent images that are displayed as superimposed images A storage device that stores compressed data that is compressed for each tile image obtained by dividing an image of each layer of hierarchical data including a layer composed of image data obtained by combining the plurality of component images with a predetermined size From the hierarchy corresponding to the requested resolution, the function of reading the compressed data of the tile image corresponding to the display area,
A function for decoding the read compressed data ;
Depending on whether or not the hierarchy corresponding to the requested resolution is composed of the data of the plurality of component images, whether or not to superimpose the decoded tile images is switched, and an image of the display area is generated. Function to display on the display device,
The recording medium which recorded the computer program characterized by making a computer implement | achieve. A function of accepting designation of a plurality of component images to be displayed by overlapping and displaying;
Read out data of each constituent image from the memory, hierarchical data obtained by hierarchizing a plurality of pieces of image data representing the display image at different resolutions in the order of resolution, and a hierarchy constituted by the data of the plurality of constituent images, A hierarchy composed of image data obtained by synthesizing the plurality of component images, and a function of compressing each tile image obtained by dividing an image of each hierarchy by a predetermined size;
Of the compressed data of the data by the configured hierarchy of said plurality of component images represent the same region on the display image, to generate the composite data by connecting the compressed data of tile images of each constituent image rice planting, each compressed data by associating a position on the display image, and a function of outputting to the storage device as the image data of the display image,
The recording medium which recorded the computer program characterized by making a computer implement | achieve. A data structure of an image file displayed by changing a display area in response to a display area movement request input in the image processing apparatus,
Compressed data obtained by compressing a plurality of image data representing images to be displayed at different resolutions for each tile image obtained by dividing each image by a predetermined size is associated with the position and resolution on the display image,
Wherein the plurality of image data includes the resolution, and an image composed data by the data of a plurality of constituent images to be displayed image by displaying superimposed, and a data of the synthesized image of the plurality of constituent images ,
In the image processing device, the compressed data of the tile image corresponding to the display area is decoded, and depending on whether or not the image data to be decoded determined by the requested resolution is constituted by the data of the constituent image, terms of whether synthesized by superimposing the configuration image is switched, the image of the display area is generated display data structure of an image file, wherein Rukoto. A recording medium on which an image file having the data structure according to claim 13 is recorded.

Images