JP5061177B2 - Image processing apparatus, image data generation apparatus, image processing method, image data generation method, and data structure of image file - Google Patents

Description

  The present invention relates to an image processing technique for changing a display image as the viewpoint moves.

  Home entertainment systems have been proposed that not only execute game programs, but also play video. In this home entertainment system, the GPU generates a three-dimensional image using polygons (see, for example, Patent Document 1).

  Regardless of the purpose of image display, how efficiently an image is displayed is always an important issue. In particular, various ideas are required to draw a high-definition image at high speed. For example, a technique for efficiently storing texture data and performing mapping efficiently has been proposed (see, for example, Non-Patent Documents 1 and 2). .

US Pat. No. 6,563,999

Sylvain Fefebvre, et.al., Unified Texture Management for Arbitrary Meshes, Repport de recherche, N5210, May 2004, Institut National De Recherche En Informatique Et En Automatique Martin Kraus, et.al., Adaptive Texture Maps, Graphics Hardware (2002), pp1-10, The Eurographics Association

  Even if the display image has a high definition, keeping the data size small and drawing at high speed are always important issues in order to display the image with high responsiveness. In addition, there is a demand for a technique that can express a sense of reality by using a high-definition image to change a display image corresponding to various movements of the viewpoint.

  The present invention has been made in view of such problems, and an object thereof is to provide an image processing technique capable of displaying a high-definition image efficiently and with a sense of reality.

  One embodiment of the present invention relates to an image processing apparatus. The image processing apparatus newly stores a display target image in accordance with a storage unit storing a plurality of image data used for each of a plurality of processes for rendering a display target image and a viewpoint movement request input by a user. A drawing area specifying unit that specifies the frame coordinates of the area to be displayed, and a display image processing unit that draws a display image area corresponding to the frame coordinates using image data stored in the storage unit and outputs the image to the display device The display image processing unit is characterized in that a process to be performed among a plurality of processes using any one of the plurality of image data is made different depending on the height of the viewpoint based on the viewpoint movement request.

  Another embodiment of the present invention relates to an image data generation apparatus. The image data generation device is an image data generation device that generates data of an image obtained by photographing an object, and a measurement image acquisition unit that obtains a plurality of measurement images obtained by photographing the object with different directions of light sources; A color map creation unit that creates a color map that holds color information for each pixel from a measurement image in a predetermined light source direction of the measurement image, and the measurement image is analyzed by an illuminance difference stereo method, A normal map creation unit that creates a normal map that represents a line vector corresponding to a pixel in the image, and scans the normal map in a predetermined direction in a predetermined direction, and in the predetermined direction obtained from the normal vector A height map creation unit that creates a height map that represents the height of the target object corresponding to the pixel by integrating the inclination of the target object, and a color map, normal map, and height map Characterized in that it comprises a storage unit for storing the image data.

  Yet another embodiment of the present invention relates to an image processing method. According to the viewpoint movement request input by the user, the image processing method specifies a frame coordinate of a region to be newly displayed in the display target image, and renders the display target image stored in the storage device. Reading at least one of a plurality of image data used for each of a plurality of processes for rendering a display image region corresponding to the frame coordinates using the read image data, and displaying the rendered region data on a display device And the step of drawing is characterized in that the processing to be performed among the plurality of processes using any one of the plurality of image data is made different depending on the height of the viewpoint based on the viewpoint movement request. .

  Still another embodiment of the present invention relates to an image data generation method. This image data generation method is an image data generation method for generating data of an image obtained by photographing an object, and a step of acquiring a plurality of measurement images obtained by photographing an object with different directions of a light source and storing them in a memory And a step of creating a color map that holds color information for each pixel using a measurement image in the direction of a predetermined light source out of the measurement image read from the memory, and the measurement image read from the memory Analyzing with the stereo method, creating a normal map that represents the normal vector of the object corresponding to the pixels in the image, and scanning the normal map in the specified direction to obtain it from the normal vector A step of creating a height map representing the height of the object corresponding to the pixel by integrating the inclination of the object in the predetermined direction, a color map, And storing normal map, and the height map in the storage device as the image data, characterized in that it comprises a.

  Still another embodiment of the present invention relates to a data structure of an image file. This data structure is a data structure of an image file read from a storage device to display at least a part of an image on a display, and holds color information used for expressing the color of a display image for each pixel. A color map, a height map used for expressing parallax with respect to an object in the image and holding height information for the image plane of the object for each pixel, and irregularities on the surface of the object in the image The normal map that holds the normal vector information of the surface of the target object for each pixel and the shadow of the target object for each pixel that is used to add a shadow corresponding to the set light source direction. And a horizon map that holds information on the direction of the light source for each pixel.

  It should be noted that any combination of the above-described constituent elements and a representation of the present invention converted between a method, an apparatus, a system, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, realistic image display can be performed efficiently.

It is a figure which shows the use environment of the image processing system of this Embodiment. It is a figure which shows the external appearance structure of the input device applicable to the image processing system of FIG. It is a figure which shows the structure of the image processing apparatus in this Embodiment. It is a conceptual diagram of the hierarchical structure of the image data used in this Embodiment. It is a figure which shows in detail the structure of the control part which has a function to display the image in this Embodiment. It is a figure which shows the more detailed structure of the display image process part and buffer memory in this Embodiment. It is a figure for demonstrating the principle of parallax mapping. It is a figure for demonstrating the principle of horizon mapping. In this Embodiment, it is a figure for demonstrating the example of the aspect which changes the process which a display image process part performs with the height of a viewpoint. In this Embodiment, it is a conceptual diagram at the time of changing the data structure of a color map, a height map, a normal map, and a horizon map. It is a flowchart which shows the process sequence of the image display in this Embodiment. It is a figure which shows the structure of the image data generation apparatus which produces | generates the image data in this Embodiment. It is a figure for demonstrating the acquisition aspect of a measurement image at the time of using a measurement image acquisition part as a camera in this Embodiment. It is a figure for demonstrating the principle in which a height map preparation part produces a height map using a normal map in this Embodiment. It is a figure for demonstrating the method which scans diagonally with respect to a normal map in this Embodiment, and produces a height map. It is a flowchart which shows the procedure which produces the hierarchical data of a color map, a normal map, a height map, and a horizon map in this Embodiment. It is a flowchart which shows another procedure which produces the hierarchical data of a color map, a normal map, a height map, and a horizon map in this Embodiment.

  In the present embodiment, an image of an uneven object is displayed with a fine and realistic feeling. Taking oil painting as an example, the touch of a brush, including how to put a paint on it, varies depending on the author, which is one of the expression methods. At exhibitions etc., viewers can enjoy the work by bringing their faces closer to the painting and seeing such a touch, or moving away to see the entire composition. The unevenness on the surface of the object is not only an oil painting but an important element for expressing the texture of the object. When the object is reproduced on the screen, such unevenness is hard to appear only by increasing the number of pixels and making the color information detailed. Therefore, in the present embodiment, it is possible to display an image with a sense of presence by expressing shading due to surface irregularities, fine specular reflection, hiding the line of sight, and the like.

  FIG. 1 shows a use environment of an image processing system 1 according to the present embodiment. The image processing system 1 includes an image processing device 10 that executes an application program including image processing, and a display device 12 that outputs a processing result by the image processing device 10. The display device 12 may be a television having a display that outputs an image and a speaker that outputs sound.

  The display device 12 may be connected to the image processing device 10 by a wired cable, or may be wirelessly connected by a wireless local area network (LAN) or the like. In the image processing system 1, the image processing apparatus 10 may be connected to an external network such as the Internet via the cable 14 to download and acquire content including compressed image data. The image processing apparatus 10 may be connected to an external network by wireless communication.

  In response to a viewpoint movement request from the user, the image processing apparatus 10 performs processing for changing the display area, such as enlargement / reduction processing of an image displayed on the display of the display device 12, movement processing in the vertical and horizontal directions, and the like. When the user operates the input device while viewing the image displayed on the display, the input device transmits a viewpoint movement request signal to the image processing device 10.

  FIG. 2 shows an external configuration of the input device 20. The input device 20 includes a cross key 21, analog sticks 27a and 27b, and four types of operation buttons 26 as operation means that can be operated by the user. The four types of operation buttons 26 include a circle button 22, a x button 23, a square button 24, and a triangle button 25.

  In the image processing system 1, the operation means of the input device 20 receives a request for changing the height of the viewpoint, that is, a request for enlarging / reducing the display image, and a request for moving the viewpoint on the horizontal plane, that is, a request for scrolling in the vertical and horizontal directions. A function for input is assigned. For example, the input function of the display image enlargement / reduction request is assigned to the right analog stick 27b. The user can input a display image reduction request by pulling the analog stick 27b forward, and can input a display image enlargement request by pressing the analog stick 27b from the front.

  Also, the input function of the scroll request in the vertical and horizontal directions is assigned to the cross key 21. The user can input a movement request in the direction in which the cross key 21 is pressed by pressing the cross key 21. Note that the viewpoint movement request input function may be assigned to another operation means. For example, the scroll request input function may be assigned to the analog stick 27a.

  FIG. 3 shows the configuration of the image processing apparatus 10. The image processing apparatus 10 includes a wireless interface 40, a switch 42, a display processing unit 44, a hard disk drive 50, a recording medium mounting unit 52, a disk drive 54, a main memory 60, a buffer memory 70, and a control unit 100. . The display processing unit 44 has a frame memory that buffers data to be displayed on the display of the display device 12.

  The switch 42 is an Ethernet switch (Ethernet is a registered trademark), and is a device that transmits and receives data by connecting to an external device in a wired or wireless manner. The switch 42 is configured to be connected to an external network via the cable 14 and receive image data and the like from the server. The switch 42 is connected to the wireless interface 40, and the wireless interface 40 is connected to the input device 20 using a predetermined wireless communication protocol. A signal input from the user in the input device 20 is supplied to the control unit 100 via the wireless interface 40 and the switch 42.

  The hard disk drive 50 functions as a storage device that stores data. Image data received via the switch 42 is stored in the hard disk drive 50. When a removable recording medium such as a memory card is mounted, the recording medium mounting unit 52 reads data from the removable recording medium. When a read-only ROM disk is loaded, the disk drive 54 drives and recognizes the ROM disk to read data. The ROM disk may be an optical disk or a magneto-optical disk. Image data may be stored in these recording media.

  The control unit 100 includes a multi-core CPU, and includes one general-purpose processor core and a plurality of simple processor cores in one CPU. The general-purpose processor core is called PPU (PowerPC Processor Unit), and the remaining processor cores are called SPU (Synergistic Processor Unit).

  The control unit 100 includes a memory controller connected to the main memory 60 and the buffer memory 70. The PPU has a register, has a main processor as an operation execution subject, and efficiently assigns a task as a basic processing unit in an application to be executed to each SPU. Note that the PPU itself may execute the task. The SPU has a register, and includes a sub-processor as an operation execution subject and a local memory as a local storage area. The local memory may be used as the buffer memory 70.

  The main memory 60 and the buffer memory 70 are storage devices and are configured as a RAM (Random Access Memory). The SPU has a dedicated DMA (Direct Memory Access) controller as a control unit, can transfer data between the main memory 60 and the buffer memory 70 at high speed, and the frame memory and the buffer memory 70 in the display processing unit 44 can be transferred. High-speed data transfer can be realized. The control unit 100 according to the present embodiment realizes a high-speed image processing function by operating a plurality of SPUs in parallel. The display processing unit 44 is connected to the display device 12 and outputs an image processing result according to a request from the user.

  The image processing apparatus 10 according to the present embodiment uses a main part of the compressed image data from the hard disk drive 50 in order to smoothly change the display image when the display image enlargement / reduction process or the display area movement process is performed. It is loaded into the memory 60. Further, a part of the compressed image data loaded in the main memory 60 is decoded and stored in the buffer memory 70. This makes it possible to instantaneously switch an image used for generating a display image at a necessary later timing.

  In this embodiment, the data structure of the image to be displayed is not particularly limited. However, here, in order to display a high-definition image more efficiently, a form using hierarchical image data having a hierarchical structure is used. explain. Hierarchical image data is image data composed of images of different resolutions generated by reducing an original image in a plurality of stages. Each layer image is divided into one or a plurality of tile images. For example, the image with the lowest resolution is composed of one tile image, and the original image with the highest resolution is composed of the largest number of tile images. At the time of image display, the tile image used for drawing is switched to a tile image of a different hierarchy when the display image has a predetermined resolution, so that enlarged display or reduced display is quickly performed.

  FIG. 4 shows a conceptual diagram of a hierarchical structure of image data used in the present embodiment. The image data has a hierarchical structure including a 0th hierarchy 30, a first hierarchy 32, a second hierarchy 34, and a third hierarchy 36 in the depth (Z-axis) direction. Although only four layers are shown in the figure, the number of layers is not limited to this. Hereinafter, image data having such a hierarchical structure is referred to as “hierarchical data”. However, the hierarchical data in the figure is conceptual, and actually, an image is expressed by a set of a plurality of data as described later.

  The hierarchical data shown in FIG. 4 has a quadtree hierarchical structure, and each hierarchy is composed of one or more tile images 38. All the tile images 38 are formed in the same size having the same number of pixels, and have, for example, 256 × 256 pixels. The image data of each layer expresses one image at different resolutions, and the original image of the third layer 36 having the highest resolution is reduced in a plurality of stages to obtain the second layer 34, the first layer 32, the 0th layer. Image data of the hierarchy 30 is generated. For example, the resolution of the Nth layer (N is an integer greater than or equal to 0) may be ½ of the resolution of the (N + 1) th layer in both the left and right (X axis) direction and the up and down (Y axis) direction.

  In the image processing apparatus 10, the hierarchical data is held in a storage device such as the hard disk drive 50 in a state compressed in a predetermined compression format, and is read from the storage device and decoded before being displayed on the display. . The image processing apparatus 10 according to the present embodiment has a decoding function corresponding to a plurality of types of compression formats, and can decode, for example, compressed data in the S3TC format, JPEG format, and JPEG2000 format.

  As shown in FIG. 4, the hierarchical structure of the hierarchical data is set with the horizontal direction as the X axis, the vertical direction as the Y axis, and the depth direction as the Z axis, thereby constructing a virtual three-dimensional space. When the image processing device 10 derives the change amount of the display image from the viewpoint movement request signal supplied from the input device 20, the image processing device 10 derives the coordinates (frame coordinates) of the four corners of the frame in the virtual space using the change amount. The frame coordinates in the virtual space are used for loading compressed data into the main memory, which will be described later, and drawing processing of a display image. Instead of the frame coordinates in the virtual space, the image processing apparatus 10 may derive information for specifying the hierarchy and texture coordinates (UV coordinates) in the hierarchy. Hereinafter, the combination of the hierarchy specifying information and the texture coordinates is also referred to as frame coordinates.

  Regardless of whether the data structure is a hierarchical structure, the image is basically displayed using a color map that holds color information for each pixel. In this embodiment, in addition to the color map, uneven information for expressing the unevenness of the object is introduced, so that a fine structure such as a touch on the surface of the painting is expressed more realistically. Therefore, in the present embodiment, the color map and the unevenness information are combined into image data.

  FIG. 5 shows in detail the configuration of the control unit 100 having a function of displaying an image in the present embodiment. The control unit 100 includes an input information acquisition unit 102 that acquires information input by the user from the input device 20, a drawing region specifying unit 110 that specifies a region to be newly displayed, and a load block that determines image data to be newly loaded. The determination unit 106 includes a load unit 108 that loads necessary compressed image data from the hard disk drive 50. The control unit 100 further includes a decoding unit 112 that decodes the compressed image data and a display image processing unit 114 that draws a display image.

  In FIG. 5, FIG. 6 and FIG. 12, which will be described later, each element described as a functional block for performing various processes may be configured by a CPU (Central Processing Unit), a memory, and other LSIs in terms of hardware. In terms of software, it is realized by a program loaded in a memory. As described above, the control unit 100 includes one PPU and a plurality of SPUs, and each functional block can be configured by the PPU and the SPU individually or in cooperation. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  The input information acquisition unit 102 acquires instruction contents such as start / end of image display, movement of the viewpoint, and the like input by the user to the input device 20. The drawing area specifying unit 110 specifies the frame coordinates of the area to be newly displayed according to the frame coordinates of the current display area and the viewpoint movement request information input by the user. If the tile image data including the area is already loaded in the main memory 60, the information is supplied to the decoding unit 112, and if not loaded, the information is supplied to the load block determining unit 106. The drawing area specifying unit 110 may specify not only the area to be displayed at that time but also an area predicted to be necessary thereafter. The drawing area specifying unit 110 also supplies the display processing unit 44 with the position of the new viewpoint corresponding to the viewpoint movement request input by the user and the specified frame coordinates.

  Based on the information from the drawing area specifying unit 110, the load block determining unit 106 specifies tile image data to be newly loaded from the hard disk drive 50 to the main memory 60, and issues a load request to the load unit 108. The load block determination unit 106 may make a load request at a predetermined timing in a state where the load unit 108 is not performing the load process, for example, at a predetermined time interval or when the user makes a viewpoint movement request. The load unit 108 loads compressed image data from the hard disk drive 50 to the main memory 60 in accordance with a request from the load block determination unit 106.

  The decoding unit 112 searches the buffer memory 70 based on the frame coordinates of the area to be newly acquired acquired from the drawing area specifying unit 110. If the area of the corresponding frame coordinate is not stored in the buffer memory 70, the main memory The tile image data is read from 60 and decoded, and the decoded data is stored in the buffer memory 70. The display image processing unit 114 reads the corresponding image data from the buffer memory 70 based on the frame coordinates of the area to be newly displayed, and draws the display image in the frame memory of the display processing unit 44. At this time, the change in appearance due to the unevenness of the object, which changes with the movement of the viewpoint, is expressed using the unevenness information.

  FIG. 6 shows a more detailed configuration of the display image processing unit 114 and the buffer memory 70. As described above, in this embodiment, the unevenness information is included in the image data in addition to the color map that holds the color information for each pixel. For this reason, the load unit 108 and the decode unit 112 load or decode not only the color map but also the unevenness information of the area necessary for display, and each decoded data is stored in the buffer memory 70. As the unevenness information, a height map 132, a normal map 136, and a horizon map 138 are used as shown in FIG. The height map 132, normal map 136, and horizon map 138 all hold data for each pixel.

  The display image processing unit 114 uses the height map 132 to calculate a coordinate shift due to the parallax, the parallax expression unit 124, the color map 134, and the normal map 136 to express the color of the object in consideration of unevenness. It includes an expression unit 126, a shadow expression unit 128 that shadows an image using the horizon map 138, and an image output unit 130 that draws a final display image in a frame memory.

  The parallax expression unit 124 uses the height map 132 that holds the height information of the object for each pixel, and the difference in appearance due to the movement of the viewpoint, which occurs due to the height of the object, is regarded as a coordinate shift. Express. For the specific calculation, a technique generally proposed as parallax mapping can be introduced (for example, Natalya Tatarchuk, Dynamic parallax occlusion mapping with approximate soft shadows, Symposium on Interactive 3D Graphics, Proceedings of the 2006 symposium on Interactive 3D graphics and games, P. 63-69, 2006).

  FIG. 7 is a diagram for explaining the principle of parallax mapping. When the shape 210 of the object is uneven with respect to the image plane 218 as shown in the drawing, the point 214 on the surface of the object on the line of sight 212 appears on the point 216 on the image plane. Accordingly, the parallax expression unit 124 creates new coordinates for converting the position of the pixel so that the color information of the pixel at the point 214 is shifted to the position of the point 216. For this reason, the parallax expression unit 124 uses a height map representing the shape 210 and linearly calculates the points 214 and 216 where the line of sight 212 from the new viewpoint acquired from the drawing area specifying unit 110 intersects the shape 210 of the target object and the image plane. Find by searching. A pixel shift when the original image is represented as a display image is represented by a texture coordinate value or the like.

  The color expression unit 126 performs rendering by shifting the pixels of the color map 134 using the texture coordinate values derived by the parallax expression unit 124. At this time, the normal map 136 that holds the normal of the surface of the object for each pixel is used to change the light hit condition to express the unevenness. The color expression unit 126 may perform the same process by bump mapping generally used in computer graphics.

  The shadow expression unit 128 adds a shadow to the image drawn by the color expression unit 126 using a horizon map 138 that holds information on whether or not a shadow is generated depending on the angle of the light source for each pixel. This technique can introduce a technique generally proposed as horizon mapping (see, for example, Peter-Pike J. Sloan et al., Interactive Horizon Mapping, Eurographics Rendering Workshop 2000, June, 2000).

  FIG. 8 is a diagram for explaining the principle of horizon mapping. If the shape 220 of the object is uneven as shown in the figure, the light from the light source 222 is not obstructed by the object, so that no shadow is generated at the point 230, but the light from the light source 224 is the object. Shadows occur because of obstruction. That is, regarding the point 230, it is determined whether or not a shadow can be formed at the angle of the light source 226. Therefore, data representing binary values as to whether or not a shadow is produced when the angle θ of the light source with respect to the normal of the image plane at each pixel such as the point 230 is changed in multiple steps is prepared as a horizon map.

  Then, the shadow expression unit 128 compares the position of the light source set in advance or the position of the light source determined by changing the viewpoint according to a preset rule by moving the viewpoint, etc., and the horizon map to determine the necessary pixels. Add shadows as appropriate. The image output unit 130 outputs the image data drawn as described above to the frame memory of the display processing unit 44.

  Here, when the color map 134 is hierarchical data as described above, the height map 132, the normal map 136, and the horizon map 138 may also be hierarchical data. In this case, similar to the color map 134, the other maps are also divided into tile images, and tile image data in an area necessary for image display is loaded and decoded as appropriate. If each map has a hierarchical structure with the same resolution, even if the resolution of the display image is changed significantly, the display image can be drawn by the same process by switching all the layers of the data to be used. A quick image can be displayed quickly.

  When the color map 134 is hierarchical data, the range of resolution that can be displayed with high responsiveness is widened as described above, compared to the case where image data of one resolution is used. In order to cope with such a change in resolution, the appearance impression of the object changes depending on whether the object is actually viewed from a distance or close. In the present embodiment, in order to express such a change in impression in the display image, the processing using the above-described unevenness information may be changed depending on the height of the viewpoint, that is, the resolution.

  FIG. 9 is a diagram for explaining an example of a mode in which the processing performed by the display image processing unit 114 is changed depending on the height of the viewpoint. In the figure, the horizontal axis is the height of the viewpoint, and the closer to the image plane, the closer to the right. Each chart represented for such a change in viewpoint is processed from the top by the parallax expression unit 124 using the height map 132, the process performed by the color expression unit 126 using the color map 134, and the normal map 136. The operation performed by using the horizon map 138 and the operation performed by the shadow expression unit 128 using the horizon map 138 are shown.

  First, between Hmax at which the height of the viewpoint is maximum and a predetermined threshold value H1, unevenness and shadow are hardly recognized as an impression when sufficiently away from the object, and parallax is not easily generated. Only 126 operates, and an image is drawn only with a color map without performing bump mapping. When the line-of-sight height is from H1 to the next threshold value H2, the color expression unit 126 draws an image using the color map 134 and the normal map 136 in order to express unevenness. At this time, the parallax expression unit 124 and the shadow expression unit 128 do not operate.

  When the line-of-sight height is from H2 to the next threshold value H3, the shadow expression unit 128 adds a shadow using the horizon map 138 in addition to the color expression unit 126 in a state where the shadow of the unevenness can be recognized. Since the influence of the parallax increases between the viewpoint height H3 and the minimum Hmin, the parallax representation unit 124 uses the height map 132 in addition to the color representation unit 126 and the shadow representation unit 128. Express the gap.

  In this way, by changing the processing content for image drawing according to the height of the viewpoint, it is possible to express changes in the impression received by people due to changes in the distance to the object in the real world, and only perform necessary and sufficient processing Thus, the processing load can be reduced. The threshold of the viewpoint height at which the process is switched may be the same as the viewpoint height when the hierarchy used for display is switched. Further, the threshold value may be changed according to the ratio between the maximum height of the object obtained from the height map and the height of the viewpoint. In addition, when applying a shadow using a horizon map, the direction of the light source may be changed according to the height of the viewpoint. Moreover, you may switch a point light source and a surface light source. Such a change in the light source also expands the range of possible expressions.

  When the processing performed by the display image processing unit 114 increases as the viewpoint becomes lower in this way, as shown in the lowermost stage of FIG. 9, adjustment is performed so that the moving speed of the display area decreases as the viewpoint decreases. You may make it absorb the time which requires. Specifically, the peak value of a transfer function such as a Gaussian function is set lower as the viewpoint becomes lower, and a display image corresponding to the signal is input from the input of the signal by performing a convolution operation with the viewpoint movement request signal input by the user. What is necessary is just to adjust time until is displayed.

  Further, even when the color map 134, the height map 132, the normal map 136, and the horizon map 138 are set as hierarchical data, when the map to be used is changed depending on the height of the viewpoint as described above, the number of hierarchies is set for each map. It may be different. FIG. 10 is a conceptual diagram when the data structures of the color map 134, the height map 132, the normal map 136, and the horizon map 138 are made different. In the figure, the horizontal axis is the height of the viewpoint, and the closer to the image plane, the closer to the right. The hierarchical structure of each map is shown in the same manner as shown in FIG.

  As shown in the figure, the color map 134 is necessary at all viewpoint heights, that is, at all resolutions, and therefore has the largest number of layers. Then, for example, when setting to perform bump mapping using the normal map 136 from the height of the viewpoint when displaying an image using the first hierarchy next to the 0th hierarchy which is the top hierarchy of the color map 134, The normal map 136 may be prepared from a layer corresponding to the first layer of the color map 134.

  If the horizon mapping using the horizon map 138 is set from the height of the viewpoint when the image is displayed using the second hierarchy of the color map 134, the horizon map 138 is displayed on the second hierarchy of the color map 134. Prepare from the corresponding hierarchy. Similarly, when disparity mapping is performed from the viewpoint height when an image is displayed using the third layer of the color map 134, the height map 132 is prepared from the layer corresponding to the third layer of the color map 134. I will do better.

  By preparing the data in this way, the data size of the entire image data can be reduced. Note that the change in the combination of processing due to the change in the height of the viewpoint shown in FIGS. 9 and 10 is merely an example, and may be set as appropriate depending on the nature of the object in the image, the expression intention of the image data creator, and the like. . Further, the number of layers of various maps may be determined independently of the change in processing depending on the height of the viewpoint, depending on the nature of the image to be displayed.

  In a state where the image processing apparatus 10 is connected to the network, a part of the map or a part of the hierarchy may be charged, and only the user who paid the fee may download. For example, if only the color map 134 is provided at a low price, and the user who displays the image using the color map 134 enlarges the image and has a resolution to which an additional map can be applied, confirm whether or not to purchase the map. Display a message to do. Allow users to download additional maps once they are willing to purchase.

  Then, the image processing apparatus 10 performs corresponding processing using the newly downloaded map. Thereby, the user can see the high-definition image expressed to the unevenness for the first time. In addition, the color map 134 may initially provide only a low-resolution layer and charge for downloading a high-resolution layer necessary for enlargement.

  Next, the operation of the image processing apparatus 10 configured as described above will be described. FIG. 11 is a flowchart showing the image display processing procedure. First, when the user gives an instruction to start image display, image data necessary for displaying a predetermined initial image such as an entire image to be displayed is read from the hard disk drive 50, decoded, and then displayed. Create and display (S10). The initial image is continuously displayed until the user inputs a viewpoint movement request to the input device 20 (N in S12, S10). When a display area movement request is made (Y in S12), the drawing area specifying unit 110 of the control unit 100 derives frame coordinates and a new viewpoint of the display area to be newly displayed according to the request (S14).

  Data necessary for drawing a new image is loaded from the hard disk drive 50 by the load unit 108 as necessary, decoded by the decoding unit 112, and stored in the buffer memory 70 (S16). Then, the parallax expression unit 124 of the display image processing unit 114 reads the height map 132 of the buffer memory 70 and calculates the pixel shift due to the parallax by performing parallax mapping (S18). Next, the color expression unit 126 reads the color map 134 and the normal map 136 in the buffer memory 70, and draws an image while expressing unevenness (S20).

  Next, the shadow representation unit 128 reads the horizon map 138 in the buffer memory 70, and adds a shadow to the image by performing horizon mapping (S22). As described above, any one of the processes of S18, S20, and S22 may be selectively performed depending on the height of the viewpoint or other settings. In the bump mapping of S20, the color map 134 may be mapped as it is with all the normals being vertical. The display image is updated by drawing the image thus drawn in the frame memory of the display processing unit 44 and displaying it on the display device 12 (S24).

  Next, a method for creating image data used in this embodiment will be described. FIG. 12 shows a configuration of an image data generation apparatus that generates image data in the present embodiment. The image data generation device 300 includes a measurement image acquisition unit 302 that acquires a measurement image obtained by capturing an object, a normal map creation unit 304 that acquires a normal map from the measurement image and, in some cases, a height map, and a height from the normal map. A height map creation unit 306 that creates a map, a horizon map creation unit 308 that creates a horizon map from the height map, a color map creation unit 310 that creates a color map from the measurement image, and the data of each created map are compressed A storage device 312 is included.

  The measurement image acquisition unit 302 acquires, as measurement image data, data of three or more images captured by fixing the camera directly above the object and changing three or more light source directions. The measurement image data stored in the storage device 312 in advance may be read out, or may be acquired by taking a measurement image acquisition unit itself as a camera. The direction of the light source is, for example, 8 directions surrounding the center of the object and 9 directions in the same direction as the camera. As a result, nine measurement images are acquired.

  The normal map creation unit 304 acquires the normal vector of each pixel using the measurement image. The normal vector is obtained by using a photometric stereo method. Specifically, when the normal vector is n, the surface reflectance is ρ, the direction vector of the i-th light source is Li, and the brightness photographed by the camera with respect to the light source is Ii, Is solved for n.

  If the surface reflectance ρ is known, three equations are sufficient to obtain the x, y, and z coordinates that are elements of the normal vector n. Assuming that the measurement image is nine images acquired by light sources in nine directions, among those images, the specularly reflected image and the shadowed image are detected from the pixel color information and excluded. The normal vector can be derived with high accuracy by establishing the above equation.

  The height map creation unit 306 obtains the inclination of the object in a predetermined direction from the normal vector of the normal map created by the normal map creation unit 304, and adds the displacement of the height obtained therefrom to add pixel height. Get height information for each column and create a height map. Details will be described later.

  The horizon map creation unit 308 creates a horizon map using the height map created by the height map creation unit. As described above, the horizon map is data indicating whether or not a shadow is produced by the inclination of the light source. Therefore, if the height information of the object is obtained from the height map, the direction of the light source is virtually set to a predetermined angle. The data is obtained by shaking.

  For example, in the RGBA color model, four types of data of red, green, blue, and alpha values are held for each pixel, and generally 256 gradation values can be set for each. If this data structure is used as it is, it is possible to hold for each pixel whether or not it becomes a shadow when the angle from the normal of the image plane is adjusted to 256 levels with respect to the four directions on the image plane of the light source. . If two such data are prepared, a horizon map with light sources in a total of eight directions can be created.

  The color map creation unit 310 obtains color information of each pixel from the measurement image when the light source is installed in the same direction as the camera, for example, and creates a color map. The maps created by the normal map creation unit 304, the height map creation unit 306, the horizon map creation unit 308, and the color map creation unit 310 are compressed and stored in the storage device 312. Further, as will be described later, the normal map creation unit 304, the height map creation unit 306, the horizon map creation unit 308, and the color map creation unit 310 may create hierarchical data of each map. In this case, as shown in FIG. 4, the data of each layer may be compressed for each tile image divided into regions of a predetermined size.

  FIG. 13 is a diagram for explaining a measurement image acquisition mode when the measurement image acquisition unit 302 is a camera. The camera 316 is fixed at a position directly above the object 314 such as a painting. Then, the light source 318 is arranged on the circumference at a predetermined height with the center of the object as the center. In the figure, eight light sources are shown on the circumference. A light source 319 is also arranged at the same position as the camera. Then, the object 314 is photographed by the camera 316 while turning on the light sources one by one.

  In this case, a ring-type strobe device including a plurality of light sources 318 may be prepared, and each light source may be turned on sequentially. Alternatively, a strobe device provided with one light source 318 that can rotate on the circumference is prepared, and rotation on the circumference in accordance with the shooting timing of the camera 316 enables shooting with light sources in multiple directions. Good.

  FIG. 14 is a diagram for explaining the principle that the height map creating unit 306 creates a height map using a normal map. In the figure, the horizontal direction represents the image plane, and the object has a shape 320. Assuming that a region divided by vertical broken lines is one pixel, the normal map holds component information of normal vectors represented by arrows for each pixel in the figure.

  In such a situation, the height map creation unit 306 scans the normal map in a predetermined direction, and calculates the inclination of the object for each pixel in the direction from each normal vector. Then, the height at the boundary between the target pixel and the next pixel is added to the height of the target at the boundary with the previous pixel on the scanning line by adding the height displacement obtained from the inclination of the target pixel. calculate.

  In the case of FIG. 14, when scanning from left to right in the figure, that is, pixels A, B, C, D,..., The height H1 of the object at the boundary between pixel A and the pixel adjacent to the left is A height H2 obtained by adding a height displacement Δ1 obtained from the inclination of the pixel A is a height at the boundary between the pixel A and the pixel B. Next, the height H3 obtained by adding the height displacement Δ2 obtained from the inclination of the pixel B to the height H2 is the height at the boundary between the pixel B and the pixel C. A height H4 obtained by adding a displacement D3 of the height of the pixel C to the height H3 is a height at the boundary between the pixel C and the pixel D.

  By repeating this process from the left end to the right end of the image, pixel height information for one line can be obtained. The left end height is given an initial value such as 0.5. By performing such processing for each row on all the rows in the image, height information of all the pixels constituting the image, that is, a height map can be obtained. Since the processing for each row is independent, the time required for processing can be shortened by performing the processing in parallel.

  With the process shown in FIG. 14, a height map can be obtained with high accuracy for an object having a continuous gradient. On the other hand, when there is a discontinuous step in the object, the normal vector may not reflect the difference in height although the height differs between adjacent pixels. In order to maintain accuracy in such a case, the height map creation unit 306 further acquires similar height information in different directions with respect to the same normal map. For example, in addition to the above-described left end to right end direction, scanning is performed in four directions from the upper end to the lower end, from the right end to the left end, and from the lower end to the upper end, thereby creating four height maps. A value obtained by averaging the heights held by the four height maps for each pixel is used as the final height map height information.

  Even in oil paintings, there are cases where steep steps are formed due to the rising of the paint, etc., but it is rare that the steps are steep even if scanned from any direction. A gradient exists in either direction. Since an accurate height can be derived if the gradient is continuous, a height map with high accuracy can be generated by averaging the height maps generated as a result of scanning in a plurality of directions as described above.

  The scanning direction may be diagonal as well as vertical and horizontal. FIG. 15 is a diagram for explaining a method of creating a height map by scanning obliquely with respect to the normal map. In the example shown in the figure, as indicated by the arrows for the image 322, scanning is performed in the direction of 45 ° from the vertical and horizontal directions of the image, from the upper left to the lower right, from the upper right to the lower left, from the lower left to the upper right, and from the lower right to the upper left. To do. Although only four arrows are shown, height information is acquired for all rows in the same direction in this case as well, and a total of four height maps are created. As described above, the average value in each pixel is used as final height map data.

  In the figure, a pixel block 326 obtained by enlarging 2 × 2 pixels of pixels E and H crossed by a certain scanning line 324 and pixels F and G adjacent to the scanning line 324 when obliquely scanned in this manner is shown. Yes. When obtaining the height of the object in the pixel H in the pixel block 326, the inclination of the direction from E to H is obtained using the normal lines of the pixels F, G, and H, and the displacement of the height in the direction is determined as the pixel. Add to the height at the boundary between E and pixel H. Thus, when creating a height map by scanning from the upper left to the lower right, initial values are given to the pixels for the leftmost column and the pixels for the uppermost row. A height map obtained by scanning in each direction can be obtained in the same manner for other directions.

  It should be noted that vertical and horizontal scanning and oblique scanning as shown in FIG. 15 are performed to create eight height maps, and the average value of all of them may be used as final height map data. The method of calculating final height map data from a plurality of height maps may not be simply an average value, and various statistical methods may be applied. For example, after calculating a height deviation in each height map for each pixel and excluding a height value that exceeds a predetermined threshold value, an average value may be obtained or applied to a neural network.

  Next, a data creation method when using a normal map, height map, and horizon map as hierarchical data will be described. As described above, the color map is hierarchized by creating a high-definition image as the lowest layer and sequentially reducing it. On the other hand, even if the lowest layer data of the normal map and the horizon map is created using the measurement image of the high-definition image, the data held in these maps is basically an independent value for each pixel, so it is reduced. Difficult to do. Therefore, it is not easy to create data of a hierarchy other than the lowest layer in the hierarchy data.

  Therefore, in the present embodiment, first, a normal map having the same resolution is created from a high-resolution measurement image corresponding to the lowermost layer of the color map, and once the height map hierarchical data is created, the normal map hierarchical data is created. Return to processing. FIG. 16 is a flowchart showing a procedure for creating hierarchical data of a color map, normal map, height map, and horizon map.

  First, the measurement image acquisition unit 302 acquires a measurement image at the lowest resolution of the hierarchical data, that is, the highest resolution (S30). Then, the color map creation unit 310 obtains an image in which the light source is in the same direction as the camera from the measurement image, and creates the color map hierarchical data by reducing the image to a predetermined resolution (S32). .

  Next, the normal map creation unit 304 creates a normal map having a resolution corresponding to the lowest layer image of the color map using the measurement image (S34). This is the lowest layer data in the hierarchy data of the normal map. Next, the height map creating unit 306 creates the bottom layer height map by scanning the normal map of the bottom layer obtained in S34 in a predetermined direction and averaging it (S36). Subsequently, the height map creation unit 306 sequentially reduces the lowermost height map by a method such as a bilinear method that is generally used to reduce an image, and generates hierarchical data corresponding to the color map hierarchical data. (S38).

  The normal map creation unit 304 generates each hierarchy of the normal map by differentiating the height map of each hierarchy except the lowest layer among the height map hierarchy data created in this way, (S40). Specifically, the height map of each layer is scanned in two vertical and horizontal directions, for example, and the height map is differentiated to obtain the tilt information of the object in each direction. Then, a final normal vector is obtained by synthesizing the normal vector for each scanning direction obtained from the inclination for each pixel, and a normal map is generated. On the other hand, the horizon map creation unit 308 generates each level of the horizon map from each level of the height map and sets it as hierarchical data (S42). Each created map is divided into tile images, compressed, and stored in the storage device 312 (S44).

  FIG. 17 is a flowchart showing another procedure for creating hierarchical data of a color map, normal map, height map, and horizon map. In this example, the measurement image is first hierarchized, and each hierarchy in the color map and normal map is directly generated from the corresponding hierarchy of the measurement image. First, the measurement image acquisition unit 302 acquires a measurement image at the lowest resolution of the hierarchical data, that is, the highest resolution (S50), and further reduces the image sequentially to a predetermined size to generate a plurality of images. (S52).

  Next, the color map creation unit 310 generates each layer of the color map directly from each layer of the layer data of the measurement image in which the light source is in the same direction as the camera, and generates the layer data (S54). The normal map creation unit 304 collects measurement images of the same hierarchy and solves the equations for each hierarchy, thereby generating a normal map of the corresponding hierarchy and creating it as hierarchical data (S56). The height map creation unit 306 creates a height map of the corresponding hierarchy by scanning each hierarchy in the hierarchy data of the normal map, and sets it as hierarchy data (S58). The horizon map creation unit 308 creates a horizon map of the corresponding hierarchy from each hierarchy in the hierarchy data of the height map and uses it as hierarchy data (S60). Each created map is divided into tile images, compressed, and stored in the storage device 312 (S62).

  Note that by using hierarchical data, an image can be displayed in response to a wider range of resolution changes, and therefore the lowermost image may be created by connecting a plurality of images obtained by capturing partial areas. In such a case, an image obtained by joining a plurality of images may be used as a measurement image, and each map may be created using the measurement image after joining. Alternatively, an image of each region before joining may be used as a measurement image, and after creating a map for each region, a final map may be created by joining each map.

  According to the present embodiment described above, a height map, a normal map, and a horizon map are prepared as display image data, in addition to a color map that holds color information for each pixel. In the image display device, by expressing parallax, unevenness, and shadow, it is possible to display an image with a sense of presence that is difficult to obtain with only color information. Such an embodiment is particularly effective for an image of an object in which a detailed surface structure is important, such as an oil painting.

  Further, by changing the processing to be performed depending on the height of the viewpoint, it is possible to reproduce the impression given to the person when the distance to the object is actually changed, and to adjust the processing load. It can also be used as one of the representation methods of the image data creator. When the user inputs an instruction for enlargement / reduction, by controlling so that the delay time from the input signal to the image update changes continuously according to the number of processes applied to the image, various effects can be achieved with more natural movement. You can add.

  Such a feature becomes more effective in a mode in which the image data has a hierarchical structure and the variable range of resolution is widened. At this time, if the height map, normal map, and horizon map other than the color map are also set as hierarchical data, and data in the same area of the corresponding hierarchy is used, the processing becomes equivalent regardless of the resolution. Good processing can be done quickly. However, when the processing to be performed differs depending on the height of the viewpoint, that is, the resolution range, the size of the image data can be suppressed by preparing only the necessary hierarchy for the height map, normal map, and horizon map. In addition, switching is facilitated by switching processing to be performed in accordance with switching of the hierarchy used for image display.

  In order to easily realize such an image display, when generating image data, a normal map is generated from a measurement image, and a height map is generated by integrating the normal map. In order to generate a height map, it is usually required to measure an object using a laser or take images from a plurality of angles. According to the present embodiment, the normal map, height map, and horizon map are created by using a measurement image in which the camera is fixed and only the light source is moved, and thus image data can be created more easily. Also, the accuracy of the height map can be increased by using the result of scanning the normal map from a plurality of directions.

  Furthermore, when each map is to be hierarchical data, normal map hierarchical data can be easily generated by creating height map hierarchical data once from the normal map and then creating each normal map hierarchy. . The horizon map can also be generated easily because each layer can be generated directly from each layer of the height map. Alternatively, the image data can be easily generated even if the hierarchical data of each map is created by creating a map corresponding to each hierarchy after the measurement image is hierarchized.

  The present invention has been described based on the embodiments. Those skilled in the art will understand that the above-described embodiment is an exemplification, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  For example, in the present embodiment, image data including each map is created based on a photographed image. On the other hand, a pseudo oil painting canvas may be displayed on the screen, and an image data generation device having an image drawing function that allows a user to draw a picture using an input device such as a mouse may be realized. At this time, the user can enlarge / reduce the picture drawn by the user and generate a normal map, height map, horizon map corresponding to the carrying of the user's brush etc. Go. Thereby, the user can draw a picture with a feeling of unevenness on the screen as if it were drawing an oil painting.

DESCRIPTION OF SYMBOLS 1 Image processing system, 10 Image processing apparatus, 12 Display apparatus, 20 Input apparatus, 38 Tile image, 44 Display processing part, 50 Hard disk drive, 60 Main memory, 70 Buffer memory, 100 Control part, 102 Input information acquisition part, 106 Load block determination unit, 108 load unit, 110 drawing area specifying unit, 112 decoding unit, 114 display image processing unit, 124 parallax expression unit, 126 color expression unit, 128 shadow expression unit, 130 image output unit, 132 height map, 134 Color map, 136 Normal map, 138 Horizon map, 300 Image data generation device, 302 Measurement image acquisition unit, 304 Normal map creation unit, 306 Height map creation unit, 308 Horizon map creation unit, 310 Color map creation unit, 312 storage device.

Claims (21)

A storage unit storing a plurality of types of image data used for each of a plurality of types of processing for rendering an image to be displayed;
In accordance with the viewpoint movement request input by the user, a drawing area specifying unit that specifies the frame coordinates of the area to be newly displayed in the display target image;
A display image processing unit that draws an area of the display image corresponding to the frame coordinates using the image data stored in the storage unit and outputs the image data to a display device;
The display image processing unit compares the height of the viewpoint with respect to the image plane to be displayed, which is determined by the viewpoint movement request, and a threshold value predetermined for the height of the viewpoint, thereby performing the plurality of types of processing. The type of processing to be performed is switched according to the height of the viewpoint, and when the processing to be performed includes processing that requires the direction of the light source, the direction of the light source to be set is changed according to the height of the viewpoint. In addition, an image processing apparatus that renders a region of the display image by performing the processing. The processing performed by the display image processing unit is as follows:
Among the image data, a process of expressing colors using a color map that holds color information for each pixel;
Processing for expressing parallax with respect to an object by performing parallax mapping using a height map that holds height information for each pixel in the image data with respect to the image plane of the image and the viewpoint. When,
By performing bump mapping using a normal map that holds information on the normal vector of the surface of the object in the image for each pixel in the image data, irregularities are expressed on the surface of the object in the image. Processing,
Of the image data, a shadow corresponding to the set light source direction is obtained by performing horizon mapping using a horizon map that holds information on the direction of the light source when the shadow of the object is generated in each pixel. And the process of attaching
The image processing apparatus according to claim 1, comprising at least one of the following.   The image processing apparatus according to claim 1, wherein the display image processing unit increases the number of processes to be performed as the viewpoint approaches the image plane. Each of the plurality of types of image data has a hierarchical structure in which two-dimensional data corresponding to a plurality of image planes representing one image at different resolutions are layered in the order of resolution,
The image processing apparatus according to claim 1, wherein the display image processing unit switches a hierarchy of data to be used according to a height of the viewpoint. The display image processing unit also switches the type of processing to be performed at the height of the viewpoint when switching the hierarchy of data to be used,
The plurality of types of image data stored in the storage unit include image data having a number of layers different from that of at least one type by including only layers used for rendering by the display image processing unit. 5. The image processing apparatus according to 4. The plurality of types of image data includes at least a color map hierarchical structure that holds color information for each pixel, and information on the direction of a light source when a shadow of an object in the image is generated for each pixel. A map hierarchy, and
  The display image processing unit expresses a color of an object using the color map, and adds a shadow corresponding to a set light source direction by performing horizon mapping using the horizon map. The image processing apparatus according to claim 4 or 5.   The display image processing unit adjusts the update speed of the display image so as to delay the time from the input of the viewpoint movement request to the update of the display image as the number of processes to be performed increases. 7. The image processing device according to any one of items 1 to 6. An image data generation device that generates data of a plurality of types of images used to display an image of an object on a display device from a measurement image obtained by photographing the object with a camera,
A measurement image acquisition unit for acquiring a plurality of measurement images obtained by photographing the object with different directions of the light source;
A color map creation unit that creates a color map that holds color information for each pixel from a measurement image in a direction of a predetermined light source among the measurement images;
Analyzing the measurement image by illuminance difference stereo method, a normal map creating unit for creating a normal map representing the normal vector of the object corresponding to the pixels in the image;
A height map that represents the height of the target object in correspondence with the pixels by scanning the entire normal map in a predetermined direction and integrating the inclination of the target object in the predetermined direction obtained from the normal vector. A height map creation section for creating
A storage unit for storing the color map, the normal map, and the height map as image data;
An image data generation apparatus comprising: The color map creation unit generates color map hierarchical data in which a plurality of color maps of different resolutions generated by reducing the measurement image are layered in the order of resolution,
The height map creation unit creates a height map of the same resolution using the normal map corresponding to the resolution of the measurement image created by the normal map creation unit, and generates the reduced height map. Generate hierarchical data of height maps, which is a hierarchy of multiple height maps of different resolutions in the order of resolution,
The normal map creation unit scans the height map of the hierarchy other than the hierarchy corresponding to the resolution of the measurement image in a plurality of directions in the height map of each hierarchy in the hierarchy data of the height map in a plurality of directions. By differentiating the height, the inclination information of the object in each scanning direction is acquired, and the normal vector obtained by combining the normal vectors in each scanning direction obtained from the inclination for each pixel is calculated. 9. The image data generation apparatus according to claim 8, wherein normal image hierarchy data is generated by hierarchizing a plurality of resolution normal maps in the order of resolution.   The height map creation unit creates a final height map by scanning the entire normal map a plurality of times with different scanning directions and averaging the plurality of height maps acquired for each scanning direction. The image data generation device according to claim 8 or 9, characterized by the above. A horizon map creating unit that creates a horizon map that represents information of the direction of the light source when a shadow of an object occurs in each pixel using the height map,
The image data generation apparatus according to claim 8, wherein the storage unit also stores a horizon map as image data. By using the height map of each hierarchy in the hierarchy data of the height map, generating a horizon map that represents information on the direction of the light source for each pixel when the shadow of the object is generated in each pixel, A horizon map creation unit that generates hierarchical data of a horizon map obtained by hierarchizing a plurality of horizon maps different in resolution order,
The image data generation apparatus according to claim 9, wherein the storage unit also stores a horizon map as image data. The measurement image acquisition unit generates, for each light source direction, hierarchical data of measurement images obtained by hierarchizing a plurality of measurement image data with different resolutions generated by reducing each measurement image having different directions of the light source.
The normal map creation unit analyzes the measurement image data of each layer in the hierarchy data of the measurement image, and creates a normal map corresponding to each layer, thereby generating a plurality of normal maps having different resolutions in the order of resolution. 9. The image data generation apparatus according to claim 8, wherein hierarchical image data is generated. In accordance with the viewpoint movement request input by the user, specifying the frame coordinates of the area to be newly displayed in the display target image;
At least one of a plurality of types of image data stored in a storage device and used for each of a plurality of types of processing for rendering an image to be displayed is read out and used to display a display image corresponding to the frame coordinates. Drawing an area;
Outputting the data of the drawn area to a display device,
The drawing step includes:
By comparing the height of the viewpoint with respect to the image plane to be displayed, which is determined by the viewpoint movement request, and a threshold value predetermined for the height of the viewpoint, the type of processing to be performed among the plurality of types of processing is determined. When the process to be executed includes a process that requires the direction of the light source, the direction of the light source to be set is changed according to the height of the viewpoint, and the process is performed. An image processing method characterized by drawing a region of the display image. An image data generation method for generating data of a plurality of types of images used to display an image of a target object on a display device from a measurement image obtained by photographing the target object with a camera,
Acquiring a plurality of measurement images obtained by photographing the object with different directions of the light source and storing them in a memory;
Using the measurement image in the direction of a predetermined light source among the measurement images read out from the memory, creating a color map that holds color information for each pixel;
Analyzing the measurement image read from the memory by an illuminance difference stereo method, and creating a normal map representing the normal vector of the object corresponding to the pixels in the image;
A height map that represents the height of the target object in correspondence with the pixels by scanning the entire normal map in a predetermined direction and integrating the inclination of the target object in the predetermined direction obtained from the normal vector. The steps of creating
Storing the color map, the normal map, and the height map as image data in a storage device;
An image data generation method comprising: In accordance with the viewpoint movement request input by the user, a function for specifying the frame coordinates of the area to be newly displayed in the display target image;
At least one of a plurality of types of image data stored in a storage device and used for each of a plurality of types of processing for rendering an image to be displayed is read out and used to display a display image corresponding to the frame coordinates. The ability to draw areas,
A function of outputting the data of the drawn display image to a display device;
A computer program characterized by causing a computer to realize
The drawing function is
By comparing the height of the viewpoint with respect to the image plane to be displayed, which is determined by the viewpoint movement request, and a threshold value predetermined for the height of the viewpoint, the type of processing to be performed among the plurality of types of processing is determined. When the process to be executed includes a process that requires the direction of the light source, the direction of the light source to be set is changed according to the height of the viewpoint, and the process is performed. A computer program for drawing an area of the display image. A computer program that causes a computer to realize a function of generating data of a plurality of types of images used to display an image of the target object on a display device from a measurement image obtained by photographing the target object with a camera,
A function of acquiring a plurality of measurement images obtained by photographing the object with different light source directions and storing them in a memory;
A function of creating a color map that holds color information for each pixel by using a measurement image in the direction of a predetermined light source among the measurement images read from the memory;
A function of analyzing the measurement image read from the memory by an illuminance difference stereo method, and creating a normal map representing the normal vector of the object corresponding to the pixels in the image;
A height map that represents the height of the target object in correspondence with the pixels by scanning the entire normal map in a predetermined direction and integrating the inclination of the target object in the predetermined direction obtained from the normal vector. With the ability to create
A function of storing the color map, the normal map, and the height map as image data in a storage device;
A computer program for causing a computer to realize the above. In accordance with the viewpoint movement request input by the user, a function for specifying the frame coordinates of the area to be newly displayed in the display target image;
At least one of a plurality of types of image data stored in a storage device and used for each of a plurality of types of processing for rendering an image to be displayed is read out and used to display a display image corresponding to the frame coordinates. The ability to draw areas,
A function of outputting the data of the drawn display image to a display device;
A computer-readable recording medium on which a computer program is recorded,
The drawing function is
By comparing the height of the viewpoint with respect to the image plane to be displayed, which is determined by the viewpoint movement request, and a threshold value predetermined for the height of the viewpoint, the type of processing to be performed among the plurality of types of processing is determined. When the process to be executed includes a process that requires the direction of the light source, the direction of the light source to be set is changed according to the height of the viewpoint, and the process is performed. A recording medium having a computer program recorded thereon, wherein a region of the display image is drawn. A recording medium on which a computer program for causing a computer to realize a function of generating data of a plurality of types of images used to display an image of the object on a display device from a measurement image obtained by photographing the object with a camera Because
A function of acquiring a plurality of measurement images obtained by photographing the object with different light source directions and storing them in a memory;
A function of creating a color map that holds color information for each pixel by using a measurement image in the direction of a predetermined light source among the measurement images read from the memory;
A function of analyzing the measurement image read from the memory by an illuminance difference stereo method, and creating a normal map representing the normal vector of the object corresponding to the pixels in the image;
A height map that represents the height of the target object in correspondence with the pixels by scanning the entire normal map in a predetermined direction and integrating the inclination of the target object in the predetermined direction obtained from the normal vector. With the ability to create
A function of storing the color map, the normal map, and the height map as image data in a storage device;
The recording medium which recorded the computer program characterized by making a computer implement | achieve. A data structure of an image file read from a storage device by an image processing device for displaying at least a part of an image on a display,
A color map for storing color information for each pixel, which is used for processing to express the color of the display image
A height map that holds, for each pixel, information on the height of the object relative to the image plane, which is used for processing to express parallax with respect to the object in the image;
A normal map that holds information on the normal vector of the surface of the object for each pixel, which is used for processing to express irregularities on the surface of the object in the image;
A horizon map that holds information on the direction of the light source when the shadow of the object is generated on each pixel, which is used for the process of applying a shadow corresponding to the set direction of the light source,
,
Among the types of processing using the color map, height map, normal map, and horizon map, the type of processing performed by the image processing apparatus is determined by the input viewpoint movement request, and the viewpoint height relative to the image plane is determined. The light source to be set when the map corresponding to the type of processing is read from the storage device by the image processing device and the processing to be performed includes processing that requires the direction of the light source. A data structure of an image file, wherein an image of an area to be newly displayed is rendered by changing the direction of the image according to the height of the viewpoint and executing the processing.   Each of the color map, the height map, the normal map, and the horizon map has a hierarchical structure in which data corresponding to image data of each resolution when a single image is represented at different resolutions is layered in order of resolution. 21. The data structure of an image file according to claim 20, wherein a hierarchy of data used for drawing is switched according to the height of the viewpoint by the image processing apparatus.

Images