JP2005327314A - Image display method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image display method which displays a large number of retrieval object images at high speed in a three-dimensional space, with intelligible image expression, and by which user's eyes are hardly tired. <P>SOLUTION: The image display method includes: processing for reading a plurality of pieces of image data and arrangement information of each image in the three-dimensional space; viewpoint setting/transfer processing in a space for specifying setting/transfer of viewpoints in the space; processing for calculating simplified two-dimensional arrangement expressing an image data original form as a two-dimensional image converted by parallel magnifying projection processing for performing parallel movement processing corresponding to directions of the respective images and magnifying/reducing processing corresponding to size of the respective images on the basis of relation between the arrangement information of each image and the viewpoint information in the space; display image generation processing for generating a display image according to the simplified two-dimensional arrangement; and processing for displaying the display image. The image expression is made intelligible by guide display of a multi-resolution composite image, a segmented image and a background image, fatigue of the eyes is evaluated and minimized on the basis of optical flow. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a method for easily displaying an image on a display at high speed. In particular, the present invention relates to a high-speed display method for a large amount of images in a three-dimensional space, and a display method in which eyes are not tired by zooming or scrolling images. For example, the present invention can be applied to an electronic art museum system or a visual image search (hereinafter referred to as browsing search) system.

  In the multimedia information age, opportunities to display images on computer displays are increasing. With the progress of the Internet, the importance of techniques for easily and reliably searching for desired image information is increasing. When searching for text on the Internet, narrowing down by keywords is effective and has been put to practical use as a search site. However, when searching for multimedia materials represented by images, the use of keywords and the automatic extraction of material information are still in the research stage. At present, many candidates are presented as search results and judged by humans. Browsing search is widely used. This technology for improving browsing search is an important key technology.

  On the other hand, even in an individual or office environment, the amount of multimedia data stored in a computer is increasing year by year, and a technique for finding desired data or browsing efficiently is also important here. A general processing flow for searching image data such as the Internet in the prior art is roughly as follows.

  First, in order to narrow down the desired data by the user, the search target is limited to some extent by using a keyword or the like by using a search search site by keyword or the like. Next, the search target is further limited using image features such as color distribution. Further, the search target narrowed down to some extent by the above narrowing is displayed on the display, and the user conducts browsing search visually.

  Therefore, it is considered that the browsing search for the image has the following characteristics.

  The first feature is that the number of image data to be displayed for browsing search is quite large and can be several hundred to several thousand. The display target is selected from a huge population, and which element is selected is determined by a narrowing operation immediately before display, and it often takes a little time for display preprocessing. The second feature is that in order to perform the narrowing-down process, it is necessary to read the population once into the computer, so that it can be assumed that the display object is stored in the storage device of the computer that performs the search. Even when browsing a large number of images managed by individual users instead of browsing search, it is considered that the display target has these properties.

  Further, the following conditions are required when displaying a large amount of images.

  The first condition is that sufficient visibility is obtained for the display image. However, it is sufficient that the user knows what is being displayed, and precise fine display only needs to be performed separately. Therefore, display precision is not so limited in list display.

  The second condition is a list of display images. If the listing property is ensured between the display images, it is easy to determine how far a large number of images have been visually confirmed, and it is possible to easily compare the images.

  The third condition is the operability of the display image. Since there are a large number of images, it is required that operations such as scrolling and zooming the display image can be performed at a sufficiently high speed. Further, when the operability is improved, there is an advantage that even if the search operation is continued, the user is not tired.

  As display methods capable of browsing search, typical methods conventionally used include the following.

  The first conventional browsing search method is a method (Japanese Patent Laid-Open No. 62-248376) for displaying an image in a reduced size. An image (thumbnail image) obtained by reducing a plurality of images to be displayed is created and placed on the display screen. According to this method, there is an advantage that it is easy to understand how far it has been seen. However, since it is necessary to reduce the size of each image as the number of display images increases, there is a drawback that the details of the image cannot be seen and are difficult to understand.

  The conventional second browsing search method is a method of dividing and displaying pages. In this method, the display image is not reduced, or the size of the display image is secured to some extent while maintaining a certain reduction ratio, and the display image is divided into a plurality of pages by the amount that fits on one page of the display screen. This is a method of editing and displaying each page in turn. Although there are advantages in that individual images are easy to understand, there is a drawback in that the list is lacking, it is not easy to determine how far the images have been viewed, and comparison between display images is difficult.

  A conventional third browsing search method is a method of scrolling and displaying. Arranged so that a list can be displayed on a virtual large screen, and in the display processing on the actual display, a scroll function is provided so that an area of one display is displayed as a part of the virtual large screen It is. According to this method, it becomes easier to compare with other images than the page division method, which is the second browsing search method, but it is still not sufficient.

  The conventional fourth browsing search method is a method of arranging and displaying in a virtual three-dimensional space. In this method, each display image is virtually arranged in a virtual three-dimensional space. A function for moving the viewpoint in the virtual three-dimensional space is provided. As the viewpoint approaches each individual image, the image is zoomed and displayed. Our ability to grasp 3D space makes it easy to judge how far you have seen and to compare with other images, and also to ensure the visibility of individual images. . There is JP-A-9-259130 as a known example. This discloses a technique of hierarchically arranging information and performing a browsing search.

  In this conventional technique of displaying in a virtual three-dimensional space, a three-dimensional object is projected onto a display using central projection (perspective projection) in order to obtain a display screen on the display (projection plane). Yes. This central projection, as shown in FIG. 22, connects the points on the surface of the three-dimensional object and the points on the display in the following relationship. The line-of-sight direction is the Z axis, the display is placed at, for example, Z = 1, and the point where the line segment connecting the projection center and one point on the surface of the three-dimensional object intersects the display is the corresponding point. Any value other than 1 can be used as the value of Z on which the display is placed. If the coordinates of the point on the surface of the three-dimensional object are (X, Y, Z), the coordinates (x, y) of the corresponding point on the display are x = X / Z and y = Y / Z. In principle, three-dimensional display can be realized by drawing all pixels on the surface of the three-dimensional object at corresponding points on the display. Since the display is composed of pixels, the process is actually a process of assigning a color to each pixel of the display based on the above correspondence. According to this central projection, it is possible to obtain a realistic display screen that is close to the appearance when a human views a three-dimensional object placed in a virtual three-dimensional space from the projection center.

  In the prior art, when each image is arranged in a virtual three-dimensional space and displayed from a viewpoint set in the space using this central projection, the projection center in FIG. A realistic and realistic display screen can be obtained when each image arranged in the dimensional space is viewed.

  Therefore, when the number of display target images is relatively large, a three-dimensional display method is excellent as a browsing search, but this method has the following problems.

  The first problem is that the three-dimensional display increases the amount of calculation compared to the two-dimensional display, and takes a long time for display processing. In the three-dimensional space display process, in particular, when a large amount of images are displayed on the display, the display speed is lowered and it becomes difficult to use. Since a browsing search is performed by moving the viewpoint, smooth display is required in accordance with the movement. However, if the display speed is reduced, the movement cannot be performed smoothly, and the usability is greatly reduced.

  The second problem is that it is difficult to understand the contents of individual images during viewpoint movement. When the viewpoint is moved in the virtual three-dimensional space, if the viewpoint movement speed is increased, it is difficult to grasp the contents of the image moving at a high speed. In addition, because the viewpoint is moved in a virtual three-dimensional space, the image is displayed with distortion due to the viewing angle with respect to each image, and the distortion changes due to the movement of the viewpoint. There is.

  The third problem is that even if a large number of small images are arranged in a virtual three-dimensional space, it is difficult to see them three-dimensionally. It is difficult to grasp the three-dimensional depth. If anything, images with different sizes appear to be arranged two-dimensionally. In this case, there is no point in arranging in a three-dimensional space, and there is a risk of causing some confusion.

  The fourth problem is that if the viewpoint movement process is repeated, the user's eyes become tired and the usability is reduced. In order to grasp the contents of the image moving in the virtual three-dimensional space, the user needs to move the eye viewpoint (attention point) as the image moves, and this viewpoint movement is one of the causes of eye fatigue. It has become one.

  In view of the above problems, the present invention provides an image display method for displaying an image to be searched virtually arranged in a virtual three-dimensional space, and displays a large amount of search target images in the virtual three-dimensional space at high speed. An object of the present invention is to provide an image display method capable of processing and easily grasping the image contents even if the viewpoint moving speed in the virtual three-dimensional space is increased. It is another object of the present invention to provide an image display method based on an image expression technique that makes it easy to intuitively grasp the arrangement position of an image in a virtual three-dimensional space. It is another object of the present invention to provide an image display method in which a user's eyes are not tired even when the viewpoint is moved in a virtual three-dimensional space.

  In order to achieve the object of displaying an image to be searched at high speed in a three-dimensional space, the image display method of the present invention is an image display method for arranging and displaying an image in a virtual three-dimensional space. A process for reading image data of one or more images, a process for reading arrangement information of each image in the virtual three-dimensional space, and a parameter in the virtual three-dimensional space based on the viewpoint position and line-of-sight direction parameters. Based on the viewpoint setting / movement processing in the space that specifies the setting / movement of the viewpoint and the relationship between the placement information of each image in the virtual three-dimensional space and the viewpoint information in the space, The two-dimensional arrangement of each image seen from the viewpoint in the space is converted by parallel enlargement projection processing that performs parallel movement processing corresponding to the direction of each image viewed from the viewpoint and enlargement / reduction processing corresponding to the size. Image similarity Processing for obtaining a simplified two-dimensional arrangement expressed as a two-dimensional arrangement of images and display image generation for processing a read image data and generating a display image according to the obtained simplified two-dimensional arrangement of each image And a process for displaying the generated display image on a display device.

  Further, in addition to the display processing using the parallel enlargement projection processing, the far-distance image over-reduction projection processing, the small image omission processing, the far-distance image omission processing, the near-distance image omission processing, the overhang image omission processing, and the hiding determination efficiency processing It is possible to use a combination of image display speed-up techniques such as

  By the above method, a large amount of search target images can be displayed at high speed in a virtual three-dimensional space using a simplified two-dimensional arrangement by parallel enlargement projection processing.

  In order to achieve the object of displaying the search target image in a three-dimensional space at high speed with easy-to-understand expression, the image display method of the present invention arranges and displays an image in a virtual three-dimensional space. An image display method, wherein the image data is multi-resolution image data composed of a set of image data having one or more resolutions, and an image data reading process for reading one or more resolution image data of one or more images A process for reading the arrangement information of each image in the virtual three-dimensional space, and a viewpoint setting / moving process in the space that specifies setting / movement of the viewpoint in the space by the viewpoint position and the line-of-sight direction parameter in the virtual three-dimensional space; Then, the projection processing based on the relationship between the arrangement information of each image in the virtual three-dimensional space and the viewpoint information in the space obtains a two-dimensional arrangement of the position, orientation, and size of each image seen from the viewpoint in the space. And a display image generation process for selecting one or more resolution image data from among the multi-resolution image data read in accordance with the obtained two-dimensional arrangement and writing it to the display image to generate the display image. And a process for displaying the displayed image on a display device.

  In addition, as an image display method of the present invention for displaying an image to be searched in an easy-to-understand expression in a three-dimensional space, a process of reading image data of one or more images, and a virtual three-dimensional space of each image Processing to read the placement information in the space, the viewpoint setting / movement process in which the setting / movement of the viewpoint in the space is specified by the parameters of the viewpoint position and the line-of-sight direction in the virtual three-dimensional space, and the virtual three-dimensional space of each image A process for obtaining a two-dimensional arrangement of the position, orientation, and size of each image viewed from the viewpoint in space by a projection process based on the relationship between the arrangement information in the image and the viewpoint information in the space, and the obtained two-dimensional A display image generation process for generating a display image by writing into the display image cut-out image data cut out from the read image data by an area of the size of the image of the two-dimensional arrangement according to the arrangement; Display image, characterized in that it comprises a process of displaying on the display device.

  Furthermore, in addition to multi-resolution image composition display processing and important area extraction display processing, alternative display processing when omitted, processing for limiting the range of viewpoint movement, highlight display processing for specific images, display processing for coordinate axes, etc., display with frame of image It is possible to use a combination of image expression techniques such as processing and image layout expansion processing.

  By the above method, it is possible to perform image representation suitable for displaying a large amount of images in a virtual three-dimensional space that is easy for the user to use, using a multi-resolution composite image, a cut-out image, or the like.

  In order to achieve the object of making the arrangement position of each image arranged in the virtual three-dimensional space easy to understand, the image display method of the present invention provides an image display in which an image is arranged and displayed in the virtual three-dimensional space. A method of reading image data of one or more images, a process of reading arrangement information of each image in a virtual three-dimensional space, and a background image virtually existing as a background in the virtual three-dimensional space A process for reading image data, a process for reading arrangement information of a background image in a virtual three-dimensional space, and a setting and movement of a viewpoint in the space by parameters of the viewpoint position and the line-of-sight direction in the virtual three-dimensional space Each image and background image that can be viewed from the viewpoint in space by the viewpoint setting / moving process and the projection processing based on the relationship between the arrangement information of each image and background image in the virtual three-dimensional space and the viewpoint information in space. Display for generating a display image by processing the read image data and background image data in accordance with the processing for obtaining the two-dimensional arrangement of the position, orientation, and size of the image, and the two-dimensional arrangement of each obtained image and background image An image generation process and a process for displaying the generated display image on a display device are provided.

  By the above method, it is possible to display a background object image serving as a guide for making the arrangement position easy to understand in the virtual three-dimensional space, and to express the position of each image in the virtual three-dimensional space in an easy-to-understand manner. .

  In order to achieve the object of providing an image display method in which the user's eyes are not tired, the image display method of the present invention is an image display method for displaying an image while moving it, and a process of reading image data; Based on the read image data and the movement parameter, a process for reading a movement parameter for designating an image movement method, an optical flow determined between display images of adjacent frames in time, and an actual pixel of each image A movement parameter / image that adjusts at least one of the movement parameter and the image data so as to reduce a viewpoint movement error that is a difference from movement and to increase an image movement amount between temporally adjacent frames. In accordance with data adjustment processing and the adjusted movement parameter / image data, the read image data is processed to display a display image. A display image generation processing to be generated, characterized by comprising a process of displaying on the display device a display image that is generated.

  By the above method, the movement parameter optimization processing method of the image display process of the present invention is used, and the movement parameter is optimized based on the optical flow to suppress eye fatigue of the user who performs browsing search and browsing. The possible image display can be performed.

  According to the three-dimensional display processing speed-up technology for image display processing of the present invention, a large amount of search target images can be displayed at high speed in a virtual three-dimensional space using a simplified two-dimensional arrangement by parallel enlargement projection processing. it can.

  Further, according to the three-dimensional image expression technique of the image display processing of the present invention, an image suitable for displaying a large amount of images in a virtual three-dimensional space that is easy for the user to understand using a multi-resolution composite image, a cut-out image, or the like. Can express.

  Further, according to the 3D space expression technology using the background image of the image display processing of the present invention, it is possible to display a background object image serving as a guide for making the arrangement position easy to understand in the virtual 3D space, The position of each image in the virtual three-dimensional space can be easily expressed.

  Further, according to the movement parameter optimization processing technology of the image display processing of the present invention, it is possible to optimize the movement parameter based on the optical flow and to suppress eye fatigue of the user who performs browsing search and browsing. Image display processing can be performed.

  An embodiment of an image display method of the present invention will be described with reference to the drawings.

  The image display method of the present invention displays an image in a three-dimensional virtual space by combining one or more of the following basic techniques of the present invention.

  The first basic technique is a three-dimensional display processing speed-up technique optimized for displaying a large amount of images. As the three-dimensional display processing speed-up technology, a plurality of display processing speed-up technologies are newly disclosed in the present invention, and they can be used in any combination. As will be described later, display processing using parallel enlargement projection processing, long-distance image over-reduction projection processing, small image omission processing, long-distance image omission processing, short-distance image omission processing, protruding image omission processing, and hiding determination efficiency processing In combination with image display speed-up technology. These image display speed-up techniques will be described in detail later in the first embodiment.

  The second basic technique is a three-dimensional image expression technique optimized for displaying a large amount of images. As this three-dimensional image expression technique, a plurality of image expression techniques are newly disclosed in the present invention, and they can be used in any combination. As described later, multi-resolution image composition display processing, important area cut-out display processing, alternative display processing at the time of omission, processing for limiting the range of viewpoint movement, highlight display processing for specific images, display processing for coordinate axes, etc., image frame attachment A combination of image representation techniques such as display processing and image layout expansion processing is used. These image expression techniques will be described in detail later in the second embodiment.

  The third basic technique is a three-dimensional space expression technique using a background image. The three-dimensional space expression technique using the background image will be described later in detail in Embodiment 3 below.

  The fourth basic technique is a movement parameter optimization processing technique. This technique optimizes movement parameters based on optical flow to suppress eye fatigue. This movement parameter optimization processing technique will be described in detail later in a fourth embodiment.

  In addition, the above-described three-dimensional display processing speed-up technology of the present invention, three-dimensional image representation technology, three-dimensional space representation technology using a background image, and movement parameter optimization processing technology are technologies for improving image display processing. Each can be applied independently, and can be used in any combination.

  In the following description, the terms image and display are used in the following sense. This is a standard used in the display method. An image is an array in which small rectangular areas called pixels are two-dimensionally spread. An image with a large number of pixels is called a high resolution or large image, and an image with a small number of pixels is called a low resolution or small image. Each pixel can be set in color (including lightness here) by a set of several numerical values. As an example of a method of associating a set of numerical values and colors, there is an RGB expression method. When each color component is expressed by 8 bits in the RGB expression method, the brightest white is, for example, (255, 255, 255).

  A display is a display device that can be physically viewed by a user. By writing a value to a special memory in the computer, it is possible to display a color corresponding to a corresponding portion of the display. Therefore, it can be regarded as one special image in the display method. In the present invention, a display is used in this sense.

  The basic technique for realizing the image display method of the present invention will be described below.

(Embodiment 1)
The first embodiment is a three-dimensional display processing speed-up technique optimized for a large amount of image display, which is one of the basic techniques of the image display method of the present invention.

  As one of the three-dimensional display processing speed-up techniques optimized for displaying a large amount of images according to the present invention, first, image generation processing according to a simplified two-dimensional arrangement by parallel enlargement projection will be described.

  As described in the prior art, if the central projection of the prior art is used, a display screen with a realistic sensation that is close to the appearance when each image arranged in the virtual three-dimensional space is viewed with the viewpoint in space as the projection center Can be obtained.

  However, as described in the prior art, various problems such as a problem of display processing speed are caused. In the present invention, a projection method capable of obtaining a simplified converted image that is sufficient in browsing search and browsing processing is used without using central projection.

  A general three-dimensional object may face various directions depending on the arrangement angle. However, in browsing search and browsing, priority is given to making it easy to understand what each displayed image is. What is necessary is just to display so that it may face with respect to a gaze direction. When the orientation of each image is determined in this way, for example, a rectangular image is displayed in a rectangular shape on the display. That is, a converted image obtained by enlarging / reducing and rotating the original image around a certain position on the display is displayed. Here, the outer shape is the same as the original shape of the original image. In particular, when the pixel direction of each image (the direction of the straight line connecting adjacent pixels) is parallel to the display pixel direction, that is, when the rectangular image data faces the display without any inclination. In this case, the rotation process is not necessary. Even when the top and bottom are considered, the rotation angle is a multiple of 90 degrees (0 degrees, 90 degrees, 180 degrees, and 270 degrees). In actual browsing search and browsing, each image is often displayed and searched and browsed in this way, and the display sufficiently fulfills the purpose of browsing search and browsing.

  Therefore, in the present invention, when obtaining a projection of an image to be arranged in the virtual three-dimensional space on the display, the image data is converted based on the relationship between the arrangement of each image in the virtual three-dimensional space and the in-space viewpoint information. On the other hand, projection conversion is performed by parallel translation processing corresponding to the direction of each image viewed from the in-space viewpoint and parallel enlargement projection processing that performs only enlargement / reduction processing corresponding to the size of each image viewed from the in-space viewpoint. As this parallel enlargement projection processing, a rotation processing of a multiple of 90 degrees in consideration of the vertical direction can be included.

  According to this parallel enlargement projection process, a simplified two-dimensional arrangement in which the two-dimensional arrangement of each image seen from the viewpoint in space is expressed as the two-dimensional arrangement of the similar image of the original image can be obtained. Since the display processing using the simplified two-dimensional arrangement by the parallel enlargement projection processing is only the translation processing of the original image data, the enlargement / reduction processing, and the rotation processing of a multiple of 90 degrees, the general central projection processing and arbitrary angle It can be executed at higher speed than the rotation process. In this way, the amount of processing calculation in the case of obtaining a two-dimensional arrangement on the display surface of an image writing frame similar to the original shape of the image data and writing the display image using the original image data to the image writing frame is as follows. Small and fast processing is possible.

  The present invention is a simplified two-dimensional image that is sufficiently simplified for browsing search and browsing by only parallel enlargement projection transformation, regardless of the actual visual state viewed from the viewpoint in the space. By displaying the display image by arrangement, the projection processing and the display processing are simplified and the processing speed is increased.

In the field of computer vision, which studies the process of obtaining three-dimensional information from images, weak perspective projection (weak perspective projection) is used as an approximation of central projection.
projection and paraperspective
projection) is used. These coincide with the central projection when the display and each image are parallel in the virtual three-dimensional space. Even if this is not the case, if the direction of the pixels of each image is parallel to the direction of the pixels of the display, the weak perspective projection and the parallel magnified projection match. Therefore, the parallel magnified projection of the present invention is a projection method that generalizes weak perspective projection so that projection can be performed with respect to an arbitrary three-dimensional arrangement by translation processing, enlargement / reduction processing, and rotation processing of a multiple of 90 degrees. It can be said that.

  The basic flow of the parallel enlargement projection processing of the present invention will be described in detail with reference to the flowchart of FIG.

  First, representative points are determined from the image (step S101). One representative point is determined for each image to be displayed. The representative point is, for example, the center of gravity.

  Next, the corresponding point on the display of the representative point is determined (step S102). The point (x0, y0) where the line segment connecting the representative point determined in step S101 and the viewpoint in space passes through the display is determined as the corresponding point on the display of the representative point.

  Next, the enlargement / reduction ratio of the entire image is determined (step S103). Here, the reciprocal of the Z value Z0 of the representative point determined in step S102 is set as the enlargement / reduction ratio of the entire image.

  Next, the enlargement / reduction ratio of each axis is determined (step S104). The enlargement / reduction ratio of each axis is determined as follows. First, orthogonal coordinate systems u and v are introduced on the image. The origin of this u, v coordinate system is a representative point. Component display in the XYZ coordinate system of unit vectors (pixel interval lengths) in the u and v directions is (uX, uY, uZ) and (vX, vY, vZ). At this time, if | uX | ≧ | vX |, the u axis is multiplied by uX and associated with the X axis, and the v axis is multiplied by vY and associated with the Y axis. That is, the point (u, v) on the image is associated with the point (uXu, vYv, Zo) in the XYZ coordinate system. Conversely, if | uX | <| vX |, the v-axis is multiplied by vX and associated with the X-axis. The u axis is multiplied by uY to correspond to the Y axis.

  Next, a process for associating each pixel of the image is performed (step S105). Each pixel (u, v) of the image is multiplied by the enlargement / reduction ratio of each axis to be converted to a value on the X and Y axes, and further multiplied by the enlargement / reduction ratio of the entire image. These are set as δx and δy. The pixel on the display to which the pixel corresponds is (x0 + δx, y0 + δy). However, it is possible to define the enlargement / reduction ratio of each axis to a value other than the above. For example, (u2X + u2Y) 1/2 or (v2X + v2Y) 1/2 may be used. The present invention is not limited to the details of the definition of the enlargement / reduction ratio, and can be applied.

  The above is the basic flow of the parallel enlargement projection processing of the present invention.

  Next, taking the case where the image to be projected is a rectangle as an example, a flowchart of FIG. 2 shows an example of a projection calculation process on a display of a rectangular image using parallel enlargement projection.

  First, images to be subjected to parallel enlargement projection processing are arranged in order of distance from the viewing direction, and the first image in this permutation is selected (step S201).

  Next, as shown in step S101 of the flowchart of FIG. 1, representative points are determined from the image, and as shown in step S102 of the flowchart of FIG. 1, the representative points of the image projected on the display are calculated. Further, corresponding points of the four vertices are also calculated (step S202). The enlargement / reduction ratio is obtained from the coordinate value of the representative point of the selected image, and the coordinates (X1, Y1), (X1 + w1, Y1), (X1 + w1, Y1), (4) of the rectangle projected onto the display from the coordinate values of the four vertices. X1, Y1 + h1) and (X1 + w1, Y1 + h1) are obtained.

  Next, the pixel to be processed is initialized (step S203). Here, the pixel (i, j) on the display to be processed is set to (X1, Y1).

  Next, corresponding image pixels are calculated (step S204). The pixel (I, J) on the image corresponding to the pixel (i, j) on the display is obtained.

  Next, the color of the pixel on the image is copied to the pixel on the display (step S205). Here, the color of the pixel (I, J) on the image is copied to the pixel (i, j) on the display.

  Next, the processing target pixel is moved. First, movement in the X-axis direction is performed. By increasing i by one, the pixel to be processed is set to the next pixel in the X-axis direction (step S206). If i = X1 + w1 + 1 (step S207: Y), it means that the rectangular image range has been exceeded in the X-axis direction, so that the X coordinate of the pixel to be processed is returned to the left end, and then in the Y-axis direction. Move. That is, i is returned to X1, and j is increased by 1 (step S208). If j = Y1 + h + 1 (step S209: Y), this means that the rectangular image range has been exceeded in the Y-axis direction, and the pixel has been processed over the entire rectangular image range. The process returns to the image selection processing step S201.

  Next, if i = X1 + ω1 + 1 has not been reached in step S207 (step S207: N), or if j = Y1 + h + 1 has not been reached in step S209 (step S209: N), the process returns to step S204, and the corresponding image again. The pixel is calculated and the color of the pixel on the image is copied to the pixel on the display (step S205).

  When the processing from step S204 to step S209 is repeated and j = Y1 + h + 1 is reached in step S209 (step S209: Y), it is confirmed whether or not the next image in the column remains (step S210: Y) Returning to step S201, the above process is repeated.

  If there remains no image to be selected (step S210: N), the process ends. Here, the case where the selected image does not remain includes the case where the representative point of the selected image is behind the viewpoint position (Z <0).

  The above is an example of the flow of processing for obtaining a two-dimensional arrangement image on a display using parallel enlargement projection for image data of a rectangular image.

  FIG. 3 shows an example of a display image using a simplified two-dimensional arrangement by parallel expansion projection processing of the present invention and an example of a two-dimensional display image by central projection processing of the prior art. Here, the original image is a rectangular image. As shown in FIG. 3 (a), in the example of the display image using the simplified two-dimensional arrangement by the parallel enlargement projection processing of the present invention, each image in the display image is a translation processing and enlargement of the original rectangular image.・ Similar images with only reduction processing and rotation processing, and display processing is possible at high speed. In the example of the two-dimensional display image by the center projection processing of the prior art in FIG. 3B, each image is obtained in the virtual three-dimensional space, and an image close to visual observation from the viewpoint in the space is obtained. Large and high-speed display processing is difficult.

  Next, the second 3D display processing speed-up technology of the present invention will be described.

  The second 3D display processing speed-up technology is a 3D display processing speed-up technology using multi-resolution images. By combining with the display image processing using the simplified two-dimensional arrangement by the parallel enlargement projection processing, it is possible to contribute to speeding up the three-dimensional display processing. Note that the three-dimensional image display processing using the multi-resolution image can be used from the viewpoint of a three-dimensional image expression technique optimized for a large amount of image display as described later in the second embodiment. In the first embodiment, aspects as a three-dimensional display processing speed-up technology will be described.

  Since the number of images to be displayed is enormous, enormous reading time and storage capacity are required to read all the images and place them in the main memory.

  When an image to be displayed is arranged in a virtual three-dimensional space, an image existing far away is displayed small, and even a low resolution is often sufficient. Thus, the resolution can be adjusted according to the distance from the viewpoint in space and the display size of the image. By using a reduced image, display speed can be increased and memory can be saved. Using this property, the three-dimensional display processing speed-up technology using a multi-resolution image of the present invention prepares a plurality of image data with different resolutions for an image to be displayed as a two-dimensional display image. When writing, image data having a resolution closest to the resolution according to the distance from the viewpoint in space or the display size of the image is selected, and image display processing is performed using the image data.

  FIG. 4 shows an example of generating a display image using a multi-resolution image. As shown in FIG. 4, a plurality of multi-resolution image data having different sizes are prepared from the original image data. Here, the contents of the image are simply scaled in advance according to the size. After determining the position and size of the image writing frame as a result of the projection process, select the multi-resolution image having the closest size to the image writing frame, and make fine adjustments as necessary. Write.

  As described above, by selecting image data with an appropriate resolution from the multi-resolution images that are hierarchized, the amount of image data can be reduced, the amount of calculation can be reduced, and the display processing can be speeded up. .

  Next, multiresolution image data to be used will be described. The multi-resolution image used in the three-dimensional display processing speed-up technology using the multi-resolution image may be prepared as a set of image data having one or more different resolutions, and the image data capacity may be different. The simplest multi-resolution image data includes a plurality of low-resolution images in which pixel data is thinned out spatially and uniformly with respect to the original image data. Further, as described later in the second embodiment, multi-resolution image data based on a composite image in which the resolution of an important local region is kept high and the resolution of other regions is lowered, or a cut-out image that cuts out only an important local region from an original image The multi-resolution image data by can also be mentioned. Thus, the multi-resolution image data used in the three-dimensional display processing speed-up technology using the multi-resolution image of the present invention is not limited to the content.

  In the image display processing using this multi-resolution image, the present invention further discloses the following three-dimensional display processing speed-up technology.

  The first is adjustment processing for reading multi-resolution image data into the memory.

  This is a process for adjusting whether to read this multi-resolution image data set collectively or to read only the image data of the corresponding resolution after the resolution to be displayed is determined among the multi-resolution images. In other words, in the multi-resolution image data reading process, the time of the reading process is before the display image generation process, and the process of reading all the sets of multi-resolution image data of the image to be read is performed. This is the time when one resolution image data is selected in the generation process, and is a process for selecting one of the processes for reading only the selected resolution image data among the multi-resolution image data of the image to be read.

  Thus, by making it possible to select either of the processes, the memory capacity and the display speed can be balanced. If there is enough memory available for reading multi-resolution image data, you can select the processing to read a set of multi-resolution image data as in the former, and if there is not enough available memory As in the latter case, the image data having the corresponding resolution may be read when the resolution to be used is determined.

  The second is an adjustment process for deleting the read multi-resolution image data from the memory.

  It is also possible to adjust the process of erasing the multi-resolution image data read into the memory by the reading process.

  The image data deletion process that deletes unnecessary multi-resolution image data from the memory is a process that deletes unnecessary image data when the image data recording capacity exceeds the capacity set in the memory. It is assumed that at the time when one resolution image data is selected in the image generation process, one of processes for deleting other resolution image data can be selected.

  Thirdly, in the browsing search, it is often difficult to spend time on the process before display, but it is possible to perform the process in advance for all elements of the population that can be displayed. Since the display target is only a small part of the population, image data can be read quickly before or during display by organizing the image data into a single file so that random access can be performed efficiently. Can be

  The fourth is processing for limiting the maximum resolution to a specified value. As an image used for browsing search, excessively high-resolution image data causes a reduction in processing speed, so an original image having an excessively large data amount is not used for display.

  Fifth, the resolution conversion process at the time of display is omitted by simultaneously converting the display color expression method and depth (data amount per pixel) at the time of resolution conversion. For example, a recent personal computer can use 16-bit (RGB each 5 bits) color expression. If the RGB 8-bit data is converted to 16 bits, not only the conversion process at the time of display is omitted, but also a reduction in data amount and an increase in display speed can be expected.

  The above is the technology for speeding up the three-dimensional display process using multi-resolution images.

  Next, the third 3D display processing speed-up technique of the present invention will be described.

  The third three-dimensional display processing speed-up technology of the present invention is a long-distance image excessive reduction projection processing technology. This far-distance image over-reduction projection processing technique is a process for enlarging / reducing the original image data according to the size of each image arranged in the virtual three-dimensional space viewed from the viewpoint in space. Applying a reduction ratio larger than the reduction ratio proportional to the distance between the viewpoint in the space determined from the relationship between the arrangement information in the virtual three-dimensional space and the viewpoint information in the space and each image, further reducing the size of each image It is processing to do. That is, in the virtual three-dimensional space, the farther away the image, the larger the reduction ratio and the excessive reduction projection processing.

  In the example of the far-distance image excessive reduction projection processing, when the enlargement / reduction processing is calculated according to the distance between each image and the viewpoint in the space, for example, instead of dividing by the depth coordinate value Z0 of the representative point, it is divided by Z02. By this process, the reduction ratio is determined in proportion to the square of the distance, so that an image far from the viewpoint is excessively reduced, and image writing is efficiently omitted by the small image omission process described later. , Display processing is speeded up.

  The fourth 3D display processing speed-up technology of the present invention is a small image omission process. In the small image omission processing, a threshold is provided for the size of each image on the two-dimensional arrangement in the image data writing processing of the display image generation processing, and the size is equal to or smaller than the threshold. For example, this is a process of omitting the writing process of the image itself. Since the projection calculation is omitted for the omitted image data, the amount of image processing can be reduced, and high-speed display is possible.

  The fifth 3D display processing speed-up technology of the present invention is a long-distance image omission processing. This long-distance image omission processing is processing for omitting writing of an image that exists farther than a certain distance from the in-space viewpoint in the virtual three-dimensional space. As an example of this long-distance image omission processing, a certain long-distance threshold is provided for the difference between the depth coordinate value of each image and the in-space viewpoint, that is, the distance from the in-space viewpoint to each image. An image whose distance from the viewpoint exceeds the long distance threshold is regarded as a long distance image, and the writing process of the image itself is omitted. Since the projection calculation is omitted for the omitted image data, the amount of image processing can be reduced, and high-speed display is possible.

  The sixth 3D display processing speed-up technology of the present invention is a short-range image omission processing. This short distance image omission processing is processing for omitting display of an image that is quite close to the in-space viewpoint. During browsing and browsing, images that move in the virtual three-dimensional space and are found to be uninteresting as a result of the search are not displayed in detail, but exist in the virtual three-dimensional space. In order to browse other images, the process proceeds further in the back direction. At that time, since the in-space viewpoint passes through the vicinity of the uninteresting image, the distance is close and displayed large. Since such an image is not interested in retrieval, it is useless to display it large, and image writing may be omitted. Therefore, as near-field image omission processing, display is omitted by regarding an image that is quite close to the in-space viewpoint as an image that has lost interest. As an example of the short distance image omission processing, a short distance threshold is provided for the distance from the in-space viewpoint to each image, and an image whose distance from the in-space viewpoint is smaller than the near distance threshold is set as a near-field image. The writing process of itself is omitted. Since image data for which writing is omitted does not require projection calculation, the amount of image processing can be reduced, and high-speed display is possible.

  The seventh three-dimensional display processing speed-up technology of the present invention is a protruding image omission processing. This protruding image omission processing is processing for omitting writing of an image that is located at a certain distance or more and protrudes from the display area in the image data writing processing in the display image generation processing. Here, as an example of the protruding image omission processing, a threshold is set for the direction of each image arranged in the virtual three-dimensional space, and an image existing in a direction within the threshold is written in the display image, and the direction outside the threshold is set. In other words, for an image that is present around a certain distance, only the frame line is written, and the writing process of the image itself is omitted. Here, as an example of the image direction, the direction of the center of the image viewed from the in-space viewpoint, the direction of the center of gravity weighted by the brightness of the image viewed from the in-space viewpoint, and a predetermined representative point can be employed. The method for determining whether or not the image direction is outside the set range includes a method for determining whether or not the two-dimensional arrangement position of the representative point of the image is outside the predetermined range of the display image. Here, in the case of normal CG clipping processing, it is normal to display an image of a portion of the image in the display. However, according to the protruding image omission processing of the present invention, an image to be omitted is an image. The whole writing is omitted. In browsing search and circulation, it is considered appropriate not to display an image in which the representative point is not visible. According to the protruding image omission processing of the present invention, the image data that is not written is omitted from the projection calculation, so that the amount of image processing can be reduced and high-speed display is possible.

  The eighth 3D display processing speed-up technology of the present invention is an efficiency process for hiding determination. In browsing search and circulation, the image closest to the viewpoint often occupies a considerably large display area on the display, so that an image existing at a long distance may be completely hidden by an image existing at a short distance. is there. In this case, the display speed can be increased by omitting the projection process of the completely hidden image.

  Next, an apparatus configuration example to which the three-dimensional display processing speed-up technology of the present invention described in the first embodiment is applied will be shown. FIG. 5 shows an apparatus configuration example to which the three-dimensional display processing speed-up technology of the present invention is applied. As shown in FIG. 5, an image input unit 10, an arrangement information input unit 20, an in-space viewpoint setting / movement processing unit 30, a projection processing unit 40, a display image generation unit 50, and a display unit 60 are provided.

  Image data is read by the image input unit 10. The arrangement information input unit 20 reads arrangement information of each image in the virtual three-dimensional space. The in-space viewpoint setting / movement processing unit 30 designates setting / movement of the in-space viewpoint based on the viewpoint position and the line-of-sight direction parameters in the virtual three-dimensional space. Based on the relationship between the arrangement information of each image in the virtual three-dimensional space and the in-space viewpoint information, the projection processing unit 40 projects the image data original form such as the parallel enlargement projection process described in the first embodiment. A process is executed to obtain a simplified two-dimensional arrangement of each image that can be seen from the viewpoint in space. The display image generation unit 50 writes each image as a two-dimensional display image based on the obtained simplified two-dimensional arrangement. The two-dimensional display image generated by the display unit 60 is displayed.

  As described above, according to the three-dimensional display processing speed-up technology of the present invention, a large amount of search target images can be displayed at high speed in the virtual three-dimensional space. Note that the three-dimensional display processing speed-up technology described above can be used in any combination.

(Embodiment 2)
The second embodiment is a three-dimensional image expression technique optimized for a large amount of image display, which is the second basic technique of the image display method of the present invention. Multi-resolution image composition display processing, important area cut-out display processing, alternative display processing at the time of omission, processing for limiting the range of viewpoint movement, highlight processing for specific images, display processing for coordinate axes, display processing with image frames, image layout An image expression technique such as expansion processing will be described. In addition, they can be used in any combination.

  The first image expression technique of the present invention is a three-dimensional image expression technique using a multi-resolution composite image.

  In the process of arranging and displaying an image in a virtual three-dimensional space, multi-resolution image data may be used for high-speed processing. Processing for generating multi-resolution image data in which a plurality of image data having different resolutions is set for each image, and selecting and displaying an image with an appropriate resolution according to the size at the time of display I do.

  A display example using a multi-resolution image simply reduced in accordance with the image size is shown in FIG. However, if a multi-resolution image reduced in accordance with the image size as shown in FIG. 4 is used, it becomes difficult to visually recognize the contents of the image data having a low resolution as the image data is reduced. Since a large amount of images are assumed to be displayed in the virtual three-dimensional space, and many images are displayed in a small size, it is difficult to recognize many images with a list.

  Therefore, the present invention provides a three-dimensional image expression technique suitable for browsing search using the following multi-resolution composite image.

  A multi-resolution composite image is a part of an original image that is important for a search user and is used to extract an important area part and an area part that expresses the contents of the image. It is kept large, the other part of the original image is omitted as a low-resolution image, and both area parts are combined into a single composite image. However, the high resolution does not exceed the resolution of the original image.

  The multi-resolution image data generation process includes dividing the original image into a plurality of partial areas, extracting only the specified partial area, and generating each image writing frame according to the size of each resolution image to be generated. In accordance with the size of each image writing frame, the size of the extracted partial area is maintained as large as possible, and writing to the corresponding position of each image writing frame, and data writing is performed on the unwritten portion of each image writing frame. A process of omitting and generating one piece of composite image data is provided.

An example of a multi-resolution composite image generated by the above processing is shown in FIG. As shown in FIG. 6, the central portion “rose” of the original image in FIG. 6 is determined as an important portion, and this region portion is largely maintained in the multi-resolution composite image. It can be seen that the multi-resolution composite image of FIG. 6 has a larger amount of visual information than the reduced image of FIG. 4 and is suitable for browsing search and browsing. It is necessary to determine which portion of the original image is the portion or the region portion that directly represents the content of the image. As an example of the determination method, first, it is possible to specify a portion to be expressed with high resolution by manual determination. Further, for example, as a process of extracting only the designated partial area, a partial area in which the differential value of the image data is larger than a set value among a plurality of partial areas can be designated as the original image. An example of processing for determining an important part by the differentiation of the pixel is shown in the flowchart of FIG.

  First, the attention area is initialized (step S701). The current attention area is set at the upper left corner of the image. Prepare a variable to record the area that gives the maximum value.

  Next, the target pixel is initialized (step S702). The current pixel of interest is set at the upper left corner of the current region of interest. A variable Sum for recording the sum of derivatives is set to zero.

  Next, differential calculation of the pixel value is executed (step S703). For the current pixel of interest, the absolute value (or sum of squares) of the difference in pixel value between the top, bottom, left and right is calculated. Calculate separately for RGB. Add these to the Sum.

  Next, the target pixel is moved. Here, the pixel of interest is the pixel adjacent to the X-axis direction (step S704). If the current pixel of interest exceeds the coordinates of the right edge of the region of interest (step S705: Y), the pixel of interest is set to the pixel at the left end one level down in the Y-axis direction of the region of interest (step S706). If the current pixel of interest exceeds the coordinates of the right edge and the lower edge of the region of interest (step S707: Y), it means that the differential calculation has been completed for all pixels of the region of interest, and the process proceeds to step S708.

  If the current pixel of interest is not the right edge of the region of interest (step S705: N), or if the current pixel of interest is not the right edge and the lower edge of the region of interest (step S707: N), the pixel of interest after the movement is again in step S703. Perform differential calculation.

  When the pixel of interest exceeds the coordinates of the lower end of the region of interest (step S707: Y), the sum of the differential values Sum is compared with the threshold value (step S708). When the sum of differentiation Sum is larger than the threshold value, the current attention area is recorded.

  Next, it is checked whether or not an unprocessed attention area remains. If it remains (step S709: Y), the attention area is moved to another area, for example, an image area adjacent to the right. In the case of the right end of the image, it moves to the image region at the left end one step down.

  In order to calculate the differential value in the next region of interest, the process returns to step S702.

  If no unprocessed attention area remains (step S709: N), the process ends.

  By the processing shown in the flowchart of FIG. 7, an image region having a differential value larger than the threshold value can be specified among the image regions.

  As described above, according to the three-dimensional image representation technology using the multi-resolution composite image of the first image representation technology of the present invention, even when displaying a low-resolution small image, the amount of information suitable for browsing search is increased. A multi-resolution composite image including many can be obtained.

  Next, the second image expression technique of the present invention is a three-dimensional image expression technique using important area cutout display processing. This important area cut-out display process is suitable for browsing search, similar to the display process using the multi-resolution composite image.

  A cut-out image is an important area for the search user in the original image for each image writing frame that matches the size of each resolution image to be generated. As a result, a part of the original image is cut out centering on the important part.

  An example of an important area cutout image generated by this processing is shown in FIG. As shown in FIG. 8, the cut-out image is cut out into a small image writing frame with the central portion “rose” of the original image as an important region. The cut-out image is a partial image of the original image, but an important part “rose” is efficiently represented, and it can be seen that the clipped image is suitable for browsing search and browsing.

  As in the case of the multi-resolution composite image generation processing, which part of the original image is an important area part or an area part that expresses the content of the image is the part of the original image. It can be determined that the partial region has the largest differential value.

  Next, the third image expression technique of the present invention is an alternative display process when image writing is omitted.

  When the small image omission processing or the long-distance image omission processing is executed in the three-dimensional display processing speed-up technology of the present invention, an alternative graphic (instead of the image to be written) is written for the entire image or image portion to be written. For example, a character string such as a white square) or an image title is displayed instead. Assuming that the substitute image to be alternatively displayed has a small amount of data, the calculation amount of the display process can be reduced, so that the display processing speed is not reduced.

  According to the image expression technique using this alternative display process, there is an effect that the arrangement of each image in the virtual three-dimensional space can be easily understood without significantly reducing the display processing speed.

  Next, the fourth image expression technique of the present invention is a process for limiting the movable range of the in-space viewpoint.

  The process of limiting the range of movement of the viewpoint in the space is that the viewpoint in the space can be moved by using a pointing device such as a mouse. This is a process of restricting to be within the range.

  In this way, by limiting the movement range of the in-space viewpoint, it is possible to prevent a display that causes confusion for the browsing search user, such as the in-space viewpoint being erroneously moved to a place where no image is arranged. .

  Next, a fifth image expression technique of the present invention is a highlight display process for a specific image. In browsing search and browsing, the display of the image in the specific region currently focused on is emphasized in an easy-to-see manner. Furthermore, it is possible to reduce the appearance of images in other areas because the degree of attention is low.

  Here, the specific image may be an image existing near the center of the display or an image near a pointer such as a mouse. These areas are regarded as an image of current attention. Further, as an example of the highlighting process, there can be mentioned a process of increasing the image brightness of the specific area and displaying the brightness gradually darker as the distance from the specific area increases. In addition, the image of the specific area is clearly displayed, and gradually blurred as the distance from the specific image is displayed.

  As described above, according to the image expression technique based on the highlight display process of the specific image, it is possible to call attention only to the image in the specific area, and it is possible to improve the search efficiency and the visibility.

  Here, an example of image blurring processing will be described with reference to the flowchart of FIG.

  In general, the blurring process is often performed by a convolution operation between a spatial filter having a spatial constant that determines a blurring degree as a parameter and image data. An example of the spatial filter is Gaussian f (X, Y) = exp (− (X 2 + Y 2) / 2σ 2) / 2πσ 2. In this case, σ is a spatial constant, and this degree of unevenness is blurred. The convolution operation between the spatial filter and the image data is calculated as follows, for example. An image to be filtered is represented by an input image I.

  First, the target pixel is initialized (step S901). The target pixel (x, y) is placed on the upper left of the image. Prepare an output image.

  Next, the sum calculation parameter is initialized (step S902). The sum calculation parameter Sum for storing the calculation sum is set to 0 and initialized. The pixel (u, v) for which the sum is obtained is set at the upper left of the image.

  Next, the product with the spatial filter is calculated (step S903). The product I (u, v) f (ux, vy) of the value of the pixel obtained by translating the summed pixel by (u, v) and the spatial filter is calculated and added to the sum calculation parameter Sum. This calculation is performed separately for RGB.

  Next, the total pixel is moved. The sum pixel (u, v) is set to the right one pixel in the X-axis direction (step S904). When the current sum pixel exceeds the coordinates of the right end of the attention area (step S905: Y), the sum pixel is set as the left end pixel one lower in the Y-axis direction of the attention area (step S906). If the current sum pixel is at the right end and the lower end (step S907: Y), the process proceeds to step S908, assuming that the convolution operation between the spatial filter and the image data has been completed.

  If the current sum pixel does not exceed the right end coordinate of the region of interest (step S905: N), or if the current sum pixel does not exceed the lower end coordinate (step S907: N), The product with the spatial filter is calculated again (step S903). When the sum pixel (u, v) exceeds the coordinates of the right end and the lower end (step S907: Y), the sum calculation parameter Sum is written as the value of the output image of the pixel of interest (x, y) (step S908). .

  Next, it is checked whether or not an unprocessed target pixel (x, y) remains, and if it remains (step S909: Y), (x, y) is moved right by one to move to the next target pixel. Move to. In the case of the right end, it moves to the left end one step down. In order to perform the sum calculation again, the process returns to step S901.

  If no unprocessed pixel of interest (x, y) remains (step S909: N), the process is terminated.

  The above is an example of the flow of blurring processing.

  Note that the spatial constant σ can be changed according to the pixel of interest, and a specific image can be highlighted.

  Next, a fifth image expression technique of the present invention is a display process of coordinate axes and the like.

  When displaying an image arranged in a virtual three-dimensional space, displaying a three-dimensional object such as coordinate axes separately from the image makes it easy for the user to grasp the virtual three-dimensional space. In particular, when a display image generation process using a simplified two-dimensional arrangement by parallel enlargement projection according to the present invention is used, this is an effective image expression technique for facilitating the understanding of a virtual three-dimensional space. In order to further emphasize the coordinate axes, it is possible to add an emphasis color to the coordinate axes. In addition, the color of the coordinate axis can be changed according to the depth in the virtual three-dimensional space, and the depth of the virtual three-dimensional space can be easily grasped by the color change of the coordinate axis.

  Next, a sixth image expression technique of the present invention is an image framed display process.

  Image framed display processing means that when displaying an image arranged in a virtual three-dimensional space, a frame having a certain width (or the size may be changed according to the distance) is attached to each image. Image representation processing. Here, in order to further emphasize each image frame, it is possible to add a color to the image frame. Furthermore, the depth of the virtual three-dimensional space can be easily grasped by changing the color of the image frame according to the depth in the virtual three-dimensional space.

Next, a seventh image expression technique of the present invention is an image arrangement expansion process.
This image layout expansion process is a process of correcting and moving the X and Y values of each image in a direction away from the center of the display in accordance with the distance from the viewpoint in space. In other words, the degree of overlapping of the images that overlap in the center direction of the display is alleviated, and they are spread out vertically and horizontally so that it is easy to see.

  Next, an apparatus configuration example to which the three-dimensional image expression technology of the present invention described in the second embodiment is applied will be described. FIG. 10 shows an example of an apparatus configuration to which the three-dimensional image expression technique of the present invention is applied. As shown in FIG. 10, an image input unit 10a, an arrangement information input unit 20, an in-space viewpoint setting / movement processing unit 30, a projection processing unit 40, a display image generation unit 50, and a display unit 60 are provided. The image input unit 10 a includes a multi-resolution image generation unit 11.

  In the second embodiment, image data is read by the image input unit 10a, and a multi-resolution image is generated by the multi-resolution image generation unit 11. Of course, it is also possible to read a set of multi-resolution images generated externally. The arrangement information input unit 20 reads the three-dimensional arrangement information of the image. The in-space viewpoint setting / movement processing unit 30 designates setting / movement of the in-space viewpoint based on the viewpoint position and the line-of-sight direction parameters in the virtual three-dimensional space. The projection processing unit 40 obtains a two-dimensional arrangement of each image seen from the viewpoint in space by a projection process based on the relationship between the arrangement information of each image in the virtual three-dimensional space and the viewpoint information in the space.

  The display image generation unit 50 selects an image with a corresponding resolution in accordance with the two-dimensional arrangement and writes it in the display image. In addition, the display image is written using various three-dimensional image expression techniques such as the multi-resolution composite image and the cut-out image described in the second embodiment. The generated display image is displayed by the display processing unit 60.

  As described above, according to the three-dimensional image representation technique of the present invention, image representation suitable for displaying a large amount of images in the virtual three-dimensional space can be performed. Note that the three-dimensional image representation techniques described above can be used in any combination.

(Embodiment 3)
The third basic technique is a three-dimensional space expression technique using a background image.

  The three-dimensional space expression technique using the background image is a technique that uses a background image to indicate a background object that serves as a guide for grasping the position in the space with respect to the virtual three-dimensional space in which each image is arranged. .

  Even if a large number of images are arranged and displayed in a virtual three-dimensional space, there is a drawback that it is difficult to grasp as a three-dimensional space, for example, a white background or a black background without any background objects. In order to solve this problem, a familiar scene such as one's office or room is photographed and used as a background when displayed. Since it is possible to grasp the arrangement of each image using a familiar scene as a clue, it is easy to understand what is located. Browsing search and browsing can be done just like putting everything around your room. From the viewpoint of making it easier to remember, it is desirable that the user decides the arrangement himself.

  As a method for obtaining a background image as a background, a technique is known in which a scene is photographed several times while rotating a camera, and a panoramic image is created by combining these images. By cutting out a part from the panorama image by a mouse operation or the like and displaying it on the display, an effect of looking around the scene can be obtained.

  It is also possible to set a specific function to correspond to a specific location on the panoramic image. For example, when a location on the panoramic image is clicked, a specific operation (display of another panoramic image is performed). (Execution and execution of commands such as mail transmission).

  The following processing is further disclosed as a three-dimensional expression technique using a background image as a background.

  First, the user arranges images in a virtual three-dimensional space. This enhances the ease of remembering the position where the image is arranged, and makes it easy to associate the actual image as the background with each image.

  Second, the scene is three-dimensionalized based on the video of the scene. A known technique may be used as the technique for displaying the object to be three-dimensionally from the actual photographed image. With this process, the panorama image creation process is automated.

  Third, a scene to be photographed is limited to a specific environment (for example, an office), and a dictionary storing information about objects existing in the specific environment is used. By limiting to the specific environment, the background objects appearing in the scene are limited, so that it is possible to prepare a dictionary having information on objects existing in the specific environment. In the dictionary, a number of live-action photographs or CG data are prepared for objects existing in a specific environment. When there is a movement of the viewpoint in space, or when a large image is required for narrowing down the browsing search, the image data in the dictionary can be used for image display processing.

  Fourth, when a panoramic image is generated from a real image, the real image is pasted to a known shape such as a sphere or a cylindrical surface. By using the known shape in this way, the process of joining the photographed images is simplified, and panorama image generation is simplified.

  Next, an apparatus configuration example to which the three-dimensional space expression technology using the background image of the present invention described in the third embodiment is applied will be described. FIG. 11 shows an apparatus configuration example to which the three-dimensional space expression technology using the background image of the present invention is applied. As shown in FIG. 11, an image input unit 10, an arrangement information input unit 20, a background image input unit 70, an in-space viewpoint setting / movement processing unit 30, a projection processing unit 40, a display image generation unit 50, and a display unit 60 are provided. ing.

  Image data is read by the image input unit 10. The arrangement information input unit 20 reads arrangement information of each image in the virtual three-dimensional space. The background image input unit 70 reads image data of a background image that virtually exists as a background in the virtual three-dimensional space. The in-space viewpoint setting / movement processing unit 30 designates setting / movement of the in-space viewpoint based on the viewpoint position and the line-of-sight direction parameters in the virtual three-dimensional space. The projection processing unit 40 performs a two-dimensional view of each image and background image viewed from the viewpoint in space by a projection process based on the relationship between the arrangement information of each image and background image in the virtual three-dimensional space and the viewpoint information in space. Ask for placement. The display image generation unit 50 processes the read image data and background image data according to the two-dimensional arrangement of each image and background image to generate a two-dimensional display image. The two-dimensional display image generated by the display unit 60 is displayed.

  As described above, according to the three-dimensional space expression technique using the background image of the present invention, the position of each image in the virtual three-dimensional space can be expressed in an easily understandable manner. Note that the three-dimensional image representation techniques described above can be used in any combination.

(Embodiment 4)
The fourth basic technique is a movement parameter optimization processing technique. This technique optimizes movement parameters based on optical flow to suppress eye fatigue.

  When an image is displayed on a display, display contents such as zoom and scroll by a user's operation are often moved or changed. At this time, if the moving parameters such as the moving speed of the display contents are inappropriate, the eyes are tired. The present technology optimizes movement parameters to suppress eye fatigue. This technology can be used independently of the technology for speeding up the three-dimensional display processing of a large number of images, the three-dimensional image representation technology, and the three-dimensional space representation technology using the background image. Applicable to image display.

  Zooming and scrolling can be expressed as movement of pixel values at each pixel on the display. This is shown in FIG. Other movement methods in the three-dimensional display can be basically expressed in the form of movement of pixel values in each pixel as described above. It is considered to suppress eye fatigue by correcting the method of moving the pixel value.

  First, we model that eyestrain can be handled by a computer. A general display device such as a computer rewrites a display frame one after another in both an interlace method and a non-interlace method. The display content is not continuously changed without being rewritten, and the display content is updated at regular time intervals. Now, as shown in FIG. 13, when attention is paid to a certain point in the display content, the display moves on the display as shown in FIG.

  The human eye recognizes a three-dimensional object by projecting it onto a two-dimensional retina. A moving object causes a smooth change in the brightness distribution on the retina. From the ratio between the temporal change and the spatial change of the brightness distribution, the movement of the projected image on the projection plane (hereinafter referred to as optical flow) can be obtained. FIG. 14 shows the concept of optical flow. Assuming that one image is bright on the left side and dark on the right side, the spatial change Ix of the lightness distribution is a negative number. On the other hand, the temporal change It of the lightness distribution and the movement amount d per unit time are positive numbers, and the relationship of −Ix = It / d is established between them. That is, the triangle shown in FIG. 13 is similar. Here, d is defined as an optical flow. The calculation of the optical flow d is performed when the brightness distribution can be regarded as having a linear relationship within the range of the movement width. Therefore, the movement width needs to be sufficiently small. As the movement width increases, the error from the actual movement increases. However, in actuality, it is considered that the process of blurring the projected image is performed before obtaining the change in brightness so as not to be affected by the fine change in the brightness distribution.

  It is thought that the human eye is tracking an object by generating eye movement based on this optical flow. Thanks to this function, the characters can be read even if the book is moved somewhat. In this section, we will focus on only the lightness (average value or maximum value of RGB) of the pixel values, and explain it using a model that obtains optical flow, but use a model that obtains and averages RGB individually. Also good. Although various improvements have been made to the optical flow calculation method, the present invention is not limited to the optical flow calculation method itself.

  On the other hand, when two objects having similar shapes are separated from each other so that they appear alternately, it is perceived that one object is moving. For example, it seems that the light is moving by the blinking of the railroad crossing lamp. In this case, the optical flow is not obtained (does not exist), but it is considered that the movement is perceived by taking the correspondence of the parts having similar shapes. It can be said that the human eye perceives movement in these two ways. The same applies to the case of displaying on the display. If the movement width in one drawing is sufficiently small, it is considered that the motion is perceived by obtaining the optical flow. As the movement width increases, the optical flow becomes difficult to obtain, and the value calculated from the change in brightness shifts from the actual movement. The upper limit of the movement width for which optical flow is required depends on the complexity of the display target, but is generally about the size of the basic unit of the display content (one character for a character image, line width or constituent figure for a picture). Can be considered. When the calculated motion is different from the actual motion, it is necessary to correct the position on the eye by matching the shape after the eye movement in order to continue visual tracking of the object. In the movement parameter optimization technique of the present invention, it is considered that this visual position correction causes eye fatigue.

  In accordance with the above concept, the degree of eye fatigue is defined by (Equation 1).

[Equation 1]
Eye fatigue = Number of times of drawing * Optical flow error at the point of interest However, since the point of interest is not always required, if the point of interest is not uniquely determined, the optical flow average near the center of the display or the entire display Approximate by error with value. It is also effective to use an average value of optical flows in a region where the moving speed is somewhat high. If the movement width is large and an optical flow cannot be obtained, it can be considered that the error of the optical flow takes a set maximum value. Accordingly, the relationship between the error in the optical flow and the movement width by one drawing is expressed as shown in FIG.

  As shown in FIG. 13, if one point of display content is fixed, the number of times of drawing per unit time at that point is n = 1 / s (s is the time of drawing interval), and the moving speed is v, 1 The movement width d, which is an optical flow in one drawing, is expressed by d = v / n. As shown in FIG. 15, since the error of the movement width d is considered to be a function proportional to d2 when the movement width d is small, the degree of eye fatigue per unit time is proportional to nd2. Since this is equal to v2 / n = vd, it is necessary to increase the number of times of drawing or decrease the moving speed or moving width if the moving speed is constant in order to reduce the eye fatigue. On the other hand, when the movement width d is considerably large, the optical flow error is substantially constant, so that the eye fatigue becomes smaller as the number of times of drawing decreases.

  On the other hand, the area that can be newly presented per unit time (hereinafter, the amount of information) is proportional to the moving speed v. This is equal to nd. Therefore, the amount of information that can be presented decreases if the number of times of drawing or the movement width is reduced. Since the degree of eye fatigue and the amount of information have such a trade-off relationship, an evaluation function shown in the following (Equation 2) is introduced in order to evaluate the optimum movement parameter. If the movement speed and the single movement width can be changed when moving the display object, the optimum movement can be determined by setting the evaluation function to a value that minimizes the evaluation function.

[Equation 2]
Inappropriate display = fatigue of eyes−amount of information The simple properties of “inappropriate display” defined by the above equation (2) are summarized.

  The first property is that when the moving speed v is constant, the amount of information is constant, so that an appropriate display reduces the degree of eye fatigue. When the movement width d is small, the movement width d is made smaller, and when it is larger, the movement width d is made larger.

  The second property is that when the number of times of drawing n is constant, if the moving width d is small, the nd2-nd form is minimized, so that the optimal moving width d can be obtained. When the movement width d is large, the larger the better.

  In addition to adjusting the movement speed v and the movement width d, which are movement parameters, it is also possible to change the display content itself so as to perform optimal display without eye strain. In this case, an evaluation function shown in the following (Equation 3) is introduced so as not to deviate so much from the original display content, and the display content is changed so as to minimize it.

[Equation 3]
Inappropriate display = difference from original display content + eye fatigue level−amount of information As a measure for the difference from the original display content, for example, a square average of brightness difference for each pixel is used. The evaluation function of (Equation 3) can also be used when only the display content is changed without changing the moving speed v and the moving width d.

  An example of a process for changing the display content itself and performing optimal display without eye strain will be described with reference to the flowchart of FIG.

  First, variables for storing display contents and movement parameters are initialized (step S1601). An area for storing the previous and current display contents and movement parameters at each point is prepared.

  Next, the movement parameter is read (step S1602). If the movement parameter at each point is designated, the designated movement parameter is read. If not specified, the previous movement parameter is set.

  Next, the display content is read (step S1603). If the current display content is designated, the designated display content is read. If not specified, the display contents are moved according to the previous display contents and movement parameters.

  Next, an optical flow is calculated (step S1604). The optical flow is calculated from the previous display content and the current display content. If necessary, the image is blurred as preprocessing.

  Next, the value of the evaluation function indicating the inappropriateness of display shown in (Equation 3) is calculated from the calculated optical flow of the evaluation function and the display parameters (step S1605).

  Next, the value of the evaluation function is evaluated. When the value of the evaluation function is sufficiently small, for example, when the value is equal to or less than the set threshold value (step S1606: Y), the display content after the change to be processed this time is displayed on the display, assuming that the display is not inappropriate. The process proceeds to step S1609.

  If not (step S1606: N), the read movement parameter is changed (step S1607). If the movement parameter can be changed, the movement parameter of each point is slightly changed. For example, the amount of change is increased or decreased by a predetermined amount.

  Next, the display content is changed according to the changed movement parameter (step S1608). When the movement parameter is changed, the display content is moved accordingly. When the display content can be changed, the display content is moved after the brightness of each pixel is changed slightly. The change is increased or decreased by a predetermined amount, for example. The optical flow is calculated again (step S1604).

  When the display process is to be continued (step S1609: Y), the process returns to step S1602, the processes from step S1602 to step S1608 are repeated, and the optical flow is recalculated. If the display process is not continued (step S1609: N), the process ends.

  Since the process shown in FIG. 16 takes a considerable amount of time, as a simple process, the processing method shown in the flowchart of FIG. 17 described below in which only the display content is slightly modified without changing the movement parameter is also possible.

  First, variables for storing display contents and movement parameters are initialized (step S1701). An area for storing the previous and current display contents and movement parameters at each point is prepared.

  Next, the movement parameter is read (step S1702). If the movement parameter at each point is designated, the designated movement parameter is read. If not specified, the previous movement parameter is set.

  Next, the display content is read (step S1703). If the current display content is designated, the designated display content is read. If not specified, the display contents are moved according to the previous display contents and movement parameters.

  Next, display contents are synthesized (step S1704). The display content is synthesized such that the movement width of each point of the movement parameter is an optical flow. For example, instead of the process of obtaining the optical flow from the lightness distribution, the process can be realized by changing from the spatial conversion of the optical flow and the lightness distribution to the process of obtaining the time of the lightness distribution.

  Next, the display content is corrected (step S1705). Based on the display content synthesized in step S1704, the current display content is corrected. For example, the average of both is used. The corrected display content is displayed on the display.

  If the display is to be continued (step S1706: Y), the process returns to step S1702, the processing from step S1702 to step S1705 is repeated, and the optical flow is recalculated. If the display is not continued (step S1706: N), the process ends.

  The above is the flow of processing for optimal display without eye fatigue.

  Here, before calculating the optical flow, it is necessary to determine how much the image is to be blurred. The process of blurring the image can be realized by a convolution operation with a spatial filter having a spatial constant that determines the degree of extrapolation as a parameter.

  An example of the spatial filter is Gaussian. The spatial constant of the filter that blurs the image can be determined as follows, for example.

  The first is a method using designated values. It is possible to adjust by the size of the display image.

  The second is a method in which the value of the basic unit (for example, the height of the character) in the image is used when the structure of the display object is known like a character image. If the character height is not specified, the character height is estimated based on image processing. The estimation of the character height can be executed by, for example, adding a lightness in the horizontal direction to create a one-dimensional histogram and obtaining the period of the histogram.

  In the above modeling, individual differences were not taken into account, but it was possible to cope with individual differences by changing the weight applied to each item of the evaluation function (difference from the original display content, eye fatigue, amount of information). can do. For example, this is performed by the processing shown in the flowchart of FIG.

  First, some samples whose basic units in the image are known are presented to the user (step S1801).

  Next, the user adjusts the movement parameter and sets it to the most readable value (step S1802). As a method for adjusting and setting the movement parameter, a method is also possible in which a reading test (or reading game) is performed with a number of movement parameters serving as samples, and the movement parameter is determined from the correct answer rate. For example, it can be a parameter whose correct answer rate starts to deteriorate.

  Next, each term of the evaluation function is individually calculated for each movement parameter, and the weight of each term is adjusted so that the sum is minimized with the movement parameter value set by the user (step S1803).

  With the above processing, the weight applied to each term of the evaluation function can be changed in consideration of individual differences.

  Next, an apparatus configuration example to which the movement parameter optimization processing technology of the present invention described in the fourth embodiment is applied will be described. FIG. 19 shows an apparatus configuration example to which the three-dimensional display processing speed-up technology of the present invention is applied. As shown in FIG. 19, an image input unit 10, a movement parameter input unit 80, a movement parameter / image data adjustment process 90, a display image generation unit 50, and a display unit 60 are provided.

  Image data is read by the image input unit 10. The movement parameter input unit 80 reads a movement parameter that specifies an image movement method. The movement parameter / image data adjustment processing 90 reduces the viewpoint movement error, which is the difference between the optical flow and the actual movement of each pixel of each image, based on the read image data and the movement parameter, and temporally. At least one of the movement parameter and the image data is adjusted so as to increase the movement amount of the image between adjacent frames. Only the movement parameter can be adjusted, only the image data can be adjusted, and both the movement parameter and the image data can be adjusted. The display image generation unit 50 processes the read image data according to the adjusted movement parameter / image data to generate a two-dimensional display image. The two-dimensional display image generated by the display unit 60 is displayed.

  Next, an example in which the movement parameter optimization processing technique of the present invention is applied to image display of an image arranged in a three-dimensional virtual space will be described.

  FIG. 20 shows an example of the apparatus configuration when the moving parameter optimization processing technique of the present invention is applied to the image display of an image arranged in the three-dimensional virtual space of the present invention. As shown in FIG. 20, the image input unit 10, the arrangement information input unit 20, the movement parameter calculation unit 81, the in-space viewpoint setting / movement processing unit 30, the movement parameter / image data adjustment processing 90, the projection processing unit 40, the display image A generation unit 50 and a display unit 60 are provided.

  Image data is read by the image input unit 10. The arrangement information input unit 20 reads arrangement information of each image in the virtual three-dimensional space. The in-space viewpoint setting / movement processing unit 30 designates setting / movement of the in-space viewpoint based on the viewpoint position and the line-of-sight direction parameters in the virtual three-dimensional space. The movement parameter calculation unit 81 calculates a movement parameter that specifies an image movement method. The movement parameter / image data adjustment processing 90 reduces the viewpoint movement error, which is the difference between the optical flow and the actual movement of each pixel of each image, based on the read image data and the movement parameter, and temporally. At least one of the movement parameter and the image data is adjusted so as to increase the movement amount of the image between adjacent frames. Only the movement parameter can be adjusted, only the image data can be adjusted, and both the movement parameter and the image data can be adjusted. Each of the movement parameters and image data adjusted by the projection processing unit 40, and each of the images viewed from the in-space viewpoint by the projection processing based on the relationship between the arrangement information of each image in the virtual three-dimensional space and the in-space viewpoint information. A two-dimensional arrangement of images is obtained. The display image generation unit 50 processes the read image data according to the two-dimensional arrangement of each image and the adjusted movement parameter / image data to generate a two-dimensional display image. The two-dimensional display image generated by the display unit 60 is displayed.

  As described above, according to the movement parameter optimization processing technology of the present invention, it is possible to optimize the movement parameters based on the optical flow to suppress eyestrain of the user who performs browsing search and browsing.

(Embodiment 5)
The image display method and apparatus of the present invention can be constructed using various computers by recording and providing a program describing processing steps for realizing the processing described in the above-described embodiment on a computer-readable recording medium. Can do. As shown in the example of the recording medium shown in FIG. 21, the recording medium on which the program having the processing steps for realizing the image display method and apparatus of the present invention is recorded is a portable recording medium such as a CD-ROM 102 or a flexible disk 103. Not only the medium 101 but also any of the recording medium 100 in the recording apparatus on the network and the recording medium 105 such as a computer hard disk or RAM, and when the program is executed, the program is loaded on the computer 104, Runs on main memory.

  In addition, the above-described three-dimensional display processing speed-up technology of the present invention, three-dimensional image representation technology, three-dimensional space representation technology using a background image, and movement parameter optimization processing technology are technologies for improving image display processing. Each can be applied independently, and can be used in any combination.

The following items are further disclosed with respect to the matters described in the above description and claims.
(1) The image display method according to claim 1, wherein the arrangement information of each image is set so that the orientation of all the images is orthogonal to the display screen and all the images face the front with respect to the display screen.
(2) The image display method according to claim 1, wherein the arrangement information of each image is set so that the pixel directions of all the images are parallel to the pixel directions of the display screen.
(3) In the enlargement / reduction process corresponding to the size of each image viewed from the in-space viewpoint in the parallel enlargement projection process, arrangement information of each image in the virtual three-dimensional space and the in-space viewpoint information Apply a reduction ratio larger than the reduction ratio proportional to the distance between the in-space viewpoint and each image determined from the relationship, and the farther away, the larger the reduction ratio and the excessive reduction projection processing The image display method according to (2), wherein a distance image excessive reduction projection process is added.
(4) In the writing process of the image data according to the simplified two-dimensional arrangement of each image in the display image generation process,
The image display method according to claim 1, wherein if the direction of each image indicated by the simplified two-dimensional arrangement is out of a set value range, the image writing process is omitted.
(5) A process for setting the brightness distribution on the display image is provided.
In the display image generation process, in the image data writing process according to the simplified two-dimensional arrangement of each image, the image writing process is performed with the image brightness according to the brightness distribution on the set display image. The image display method according to claim 1 to be performed. (6) A process for setting the blur distribution of the image on the display image is provided.
In the display image generation process, in the image data writing process according to the simplified two-dimensional arrangement of each image, the image writing process accompanied with the blurring process of the image according to the set blur distribution on the display image The image display method according to claim 1.
(7) A process for setting the color distribution on the display image is provided.
In the writing process of the image data according to the simplified two-dimensional arrangement of each image in the display image generation process, a process of adding a color to the frame of the image according to the set color distribution on the display image is accompanied. The image display method according to claim 1, wherein an image writing process is performed.

The flowchart which shows the basic flow of the parallel expansion projection process of Embodiment 1 of this invention. The flowchart which shows the example of the projection calculation process to the display of a rectangular image using the parallel expansion projection of Embodiment 1 of this invention. The figure which shows the example of the display image using the simplified two-dimensional arrangement | positioning by parallel expansion projection processing of Embodiment 1 of this invention, and the example of the two-dimensional display image by center projection processing of a prior art The figure which shows the example of a production | generation of the display image using a multi-resolution image The figure which shows the example of an apparatus structure to which the three-dimensional display process speed-up technique of Embodiment 1 of this invention is applied. The figure which shows the example of the multi-resolution composite image produced | generated by the process of this invention The flowchart which shows the flow of the process which judges an important part by differentiation in the multi-resolution image generation process of Embodiment 2 of this invention. The figure which shows the example of the cut-out image produced | generated by the process of this invention The flowchart which shows the flow of the image blurring process of Embodiment 2 of this invention. The figure which shows the apparatus structural example to which the three-dimensional image representation technique of Embodiment 2 of this invention is applied. The figure which shows the apparatus structural example to which the three-dimensional space expression technique using the background image using the background image of Embodiment 3 of this invention is applied. Diagram showing how pixel values move at each pixel on the display by zooming and scrolling The figure which showed the locus drawn on the display when paying attention to one point in the display contents Diagram showing the concept of optical flow The figure explaining the relation between the error of the optical flow and the movement width by one drawing Flowchart showing the flow of processing to change the display content itself and perform optimal display without eye strain Flowchart showing the flow of processing when only the display content is slightly modified without changing the movement parameter Flow chart showing the flow of processing for setting weights to be added to each term of the evaluation function considering individual differences The figure which shows the example of an apparatus structure to which the optimization processing technique of the movement parameter of Embodiment 4 of this invention is applied. The figure which shows the apparatus structural example at the time of applying the optimization processing technique of the movement parameter of this invention to the image display of virtual three-dimensional space The figure which shows the example of the recording medium which recorded the computer processing program which implement | achieves the image display processing method of this invention The figure which shows a mode that the three-dimensional object which exists in the three-dimensional space by the center projection of a prior art is projected on a display.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image input part 20 Arrangement information input part 30 In-space viewpoint setting / movement processing part 40 Projection processing part 50 Display image generation processing part 60 Display part 70 Background image input part 80 Movement parameter input part 81 Movement parameter calculation part 90 Movement parameter Image data adjustment processing unit 100 Recording medium such as hard disk at line destination 101 Portable recording medium such as CD-ROM and flexible disk 102 CD-ROM
103 flexible disk 104 computer 105 recording medium such as RAM / hard disk on computer

Claims (5)

An image display method for displaying images while moving them,
Processing to read image data,
Processing to read the movement parameters that specify how to move the image,
Based on the read image data and movement parameters, a viewpoint movement error, which is a difference between an optical flow determined between display images of temporally adjacent frames and an actual movement of each pixel of each image, and A movement parameter / image data adjustment process for adjusting at least one of the movement parameter and the image data so as to increase the amount of movement of the image between temporally adjacent frames;
Display image generation processing for generating a display image by processing the read image data according to the adjusted movement parameter / image data;
An image display method comprising a process of displaying the generated display image on a display device.   The movement parameter / image data adjustment processing uses the weighted viewpoint movement error and a weighted negative amount of the moving amount of the image as an evaluation function, and minimizes the evaluation function, The image display method according to claim 1, wherein the movement parameter and the image data are adjusted. A weight adjustment process for setting / adjusting the weight of the evaluation function;
While the weight adjustment process presents an image for adjustment test to the user, the viewpoint in the space is moved by the viewpoint setting / movement process in the space, the viewpoint movement error is small for the user, and the amount of image movement The image display method according to claim 2, wherein the user himself / herself sets an adjustment weight for the movement parameter and a weight for the image data adjustment so as to increase the value. An image display device that displays an image while moving it,
An image input unit for reading image data;
A movement parameter input section for reading movement parameters for specifying the movement method of the image;
Based on the read image data and movement parameters, a viewpoint movement error, which is a difference between an optical flow determined between display images of temporally adjacent frames and an actual movement of each pixel of each image, and A movement parameter / image data adjustment processing unit that adjusts at least one of the movement parameter or the image data so as to increase the amount of movement of the image between frames that are temporally adjacent;
A display image generation processing unit that processes the read image data to generate a display image according to the adjusted movement parameter / image data;
An image display device comprising a display unit for displaying the generated display image on a display device. A computer-readable storage medium storing a processing program for realizing an image display method for moving and displaying an image,
Processing steps to read image data;
A processing step for reading a movement parameter for specifying an image moving method;
Based on the read image data and movement parameters, a viewpoint movement error, which is a difference between an optical flow determined between display images of temporally adjacent frames and an actual movement of each pixel of each image, and A movement parameter / image data adjustment processing step for adjusting at least one of the movement parameter and the image data so as to increase an amount of movement of the image between temporally adjacent frames;
A display image generation processing step of generating a display image by processing the read image data according to the movement parameter / image data adjustment processing;
A computer-readable recording medium having recorded thereon a processing program for causing a computer to execute a processing step of displaying the generated display image on a display device.

Images