JPH11122638A - Image processor, image processing method, and computer-readable information recording medium recorded with image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor capable of expressing clearly a photographing state, and to provide a computer-readable information recording medium in which an image processing program is recorded. SOLUTION: Image data photographed by a digital camera 10 are stored in a memory card 24 with data detected by a GPS receiver 12, an azimuth sensor 14 and an altitude sensor 16, and the data are registered in a storage device 32c of a server 22 via a communication equipment 20 or a card reader 26. The storage device 32a stores 3-dimension map data and the data are read by a display circuit 30 through the retrieval, and image data of a photographing position corresponding to a position of a stereoscopic image by the map data are synthesized on an image panel formed on a stereoscopic image while keeping information of position, azimuth and altitude and the synthesized image is displayed on a display device 36.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus, an image processing method, a computer-readable information recording medium on which an image processing program is recorded, an image processing system, and a recording medium on which an image display program is recorded. An image processing apparatus, an image processing method, and an image processing program that can be applied to an electronic album system for electronically filing an image input by a camera, a video camera, or the like, and then reproducing the image while performing a search according to the purpose are recorded. The present invention relates to a computer-readable information recording medium image processing system and a recording medium on which an image display program is recorded.

[0002]

2. Description of the Related Art With the spread of electronic cameras and photo CDs in recent years, systems for storing and retrieving digitized image data have been developed. For example, a "still image display method and apparatus" described in JP-A-7-182366
Then, images taken with an electronic camera etc. were reduced, displayed continuously like a silver halide photo negative on a part of the display, the image was selected with a mouse etc., and displayed on the same display An electronic album device arranged and edited on an album has been proposed.

An electronic camera and a GPS (Global Positioning System) have been proposed as a system for facilitating recording of a shooting location.
g System: Global Positioning System) A combination with a receiver has been developed. For example, JP-A-6-26
Japanese Patent Publication No. 1244 discloses a "camera with a GPS function" for printing position data acquired by a GPS receiver on a photograph.

Further, taking advantage of the fact that the position at the time of photographing can be acquired, the photographing position can be displayed on a map. For example, in an "electronic map apparatus" described in Japanese Patent Application Laid-Open No. 7-286854, an image captured by an electronic camera is recorded with position information added thereto, and a planar map is displayed at the time of reproduction to record a captured image. An electronic map device has been proposed which can display a recorded image at a position by marking the mark on the mark and selecting the mark later.

An image input by an electronic camera such as a digital camera capable of electronically recording an image is given an index such as a file name and a file number as a name for specifying the image, and is provided to a recording device of a computer. Often saved. Also, in an electronic album that electronically realizes the equivalent of an album for organizing and storing silver halide photos on a computer, images are selected by index and the recorded images and comments on the images are displayed one by one. Or display a virtual album mount on the computer screen and arrange the images selected from the image list on the album mount so that the album images are turned over and the stored images are played back and viewed .

[0006]

However, in the electronic album as described above, for example, where a photograph image such as a photograph of a mountain scenery which has little information for identifying a subject is photographed. Thing or
In addition, it is necessary for the user to memorize clearly where the scenery is viewed from, or to record and store the shooting location in a memo. Otherwise,
This is because when the image is viewed again after a lapse of time, it is not possible to know where the image was taken.

However, in general, human memory is limited, and even if there is a memo that records the location, etc.,
It is necessary to estimate the geographical position of the image from the contents of the memo, and to view the image based on the estimation, and it may be necessary to refer to the map separately. In addition, if there are multiple images taken at different locations, even if each location can be identified, auxiliary information that expresses the positional relationship such as a map is still used for the relative positional relationship of those images. Required.

As such a map, it is conceivable to display a two-dimensional planar map and an image on the same screen. However, since the planar map is a view of the terrain viewed from directly above and has a different viewing direction from the photographed image, it is difficult to appropriately represent the actual photographing situation. Therefore, there arises a problem that the photographing direction of the photograph must be compared with the plane map, and the two must be linked and considered.

The present invention solves the above-mentioned drawbacks of the prior art, and enables an image processing apparatus, an image processing method, and a computer readable recording of an image processing program capable of clearly expressing a shooting situation such as a shooting direction. It is an object to provide an information recording medium.

[0010]

According to the present invention, there is provided an image processing apparatus for generating a three-dimensional image according to three-dimensional data and displaying the three-dimensional image on a display device. Input means for inputting first data including image data to be captured and shooting data indicating the shooting position and shooting direction of the image data; storage means for storing three-dimensional data and first data; Processing means for displaying a three-dimensional image to be displayed on the display device, the processing means reads out image data at a shooting position corresponding to the three-dimensional image displayed on the display device from the storage means, and generates an image represented by the image data according to the shooting data. It is characterized by being synthesized on a stereoscopic image and displayed on a display device.

Further, according to the present invention, in an image processing method for creating a three-dimensional image according to three-dimensional data and displaying the three-dimensional image on a display screen of a display device, the method comprises the steps of: A first display step of reading and displaying a stereoscopic image on a display screen, a moving step of moving a viewpoint position in the stereoscopic image, selecting image data to be displayed on the stereoscopic image, and displaying an image represented by the image data on the display screen. A second display step of synthesizing and displaying on the displayed stereoscopic image, wherein the image data is obtained at a photographing position corresponding to the stereoscopic image, and position data indicating the photographing position is added thereto. Is characterized in that a stereoscopic image corresponding to the viewpoint position and an image corresponding to the shooting position are displayed on the display screen in association with each other in accordance with a moving operation for moving the viewpoint.

Further, according to the present invention, a three-dimensional image corresponding to the three-dimensional data is created, and the three-dimensional image is displayed on a display screen of a display device. A computer-readable information recording medium storing an image processing program for combining an image represented by image data provided with position data indicating a three-dimensional image with a stereoscopic image and displaying the image on a display screen; , A first display procedure for displaying a stereoscopic image on a display screen, a moving procedure for moving a viewpoint position in the stereoscopic image, and selecting image data to be displayed on the stereoscopic image, and displaying an image represented by the image data on the display screen. And a second display procedure of synthesizing and displaying the stereoscopic image displayed on the stereoscopic image. The second display procedure includes the steps of: The image processing program recorded computer-readable information recording medium and displaying the image corresponding to the stereoscopic image with the shooting position according to the location on the display screen in correspondence is provided.

By adopting such a configuration, it is possible to display a map corresponding to a shooting direction on a three-dimensional map corresponding to a spatial position where the image is taken, and arrange and display an image in that direction. , Can clearly express the positional relationship, and by moving the viewpoint, it is possible to sequentially see images at different positions and directions, and to clearly reproduce the same scene as the real scene on a virtual 3D map Will be able to

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, a preferred embodiment of the present invention will be described with reference to the drawings. Referring to FIG. 1, there is shown an embodiment of an electronic album system 1 to which the present invention is applied. As shown in the figure, the electronic album system 1 is an image display system that arranges a photographic image photographed by a camera or the like on a three-dimensional map based on the photographing situation and displays the photograph image on an image display device such as a display. In this system 1, a three-dimensional map can be displayed by moving on a three-dimensional map, and the display image of the three-dimensional map changes according to the viewpoint position by moving the viewpoint for moving the viewpoint for viewing the map. At the same time, when the camera is moved to a position near the shooting position of the camera, the image shot at that position is pasted on the surface of the image panel in which the shooting position and the shooting direction are held, and these are displayed together with the three-dimensional map.

More specifically, the electronic album system 1 according to the present embodiment captures an object scene and outputs an image signal representing an image of the object scene, and is provided near the digital camera 10 and represents the position thereof. A GPS receiver 12 that outputs position coordinate data, an azimuth sensor 14 that is provided in the digital camera 10 and outputs azimuth data representing the direction as the attitude at the time of shooting, and altitude data that indicates the altitude of the digital camera 10 at the time of shooting. Output specific altitude sensor 16
The processing and processing of various information sent from these to the input circuit 18 is recorded on the memory card 24, the recorded information is read out, and transferred to the server 22 via the communication device 20.

The digital camera 10 converts a scene image converted into an electric signal by a solid-state image pickup device such as a CCD into a digital value, further performs a compression encoding process, and converts the processed digital image data into a semiconductor or the like. This is an imaging and image input device for recording on a memory card 24 composed of the above memory elements. The camera 10 records various information sent from the input circuit 18 via the connection line 100, the shooting date and the time together with the image data on the memory card 24, and the data recorded on the memory card 24. And outputs it to output 100. The memory card 24 is a storage device configured to be detachable from the camera 10. For example, the card 24 is removed from the camera 10, and is mounted on a card reader 26 connected to the server 22. May be read out by the card reader 26 and taken into the server 22 directly.

The GPS receiver 12 is a position positioning device that receives radio waves transmitted from a plurality of GPS satellites by an antenna and generates position information indicating a current position by using a time difference between the antennas. The GPS receiver 12 generates coordinate data indicated by latitude and longitude, and outputs these to the output 102 as position information indicating the current position of the camera 10.

The orientation sensor 14 is advantageously a digital camera
This is a detection device that is provided inside or in the housing of the camera 10 and moves together with the camera 10 to detect the attitude of the camera 10.
The azimuth sensor 14 of the present embodiment detects the direction of terrestrial magnetism, generates azimuth data indicating the shooting direction of the camera 10 based on the detected azimuth, and outputs the generated azimuth data to the output 104. Further, the azimuth sensor 14 may include a function of detecting the vertical direction of the camera, in which case, the azimuth sensor 14 includes an angle sensor that detects the angle of the camera 10 with respect to the horizontal, the detected angle data together with the azimuth data May be output.

The altitude sensor 16 is a detection device that detects the current altitude and generates altitude data representing the altitude.
The altitude sensor 16 in this embodiment includes a detection unit that detects a change in atmospheric pressure, converts the detected atmospheric pressure into altitude, and outputs altitude data representing the altitude to an output 106.

These outputs 100, 102, 104 and 106 are connected to an input circuit 18, which is connected to an image data sent from the camera 10 and various data (positional information) corresponding to the situation at the time of shooting at which the image data was obtained. , Azimuth data, and altitude data), and outputs the information to an appropriate output destination.

The input circuit 18 has a function of outputting data input to the inputs 100 to 106 to an output 108. In this case, the image signal captured by the camera 10 and various data at that time can be collected and transferred to the communication device 20 almost in real time. The input circuit 18 has a function of outputting various data input to the inputs 102 to 106 to the output 100. In this case, the camera 10 includes image data obtained by shooting and input generated and generated at the time of shooting.
The various data input to 100 are recorded on the memory card 24 in association with each other. The information recorded in the memory card 24 in this manner is read out by loading the memory card 24 into the card reader 26 and transferred to the server 22 or output from the camera 10 to the input circuit 18 again. Communication device
Transfer to server 22 via 20.

The GPS receiver 12 and the azimuth sensor 1
4.Altitude sensor 16 and input circuit 18
0 may be built in, and may be moved with a digital camera, for example, in a vehicle such as an automobile,
They may be installed separately.

The communication device 20 is a device for transmitting information input to the input 108 to the server 22 via the transmission line 110.
The communication device 20 in the present embodiment modulates image data and various information transmitted from the input circuit 18 and transmits the modulated data to the server 22, for example, using a wired or wireless telephone line as a transmission line. The communication device 20 stores an address of the server 22, a user name for specifying a person or device accessing the server 22, a password required for authentication at the time of access, and the like. According to the transfer procedure, the information input to the input 108 is transmitted on the transmission line of the output 110. The transmission line 110 is preferably a digital transmission line capable of high-speed communication, and may include a computer network. Further, the transmission line 110 may be constituted only by the computer network without using a telephone line. Is also good. Further, the communication device 20 may be built in the camera 10 and integrated with the input circuit 18 and the like.

The server 22 is a file server that stores and manages a three-dimensional three-dimensional map and detailed information related to the map, and also inputs and stores various data corresponding to image data obtained by photographing. The server 22 according to the present embodiment includes a processing device 28 and storage devices 32a, 32b, and 32c, and the server 22 displays necessary information based on a request sent from the display circuit 30 via the connection line 112. It has a function to be provided to the circuit 30.

More specifically, the processing device 28 is a control device that manages various databases. The processing device 28 in the present embodiment processes information input through the connection lines 110 and 114, and further processes a card reader connected to the input 114.
The storage device 32c has a function of processing the recording information of the memory card 24 read by 26 and constructing an image database mainly composed of images in the storage device 32c based on the information. Further, the processing device 28 searches for information corresponding to the request input to the input 112, and stores the relevant information in the storage devices 32a, 32b,
Display circuit 30 read from 32c and connected to output 112
It has the function of outputting to

The storage device 32 is a large-capacity storage device for storing digital data. Storage device 32 in this embodiment
Has three storage areas for storing three types of related information, and in this description, they are stored in separate storage devices.
32a, 32b, 32c. For example, the storage device 32a stores three-dimensional map data having height information including elevation data indicating the height of the land and city area data indicating the state of land use, thereby constructing a three-dimensional map database. ing. The storage device 32b stores map information data such as place names corresponding to the three-dimensional map and information on the places, and a map information database is constructed. Further, in the storage device 32c, the information sent from the input circuit 18 and the information read by the card reader 26 are classified and stored under the control of the processing device 28, and an image database is constructed. The processing device 28 is a computer system that manages these databases, and the database 32 and the processing device 28 form a file server.

The three-dimensional map database is stored and managed, for example, by converting a numerical map issued by the Geospatial Information Authority of Japan into a three-dimensional three-dimensional map topographic data format which can be easily processed by a computer. In the three-dimensional database, the map information database, and the image database, data related to the position corresponding to each other is read out from the storage device 32 by the processing device 28.

The display circuit 30 connected to the server 22 outputs a request for obtaining desired data from the server 22 to the output 112 in accordance with the operation information input to the input 116 in response to the operation on the input device 34, and further outputs the request. Is a display control device that processes data sent from the server 22 into data to be displayed on the display device 36 and displays an image on the display screen.

The display circuit 30 according to the present embodiment is constituted by a computer system. In particular, the display circuit 30 has a 3D processing function of displaying a three-dimensional map on a display screen in three dimensions, and moves within the three-dimensional map. It has a viewpoint control function for changing the viewpoint for viewing the map, and an image processing function for forming a panel for further displaying an image in the three-dimensional map and pasting an image corresponding to the location to the panel.

All or some of these functions may be constituted by hardware such as a dedicated integrated circuit.
In this embodiment, this is realized by a control program for a computer. In this case, the display circuit 30 is a computer system including a central processing unit and a main storage device that perform high-speed arithmetic processing for processing and displaying an image, and performs arithmetic processing according to a procedure according to a control program. The control program is recorded on a storage medium that can be read by a computer, and is read out by a reading device and stored in a main storage area and an auxiliary storage device of the system. A mask ROM (Read Only Memory), a flash memory, a nonvolatile RAM (Random Access Memory), and the like are advantageously applied as such storage areas and storage devices.

The provided storage medium is, for example, a disk-type or card-type recording medium on which information is recorded using magnetism and / or light, or a cartridge-type storage medium formed of a memory element such as a semiconductor. And so on. Further, such a control program may be configured to be transmitted to a display circuit 30 from a remote storage device or the like by transmitting a communication line or the like. Further, such a function of the display circuit 30 can be realized, for example, on the computer system of the server 22 shown in FIG.

The 3D processing function in the display circuit 30 is based on the three-dimensional map data, for example, approximates the undulation of the land with a triangular patch, and displays and expresses the perspective by perspective transformation or the like, and also displays the perspective. Surface / hidden line processing and surface processing are performed to create a surface model or solid model representing the terrain or structure. Further, based on the state of the terrain, for example, a mountain is a green color, a sea is a blue color, and a sunset is a red color or similar. Images to be applied are colored on those surfaces. With such a function, a three-dimensional map is displayed on the screen. The display circuit 30
In addition, it has a viewpoint control function of sequentially creating a three-dimensional map at the position where the viewpoint is changed by changing the position where the map is viewed, that is, by changing the viewpoint. For example, by moving the viewpoint and changing the direction of the viewpoint, it is possible to move a desired place back and forth, left and right, and look around the state of the land and a photographic image of an image panel described later. Also, the viewpoint position and direction can be set vertically and vertically, and in this case, the selected area can be looked down from above.

The image processing function of the display circuit 30 creates data representing an image panel for displaying an image corresponding to the viewpoint position of the three-dimensional map, and pastes the image as a texture on the surface of the image panel. Specifically, as shown in FIG. 4, the display circuit 30 moves three points from the current viewpoint position.
If there is an image to be displayed within the range of viewing the 3D map,
The image panel 40 is arranged based on the position, direction, and altitude given to the image. The image panel 40 is formed so as to face the photographing position at a predetermined distance L from the photographing position based on the photographing position P and the photographing direction D of the image. The display circuit 30 uses the latitude and longitude data and the altitude data to determine the shooting position P (Px,
Py, Pz) and the photographing direction D (Dx, Dy, Dz) from the photographing position
Is calculated based on the azimuth data. Next, the shooting position P
Is generated, and a vector of the imaging direction D and the length L is created, and the surface of the image panel 40 is arranged so as to coincide with a tangent plane to the spherical surface when the vector D is rotated. In this case, the image panel 40 is formed into a rectangle of a vertical and horizontal size corresponding to the image, and is arranged such that its center is on a straight line of the direction vector D.

When viewed from the viewpoint position of the three-dimensional map different from the shooting direction, the image panel 40 has an outer shape deformed into a parallelogram or trapezoid according to the relative angle, as shown in FIG. Is displayed. Further, the size of the image panel is enlarged or reduced according to the distance from the viewpoint position. Then, when these shooting positions and shooting directions substantially match the viewpoint positions and directions on the three-dimensional map,
As shown in FIG. 5, the display circuit 30 displays a three-dimensional map as a background, and combines and displays an image panel 52 in front of the viewpoint. In the example shown in the figure, one image panel 52 is arranged substantially at the center of the screen on the three-dimensional map.
Further, map information data such as a place name, a road name, and a sightseeing guide text in the area searched from the map information database 32b is displayed in the area 52. This display area 52 is arranged inside or outside the image panel 50,
Information other than the image obtained at the time of capturing the image, the title of the image, a comment, and the like can be further displayed in the image 50.

The display circuit 30 reduces, enlarges, and further deforms the image displayed on the image panel according to the size and shape of the image panel, and pastes the processed image on the corresponding image panel. It has an image processing function. As a result, appropriately processed and processed images are pasted as textures on a plurality of image panels arranged in various places and directions according to the viewpoint position and direction in the three-dimensional map.

In the case where a plurality of image panels are displayed on a screen, for example, as shown in FIG.
Only the image panel PA within a certain range is displayed on the screen 54, and the other image panel PA is displayed instead of a mark SR such as a flag. Thus, the screen can be complicated. Then, when the viewpoint approaches the mark SR, the display is switched to the image panel. This switching is not limited to the distance between the image panel PA and the viewpoint position. For example, an image corresponding to a condition such as a shooting date is searched, and the mark is displayed on the image panel on which the image is displayed. You should decide whether or not. The display and non-display of the image panel can be arbitrarily designated to the operator by the operation information. Therefore, unnecessary image panels can be sequentially switched to non-display or mark display.

Returning to FIG. 1, the input device 34 is a device that detects an operation by the operator and outputs operation information corresponding to the operation to the output 116. For example, the input device 34 includes a keyboard that outputs code data corresponding to a key press and a pointing device such as a mouse or a trackball for indicating a position corresponding to an operation. And output operation information for giving an instruction. For example, the input device 34 provides the display circuit 30 with operation information for causing the display circuit 30 to output a request for searching for data recorded in the server 22 and a movement instruction for moving the viewpoint to the display circuit 30 so as to display the request in the 3D map. Outputs operation information for moving the viewpoint.

The display device 36 is an image display device for displaying an image corresponding to data output from the display circuit 30 on a screen. The display screen is composed of a two-dimensional plane or a curved surface, and a three-dimensional three-dimensional map is displayed as a color image on this screen. Then, on this three-dimensional map, an image panel is displayed with the movement of the viewpoint, and the photographed image is pasted and displayed on the surface of the image panel. The image to be displayed at this time is an image having position information corresponding to the position on the three-dimensional map, and an image actually shot with the virtual three-dimensional map displayed is displayed. Of course, the display device 36 may be a three-dimensional display or a binocular stereoscopic image display device capable of displaying a three-dimensional image.

The operation of the electronic album system 1 according to the present embodiment having the above configuration will be described below with reference to FIGS.

Referring to FIG. 2, there is shown a flowchart for explaining the image data input processing operation in the electronic album system 1. The input circuit 18 acquires and processes data of the image signal and various types of position information, and records and processes them on the memory card 24 as image data or transfers them to the server 10.

First, in step S201, when the shutter button of the camera 10 is pressed, an image signal sent from the digital camera 1 is obtained by the input circuit 18, and in step S202, the image signal is output from the GPS receiver 12. The input circuit 18 acquires latitude data and longitude data indicating the current position of the camera 10.

Next, in step S203, the direction sensor
The current azimuth data of the camera 10 output from 14 is acquired by the input circuit 18, and the current altitude data of the camera 10 output from the altitude sensor 16 is acquired by the input circuit 18 in step S204. Here, the longitude data, the latitude data, the azimuth data, the altitude data, and the like obtained in the processing from step S202 to step S204 are collectively referred to as position and azimuth data.

Next, in step S205, image data obtained by compression-encoding the image signal and the position and orientation data into a predetermined format is created. Next, in step S206, the created image data is output to the output 100 of the input circuit 18 and recorded on the memory card 24 loaded in the camera 10. Subsequently, the process proceeds to step S207, when performing continuous shooting, the process is again shifted to the image signal input process of step S201, and the processes of steps S201 to S207 are repeated. When registering the image data in the server 10, the image input processing ends.

When the memory card 24 is removed from the camera 10 and the memory card 24 is loaded into a card reader 26 connected to the server 22, the image data recorded on the memory card 24 is read out and processed by the server 22. The data is transferred to the device 22 and registered in the image database 32. Since the image data is provided with the position information based on the position and orientation data, a database using the position information as a keyword is created, and an image corresponding to the position can be searched later.

The data recorded on the memory card 24 can be transmitted to the server 22 using a communication line depending on the situation. In this case, the image data once recorded in the camera 10 is read out to the input circuit 18 and transferred to the communication device 20. When the transfer button is operated, the communication device 20 accesses the server 22 of the registered address with a specific user name and password, and transmits photographed image data to the server 10. When transferring image data using a communication line, the image data is transferred to the server
Since the data can be transferred to the memory card 10, there is no need to worry about the storage capacity of the memory card 24, and there is no need to carry a large number of cards 24, which is effective.

When the image data is registered in the server 22 in this way, the registered image can be retrieved and displayed on the display device 36 together with the three-dimensional map. As shown in FIG. 3, a flowchart showing the display processing operation in this embodiment is shown. The display circuit 30 searches for a three-dimensional topographic map based on designation of a display position by an operator, and image data and map information existing in the displayed space, and then selects and displays the data.

First, in step S301, the input device 34
The operation information corresponding to the operation for is detected by the display circuit 30, three-dimensional map data corresponding to the display position intended by the operator is searched, and the three-dimensional map data corresponding to the search result is input to the display circuit 30. Is done. Then, the elevation data representing the undulation of the land and the city area data representing the land use are combined, and when an area visible from the viewpoint position according to the operation is selected, the data representing the three-dimensional three-dimensional map is converted into a computer graphic. The created data is output to the display device 36, and a three-dimensional three-dimensional map is displayed on the display screen of the display device 36.

At first, displaying a wide area is convenient for narrowing down a target place. Therefore, map data for that purpose is dynamically simplified by thinning or smoothing a three-dimensional detailed map. It is good to create data and use it hierarchically. In this case, switching is performed so that a detailed map is displayed as the user approaches the destination.

In step S302, the user moves freely in the three-dimensional map on the screen in accordance with the operation information obtained by the input device 34, and displays the three-dimensional map corresponding to the viewpoint position. In this way, the operator moves around on the map and approaches the destination. When a place name or the like is directly input to the input device 34, the map information database 32b is searched from the place name, the position is recognized, and the viewpoint position can be moved to the corresponding point.

In step S303, when the viewpoint position is moved and displayed in the three-dimensional map, a certain range from the current viewpoint position, for example, a circle having a radius R in the horizontal direction centering on the viewpoint position P is set. Existing image data is retrieved from the image database 32b. Based on the retrieved data, the display circuit 30 recognizes the shooting position P (Px, Py, Pz) in the three-dimensional map from the longitude data, the latitude data, and the altitude data in the position and orientation data of the selected image data. , The shooting direction D (Dx, Dy, Dz) from the shooting position is recognized from the azimuth data. Then, from the shooting position P,
A rectangle having a center on a direction vector D passing through a point P at an appropriate distance L in the direction of the vector D and having a vector D as a normal line and a tangent plane tangent thereto, for example, A plane having a shape proportional to the size of the photograph is set as an image panel, and the selected image data is pasted thereon as a texture. At this time,
Information such as position and orientation data possessed by the image data may be combined with the image panel and displayed therein. Here, the rectangular plane on which the image data is pasted with the texture is called an image panel PA including the image.

By pasting the image data as a texture on a rectangle having an orientation, it is only necessary to confirm the direction of the normal and the line of sight to the rectangle. Is displayed, or processed so as not to be displayed on the contrary. Furthermore, even if there is a position where the image panel should be displayed in the displayed three-dimensional space, it is within a certain range (within a radius R) from the viewpoint.
If there is no such position, if the direction is reversed and cannot be seen from the viewpoint position, or if it is not included in the current selection conditions, as shown in FIG. A mark SR having a three-dimensional size indicating that the mark SR exists is displayed.

Next, in step S304, the photographing position of the displayed image is checked, map information corresponding to the position is searched from the map information database 32b, and the searched map information is set in the vicinity of the corresponding image. Is displayed in the area 52 (FIG. 5). The display position of the area 52 may be, for example, up, down, left, or right of the image panel or inside the image panel. Whether or not to display the map information can be selected according to the operation.

Next, in step S305, a display condition for selecting under what condition an image is to be displayed is specified by the input device. In this case, the shooting date / time data, place name data,
When contents such as altitude data and display area data are input, an image that meets the conditions is displayed on the image panel.
All the selection items of the display condition can be used as long as the information is input when the image data is created. Further, it is possible to input positional information such as a place name in a map as a display condition, specify an area from the map space currently displayed on the screen with the input device 34, set the display condition, and display the map. .

The number of image panels to be displayed on the screen is
It is possible to change the condition by strictly setting image data search conditions or specifying a plurality of conditions. For example, in this example, when the value of the radius R is increased, the number of image panels to be displayed increases, and when the value of the radius R is reduced, the number of image panels to be displayed decreases. Furthermore, even within the range, only one point can be selected by adding a condition that the point is closest to the viewpoint position. In addition, since the shooting date and shooting time are recorded in the image data, by specifying the shooting date and time, it is possible to limit the display even when the user has visited and photographed many times. By doing so, it is possible to avoid a situation in which a plurality of images are unnecessarily overlapped and difficult to see.

Next, the process proceeds to step S306. If the user continues to move on the three-dimensional map, the process returns to step S301. Otherwise, the process ends. By repeating such processing, the operator operates the input device 34 to move freely in the three-dimensional space while moving the viewpoint with respect to the three-dimensional map, and looks at the image panel on which images corresponding to the respective positions are displayed. Turn around. At this time, the setting of the display condition in step S305 is performed when necessary. Then, when the user moves near the desired image panel, the same three-dimensional map as the situation at the time of shooting the image is displayed on the screen, and the situation at the time of shooting the image can be easily understood. By further moving the viewpoint, the terrain and the like are also displayed on a map of the surrounding situation, so that even if a photograph is not taken, the situation can be easily imagined.

As described above, by operating the input device 34 to change the viewpoint position and the line-of-sight direction in the three-dimensional map, it is possible to look around the three-dimensional space. If there is image data, when approaching the position, the image panel on which the corresponding image is displayed appears on the three-dimensional map while maintaining the shooting direction at the shooting position, and may move away from the position. If it goes too far, it automatically becomes the preset mark SR or is not displayed. Further, since the three-dimensional map space is created by the perspective method, the image panel to be displayed is automatically and continuously enlarged when approaching, and automatically and continuously reduced when approaching. Therefore, when the user wants to see the image in detail, the viewpoint is moved so as to approach the image panel. As described above, since the electronic album system itself is formed in a three-dimensional virtual space, it is possible to intuitively recall a situation in which a photograph was taken.

Further, since the images are directly arranged on the three-dimensional map, the absolute position and the relative positional relationship can be clearly expressed. Then, by moving in the space, the photographed images can be sequentially viewed. Also,
Since the input image itself provides information to the three-dimensional map space generated by computer graphics, the realism of the space itself can be increased.

In this manner, the viewpoint movement state in which the viewpoint is moved and the various images are viewed is selected and stored in, for example, the image database 32c. You can automatically browse through the images. At this time, the actual photographic image is also displayed at each location, so that the sense of presence as if moving from that location can be further increased.

In the first embodiment, the digital camera 10 for recording a still image has been described. However, the digital camera 10 may be for recording a moving image. For example, a video camera that captures a moving image and records it on a recording medium may be used. Next, a second embodiment of an electronic album system for converting a moving image into an album and recording and saving the album will be described below with reference to FIG.

Electronic album system 70 in this embodiment
Is a system for inputting a moving image from a video camera 72 that digitizes a moving image picked up by an image pickup device and records it on a magnetic tape, creates an electronic album corresponding to the moving image, and appreciates it. In the following description, the configuration of the same reference numeral as the configuration shown in FIG. 1 may be the same configuration, and the description is omitted.

The video camera 72 is a movie camera that shoots an object scene and outputs a digital video signal representing the scene image to an output 100. The camera 72 records a moving image signal obtained by photographing on a recording medium such as a magnetic tape 74 or a magneto-optical disk, and records the photographing date and time, and the position and orientation data output from the input circuit 18. Is recorded in correspondence with each frame of the image signal. The position and orientation data may be the same as the data in the embodiment shown in FIG. 1, and at least the position and orientation data corresponding to the first frame of the moving image is recorded corresponding to the frame of the image. The position and orientation data corresponding to the first and subsequent frames may be recorded continuously for each frame, or may be recorded together with the corresponding frames, for example, every few frames or every few seconds.

The input circuit 18 from which the moving image signal and the data such as the position and orientation data recorded on the magnetic tape 74 are read and input, are processed and processed into moving image data in a predetermined format. Communication device 20 for moving image data
Transfer to The communication device 20 compresses and encodes the image signal according to, for example, the MPEG method and transmits the image signal to the server 22. Alternatively, the magnetic tape may be removed from the camera 72, the magnetic tape 74 may be loaded into the reproducing device 76 connected to the server 22, and the recorded image signal and data such as position and orientation data may be reproduced and input to the server 22. it can.

The server 76 in this embodiment uses the recorded data from the start to the stop of recording as data of one cycle.
A storage device that stores input moving image data as a unit
Record in 32d and build a video database. It has a function of searching for a moving image registered in the moving image database 32d based on the position and orientation data recorded in the first frame of the moving image data. Other configurations of the server may be the same as those of the server shown in FIG.

The display circuit 80 according to the present embodiment, in addition to the configuration of the display circuit 30 shown in FIG. 1, particularly, when displaying a moving image on an image panel, corresponds to the position and orientation data corresponding thereto. Includes a function to display the three-dimensional map by moving the viewpoint position. Further, when the reproduction of the moving image ends, the display circuit 80 returns the viewpoint position of the three-dimensional map to the point indicated by the position and orientation data recorded in the first frame, displays the map, and displays the image of the first frame of the image image. Includes function to display on image panel. With such a function, it is possible to move the three-dimensional space serving as the background as if it were photographed while moving with the camera 72, and fuse the moving image with the background.

The operation of the electronic album device 70 according to this embodiment having the above configuration will be described below with reference to FIGS. Referring to FIG. 8, there is shown a flowchart for explaining an operation of inputting moving image data in the electronic album system 70.

First, in the recording processing in step S801, when the recording button of the video camera 74 is pressed,
The moving image signal transmitted from the GPS receiver 12, the latitude and longitude data indicating the current position of the camera 74 output from the GPS receiver 12, and the current camera output from the direction sensor 14.
The azimuth data of 74 and the current altitude data of the camera 74 output from the altitude sensor 16 are acquired by the input circuit 18, respectively. Here, longitude data, latitude data, azimuth data, altitude data, and the like are collectively referred to as position and azimuth data.
The image signal and the position and orientation data acquired by the input circuit 18 are sequentially compression-encoded, and the encoded image data is sequentially recorded on the magnetic tape 74.

When a series of photographing and recording are completed, the process proceeds to step S802, and the moving image data recorded on the magnetic tape 74 is registered in the moving image database 32c of the server 22.
In this case, as in the first embodiment, the magnetic tape 74 is loaded into the playback device 26 to read out the image data and register it in the moving image database 32c of the server 22, or
The moving image data read from the magnetic tape 74 loaded in 72 is transmitted to the server 22 via the communication device 20, and the moving image data is registered in the moving image database 32c. Each frame of the moving image data is provided with position information based on the position and orientation data, a database using the position information as a keyword is created, and a moving image corresponding to the position is later searched.

If a group of moving image data has been registered, the flow advances to step S803. If there is moving image data to be registered in addition to the currently registered moving image data, step S803 is performed.
The processing of S801 and S802 is repeated to register the moving image data in the server 22, and if not, the input processing ends.

When the image data is registered in the server 22 in this way, the registered moving image can be searched, and the searched moving image can be displayed on the display device 36 together with the three-dimensional map. Referring to FIG. 9, there is shown a flowchart showing the display processing operation in this embodiment.

The display circuit 80 searches for a three-dimensional topographic map based on the designation of the display position by the operator, as well as moving image data and map information corresponding to the displayed space, and then selects and displays it.

The operation from the three-dimensional map display processing in step S901 to the destination movement processing in step S902 may be the same as the processing in steps S301 to S302 described in the first embodiment. In the next step S903, when the three-dimensional map is displayed by moving the viewpoint in the three-dimensional map, moving image data existing within a certain range, for example, within a circle of radius R 'from the current viewpoint is displayed. Is a moving image database 32
Searched from d. In this case, in this embodiment, since the position and orientation data of the moving image changes with time, the moving image is searched based on the value of the position data recorded in the first frame of the moving image data.

Next, based on the moving image data selected by the search, the display circuit 80 uses the longitude data, the latitude data, and the altitude data recorded in the position and orientation data corresponding to the first frame in the three-dimensional map. Shooting position (viewpoint position) P '(P'x, P'y, P'z) of the first frame in
Is recognized, and a photographing direction D '(D'x, D'y, D'z) from the photographing position is recognized from the azimuth data. Then, as shown in FIG. 11, a rectangle having a center on a straight line of the direction vector D 'passing through the point P' is provided at a distance from the shooting position P 'by an appropriate distance L' in the direction of the vector D '. A vector D ′ is set as a normal line, and a plane having a shape proportional to the size of the captured moving image is set as a moving image panel on a tangent plane in contact with the vector D ′. Paste the frame as a texture. At this time, information on the position and orientation data possessed by the moving image data may be combined with the moving image panel and displayed therein. Here, a plane obtained by pasting one frame of a moving image as a texture onto a rectangular plane is referred to as a moving image panel PB.

In this embodiment, similarly to the first embodiment, the image data is pasted as a texture on a rectangle having an orientation, and the normal line of the rectangle and the direction of the line of sight are checked. When the PB is viewed from the back, it can be hidden from view, and if it is not necessary to display the moving image panel, a mark SR indicating that moving image data exists at that position is displayed. You.

Next, in step S904, map information data corresponding to the moving image data included in the display area is searched from the map information database 32b, and the map information selected by the search as shown in FIG. Area 52 near
To be displayed. Next, the process proceeds to step S905, in which the display condition for displaying the moving image is specified in the same manner as in step S305 described in the first embodiment, and the moving image is displayed on the moving image panel according to the display condition. Is set to

Next, in step S906, an operation of selecting moving image data displayed on the moving image panel is performed by the input device 34.
It is determined whether or not the processing has been performed. If the selection has been made, the process proceeds to step S907; otherwise, the process proceeds to step S901.
To S906 are repeated.

In step S907, a moving image reproducing process for reproducing and displaying a moving image on the moving image panel is performed. More specifically, as shown in FIG. 10, this moving image reproduction process is performed in step S1.
In 001, it is recognized which moving image panel currently displayed on the screen is selected in step S906, and when the process proceeds to step S1002, the position of the first frame of the moving image displayed on the selected moving image panel is determined. Then, the viewpoint position of the three-dimensional map is moved, the viewpoint direction is set to the shooting direction at the viewpoint position, and the display of the three-dimensional map is smoothly moved according to the setting.

In this manner, the process proceeds to step S1003 in a state where the three-dimensional map is displayed in the position and direction at the time of shooting, and the moving image panel is moved to a location corresponding to the position and orientation data recorded in the current frame. The image of the current frame is pasted as a texture on the surface of the arranged moving image panel (step S1004).

If it is determined in step S1005 that the moving image data is to be continuously reproduced, the frame is advanced by one in the next frame in step S1006 to update the frame, and the processing is performed in step S1002 to set the viewpoint position and line-of-sight direction. Move on to processing. By repeating this, while moving the moving image to the front, the three-dimensional space serving as the background also moves in the same manner as when the video camera 72 shoots. Therefore, a map in the shooting direction is always displayed, and the moving image and the background are well fused.

When the process is terminated in step S1005, the position of the moving image panel on the three-dimensional map is returned to the value indicated by the position and orientation data recorded in the first frame of the moving image data, and The image of the frame is pasted by texture mapping to make the state before reproducing the moving image (step S1007), and the process proceeds to step S908 shown in FIG.

In step S908, it is determined whether or not to end the entire process.
Returning to S901, while freely moving around in the three-dimensional space,
When looking at the moving image panel and wishing to view the moving image, the user can repeatedly select the moving image panel with a mouse or the like. In this case, the display condition setting process in the moving image data display condition setting process in step S905 can be performed as needed.

By operating the input device 34 as described above,
Change the viewpoint position and line-of-sight direction in the three-dimensional map,
You can look around the three-dimensional space. Then, if there is moving image data, a moving image panel is automatically displayed when approaching the position, and reproduction of the moving image is started by selecting the moving image panel. Then, the viewpoint position is set at the shooting position of the video camera 72, and the moving image can be reproduced in front while turning the line of sight to the shooting direction. At this time, if the moving image is moved away from or over the position without reproducing the moving image, the moving image panel automatically becomes a preset mark or disappears. In addition, since the moving image panel is generated in the three-dimensional space, the moving image panel to be displayed is automatically and continuously enlarged when approaching, and is automatically and continuously reduced when moving away. So, if you want to see the movie being played in detail,
It is better to perform the operation of approaching further. in this way,
Since the electronic album system itself is formed in a three-dimensional virtual space, it is possible to intuitively recall the situation in which the video was shot, and to increase the sense of reality and presence.

Further, since the moving image panel and the moving image are directly arranged on the three-dimensional map, the absolute position and the relative positional relationship can be clearly expressed. And
By moving around the space and selecting the moving image panel you want to see,
The photographed moving images can be sequentially viewed.

The moving image panel and the moving image itself are:
Since information is given to the three-dimensional map space generated by computer graphics, the realism of the space itself can be increased.

In the first and second embodiments described above, the case of a video camera and a digital camera has been described. However, these can be combined and used. In this case, in order to distinguish the image panel that displays a still image from the moving image panel, add a frame with a different color to the rectangle where the image is to be pasted, and add characters such as “Still” and “Video” to each panel. Should be displayed in correspondence with.

In an image input device such as a digital camera or a video camera, since an audio signal can be recorded on a recording medium, this audio is registered in a database in association with an image, and a moving image and a still image are recorded. Audio can be played back during playback. In this case, by registering the audio signal created separately from the time of shooting in the database in association with the still image and the moving image, for example, a narration corresponding to each scene can be reproduced.

In the above two embodiments, a topographic map is used as a three-dimensional space. However, the contents registered in the map database 32a are, for example, a model of a structure such as a building or a layout of indoor furniture or the like. And a still image or a moving image may be arranged and displayed in the three-dimensional space. In addition, place these buildings on the topographic map, select them, and enter the building to move from the outside world to the inside of the building or the indoor world, and vice versa. Or you can.

[0087]

As described above, according to the present invention, the photographing position and photographing position of a still image or a moving image obtained by photographing on a three-dimensional map based on the position and direction at the time of photographing these images. The image can be arranged and displayed while maintaining the direction, and the direction in which the image is captured and the direction in which the three-dimensional map is viewed can be displayed so as to match, and the display is not limited to the display of the captured image. Since the three-dimensional map corresponding to the photographed image is displayed, the situation at the time of photographing can be more realistically expressed, and the photographing situation can be clearly recognized.

[Brief description of the drawings]

FIG. 1 is a functional configuration diagram showing a first embodiment of an electronic album system to which the present invention has been applied.

FIG. 2 is a flowchart illustrating an input processing operation in the first embodiment.

FIG. 3 is a flowchart illustrating a display processing operation in the first embodiment.

FIG. 4 is a diagram illustrating an image panel according to the embodiment.

FIG. 5 is a diagram showing an example of display in the embodiment.

FIG. 6 is a diagram showing an example of display in the embodiment.

FIG. 7 is a functional configuration diagram showing a second embodiment of the electronic album system to which the present invention is applied.

FIG. 8 is a flowchart illustrating an input processing operation in the second embodiment.

FIG. 9 is a flowchart illustrating a display processing operation in the second embodiment.

FIG. 10 is a flowchart showing a moving image reproduction process shown in FIG. 9;

FIG. 11 is a diagram illustrating a moving image panel according to a second embodiment.

[Explanation of symbols]

 1 electronic album system 10 digital camera 12 GPS receiver 14 direction sensor 16 altitude sensor 18 input circuit 22 server 30 display circuit 34 input device 36 display device

Claims (20)

[Claims] 1. An image processing apparatus for generating a three-dimensional image according to three-dimensional data and displaying the three-dimensional image on a display device, the device comprising: image data generated in response to image capturing; A first data including image data representing a position and an image direction;
Input means for inputting the three-dimensional data and the first data; and processing means for displaying a three-dimensional image represented by the three-dimensional data on a display device. Reads image data of a shooting position corresponding to a stereoscopic image displayed on the display device from the storage unit, synthesizes an image represented by the image data on the stereoscopic image according to the shooting data, and displays the image on the display device An image processing apparatus characterized in that: 2. The image processing apparatus according to claim 1, wherein the processing unit performs three-dimensional data corresponding to a viewpoint position of the stereoscopic image in accordance with operation information from an operation unit that instructs a display state of the stereoscopic image. And first data at a shooting position corresponding to the three-dimensional data are read out from the storage means, and image data of the first data is combined with the three-dimensional data according to the shooting data, and an image represented by the image data is obtained. Is displayed on the three-dimensional image in a state where the shooting position and the shooting direction are held. 3. The image processing apparatus according to claim 2, wherein the processing unit sets a display area for displaying an image represented by the image data on the three-dimensional image according to the shooting position and the shooting direction. An image processing apparatus for pasting an image corresponding to the photographing data in the display area and displaying the image. 4. The apparatus according to claim 1, wherein the photographing data includes a photographing altitude at the photographed position. 5. The image processing apparatus according to claim 3, wherein the apparatus generates the image data and records the image data on a recording medium; Detecting means for inputting shooting data, wherein the camera shoots a scene to generate the image data according to the scene, and converts the shooting data representing the shooting position and the shooting direction to the image. An image processing apparatus which records the first data together with data on the recording medium. 6. The image processing apparatus according to claim 5, wherein the input unit reads out first data recorded on the recording medium and registers the first data in the storage unit in a searchable manner. An image processing apparatus comprising: 7. The image processing apparatus according to claim 5, wherein the camera converts moving image data representing a moving image obtained by the shooting and the shooting data corresponding to each frame of the moving image data. A camera that records the first data on the recording medium, wherein the processing unit is configured to display the stereoscopic image at a position corresponding to shooting data corresponding to a first frame of the moving image data. An image processing apparatus for setting a display area. 8. The image processing apparatus according to claim 7, wherein the processing unit displays the stereoscopic image by matching a line-of-sight direction in the stereoscopic image with the shooting direction, and the display set to the line-of-sight direction. An image processing apparatus for displaying a moving image corresponding to the shooting position in an area. 9. The image processing apparatus according to claim 5, wherein the photographing data includes a photographing altitude at the photographed position. 10. An image processing method for creating a three-dimensional image according to three-dimensional data and displaying the three-dimensional image on a display screen of a display device, the method comprising: A first display step of displaying an image on the display screen; a moving step of moving a viewpoint position in the stereoscopic image; selecting image data to be displayed in the stereoscopic image, and displaying the image represented by the image data in the display A second display step of combining and displaying on the stereoscopic image displayed on the screen, wherein the image data is obtained at a shooting position corresponding to the stereoscopic image, and the position data indicating the shooting position is The second display step is performed by associating a stereoscopic image corresponding to the viewpoint position with an image corresponding to the shooting position according to a movement operation of moving the viewpoint. Image processing method and displaying. 11. The image processing method according to claim 10, wherein the second display step sets a display area for displaying the image data on the stereoscopic image, and sets the display area in the setting step. An attaching step of attaching an image represented by the image data to the displayed display area, wherein the setting step sets the display area on the stereoscopic image according to the position data. Processing method. 12. The image processing method according to claim 11, wherein the position data includes a photographing direction indicating a direction at the time of photographing, and the setting step sets the display area according to the photographing direction. An image processing method comprising: 13. The image processing method according to claim 12, wherein the position data includes a photographing altitude indicating the altitude at the time of photographing, and the setting step sets the display area according to the photographing altitude. An image processing method comprising: 14. The image processing method according to claim 12, wherein the image data is moving image data representing a moving image, and the second displaying step includes the step of displaying the position corresponding to the moving image data. An image processing method, wherein the moving images are sequentially displayed in a display area according to data. 15. The image processing method according to claim 14, wherein the second setting step sets the display area based on the position data corresponding to a top frame of the moving image data. Characteristic image processing method. 16. The image processing method according to claim 14, wherein the first display step corresponds to position data of a moving image displayed in the second display step. An image processing method characterized by sequentially displaying a stereoscopic image at the position and direction of the viewpoint. 17. The image processing method according to claim 16, wherein in the second display step, the moving image is sequentially displayed in the display area by selecting the display area according to an operation. Characteristic image processing method. 18. The image processing method according to claim 10, wherein the three-dimensional image represented by the three-dimensional data is an image showing undulations of terrain, and the three-dimensional data is stored in a first storage unit. The image data is stored on the terrain and stored in the second storage means together with the position data. The method comprises the steps of: storing an image of the position data corresponding to a stereoscopic image displayed on the display screen; An image processing method, wherein data is retrieved and read out from the second storage means. 19. A three-dimensional image corresponding to the three-dimensional data is created, the three-dimensional image is displayed on a display screen of a display device, and a position obtained at a shooting position corresponding to the three-dimensional image and indicating the shooting position A computer-readable information recording medium storing an image processing program for combining an image represented by image data with data with the stereoscopic image and displaying the image on the display screen, wherein the three-dimensional data is stored. A first display procedure for reading the stereoscopic image from the display screen and displaying the stereoscopic image on the display screen; a moving procedure for moving a viewpoint position in the stereoscopic image; and selecting image data to be displayed in the stereoscopic image, And displaying the image to be displayed on the stereoscopic image displayed on the display screen. The second display step includes moving the viewpoint. In accordance with the movement operation, the image processing program is recorded computer-readable information recording medium that displays the image corresponding to the imaging position and the three-dimensional image corresponding to the position of the visual point on the display screen in correspondence. 20. The information recording medium according to claim 19, wherein the second display step includes a setting step of setting a display area for displaying the image data on the three-dimensional image, and the setting step. And an attaching step of attaching an image represented by the image data to the displayed display area, wherein the setting step sets the display area on the three-dimensional image according to the position data. A computer-readable information recording medium on which a processing program is recorded.