JP3466493B2 - Panorama image creation method and apparatus, recording medium recording panorama image creation program, information providing method, apparatus, recording medium recording information providing program - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information providing device such as a navigation device mounted on a moving body such as an automobile.

[0002]

2. Description of the Related Art Conventionally, for example, in a navigation device mounted on an automobile, map data is read from a storage unit built in the device based on a predetermined operation by a driver, and a map is displayed on a display installed near a driver's seat. This will guide the driver.

Further, in recent years, even if a map near the current position is displayed to the driver or the like, it is often difficult to associate the actual map with the local map. An apparatus and method have been invented, such as 33271 and Japanese Patent Application Laid-Open No. 10-132598, in which guidance information is superimposed and displayed on a live-action image captured by a camera.

In addition, as a conventional method of generating a panoramic image, 1. A method of detecting feature points in the shot images and superimposing the shot images, 2. 2. A method in which a moving body equipped with a camera for an object to be photographed is moved to combine the captured images, or 3. There was a method of joining images taken while moving a camera called a line sensor camera with a fixed slit width while moving at a constant speed.

[0005]

The method of extracting the feature points has a problem that the amount of image calculation processing is large.

Further, there is a problem that the conventional panoramic image generation method by superposing feature points cannot be used when only an image is input from one camera in the traveling direction. The problem is that the distance between the camera and the object changes due to the movement of the camera, the apparent size of the object changes, and it becomes difficult to determine the identity of the feature points, and superposition cannot be performed. Due to the problem of.

Further, the method using the line sensor camera has a problem that the generated panoramic image is distorted unless the moving speed of the camera is constant.

An object of the present invention is to reduce the amount of image calculation and to generate a panoramic image on either the left or right side by one camera oriented in the traveling direction, and a panorama with little distortion even if the moving speed of the camera changes. An object of the present invention is to provide a panoramic image generation method, an apparatus, and a recording medium recording a panoramic image generation program capable of acquiring an image.

Another object of the present invention is to provide an information providing method capable of always providing information such as route guidance, tourist information, advertisements, etc. to a driver of a moving body or a passenger using the above panoramic image generation method. , An apparatus, and a recording medium in which an information providing program is recorded.

[0010]

According to the present invention, the slit image is extracted from the width of a slit image which is an image of a vertically long rectangular portion having a minute width at an arbitrary position and a length which is the same as the vertical size of the frame. The frame of the image signal is changed according to the moving speed at the time of shooting, and even when the pan / tilt / zoom of the video camera changes, it can be obtained by shooting in a certain direction with respect to the moving method of the moving body. The slit image is created by changing the extraction position so as to extract the image information, the panorama image created in the vicinity is scrolled by the pixels of the width of the slit image created this time, and the slit image created this time is displayed. By pasting in the width of the panoramic image created by scrolling, the panoramic image on the side of the moving body is created in real time.

Therefore, high-speed processing is possible because the feature point extraction processing is not performed, and there is no limitation that the camera is oriented in the advancing direction sideways.
It is possible to generate a panoramic image in either the left or right direction with a single camera, and further, it is possible to obtain a panoramic image with little distortion even if the moving speed of the camera is not limited to a constant speed.

Further, the present invention superimposes and displays the guide information on the panoramic image. Therefore, it is easy to see because it scrolls at a relatively low speed, and because the characteristics of the panoramic image make it easy to separate even densely packed subjects, it is possible to provide easy-to-see guidance information to the driver or fellow passenger. The visibility of is dramatically improved.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Next, embodiments of the present invention will be described with reference to the drawings.

Referring to FIG. 1, an information providing apparatus according to an embodiment of the present invention includes an image acquisition section 101, a moving body position detecting section 102, a moving body traveling direction detecting section 103, and a moving body speed detecting section 104. A camera direction detection unit 105, a camera focal length detection unit 106, and a panoramic image generation unit 107.
A guidance information storage unit 108 and an image composition processing unit 109.
A display device 110 and a user operation input unit 11
1, a camera operation unit 112, and a communication control unit 113.

The image acquisition unit 101 is composed of a general video camera or a video camera for broadcasting, and sends a real image signal taken by a moving body to the image synthesis processing unit 109 and the panoramic image generation unit 107. A line sensor camera may be installed separately for generating a panoramic image, and a normal video camera image may be sent to the image synthesis processing unit 109 and a line sensor camera image may be sent to the panoramic image synthesis processing unit 107. Alternatively, a dedicated video camera may be separately provided for generating a panoramic image, and images from different video cameras may be transmitted.

The moving body position detecting section 102 is, for example, a GPS.
The position of the moving body is detected using the device, and position information (latitude, longitude, altitude, etc.) is input to the image synthesis processing unit 109. GP
The position may be detected using a method other than S.

The moving body traveling direction detection unit 103 detects the traveling direction of the moving body using, for example, an electronic compass, and inputs the traveling direction information to the image synthesis processing unit 109. Using GPS, the latest past position and the current position are measured, and the direction of travel is calculated from the difference between them.
A plurality of GPS antennas such as (registered trademark) may be installed and the traveling direction may be obtained from the phase difference of the received radio waves, or the traveling direction may be detected by another method.

The moving body speed detecting unit 104 detects the moving speed of the moving body, for example, by counting vehicle speed pulses generally equipped in the automobile, and the speed information is used as the image synthesizing processing unit 109 and the panoramic image generation. Input to the unit 107. Similar to the detection of the traveling direction, the GPS is used to measure the latest past position and the current position, the difference between them is obtained, and the moving speed is detected from the time difference at the time of the measurement and the distance moved, or otherwise. The moving speed may be detected by the above method.

The camera direction detecting section 105 includes an image acquiring section 1
For the 01 video camera, using a rotary encoder or the like, it is detected how much the angle is different relative to the traveling direction of the moving body, and the direction of the camera is detected. Image composition processing unit 10 as information
Enter in 9. If the traveling direction of the moving body is obtained, it is obvious that the absolute direction of the camera can be easily obtained from the relative angle of the camera with respect to the traveling direction of the moving body, and vice versa. Therefore, naturally, instead of detecting the relative angle of the moving body to the traveling direction, a gyro sensor is used, or a plurality of GPS antennas such as TansVector (registered trademark) of Trimble Co. The absolute direction of the camera may be detected from the phase difference. The camera direction may be detected by a method other than the above.

The camera focal length detection unit 106 detects the focal length of the lens equipped with the video camera of the image acquisition unit 101 by using a rotary encoder or the like, and synthesizes the focal length information into an image. Processing unit 109
To enter. The focal length may be detected by using a servo mechanism for driving the zoom lens remotely, or the focal length may be detected by another method.

The panoramic image generation unit 107 includes a video signal obtained from the image acquisition unit 101 and the moving body speed detection unit 1.
04, speed information obtained from the camera direction information, camera direction information obtained from the camera direction detection unit 105, camera focal length detection unit 10
A panoramic image is generated in real time based on the focal length information obtained from No. 6 and is sent to the image synthesis processing unit 109.

The panoramic image referred to here means that a line sensor camera which is generally used has a sensing line in the vertical direction as shown in FIG. 2 and the center of the sensing line is substantially horizontal, and 0 and 180 from the traveling direction.
It refers to an image obtained by shooting on the moving body with a fixed angle of displacement (for example, a direction orthogonal to the traveling direction) except for degrees, and shooting while moving at a constant speed. Alternatively, as shown in FIG. 3, a normal video camera is fixed on a moving body, and a video image captured while moving at a constant speed is long with a small width (for example, 1 pixel width) at an arbitrary position from each video frame. It is also possible to extract an image of a vertically long rectangular portion having the same size as the vertical size of the frame (hereinafter, this image is referred to as a slit image), and to define it as a band-shaped image that is created by pasting together over time. When the image information obtained from the image acquisition unit 101 installed in a moving body such as an actual car is used, the speed of the moving body at the time of shooting is not constant, and the direction of the moving body of the camera (pan / tilt Etc.) and zoom (focal length) are not necessarily fixed, it is impossible to obtain a correct panoramic image by the above method.

The panoramic image generator 107 generates a correct panoramic image from the image information obtained under such conditions using the speed information, camera direction information, and focal length information. Here, a case will be described where the image acquisition unit 101 generates a panoramic image from an image captured by a normal video camera.

In order to generate a correct panoramic image, the width of the slit image extracted from the video signal frame obtained from the image acquisition unit 101 by the velocity fluctuation correcting means described later is set to the width of the moving body at the time when the frame is photographed. The speed fluctuation is corrected by changing the speed according to the speed, and even when the pan / tilt / zoom changes, the camera operation correction unit described later captures a fixed direction with respect to the moving direction of the moving body. The extraction position of the slit image is changed so as to extract the image information obtained as a result, and fluctuations due to pan / tilt / zoom operations are corrected. The slit image obtained by applying the correction in this way is created, and the most recently created panoramic image is scrolled by the number of pixels of the width of the slit image just created (the horizontal width of the panoramic image is set to a constant value, and scrolling is performed. The part beyond that is discarded by), and paste the slit image just created into the space at the edge of the panoramic image created by scrolling. It is possible to create a correct corrected panoramic image in real time by sequentially performing a series of steps from creation of the slit image to pasting in time. The created panoramic images are sequentially created as soon as they are created.
Sent to 09.

Here, the speed fluctuation correcting means will be described. FIG. 4 shows the relationship between the images taken by the camera in the traveling direction while moving. Since the moving speed fluctuates, the position interval R ₁ (obtained from the moving speed) fluctuates between the time points T _i and T _{i + 1} . At this time, if the horizontal angle of view of the camera is θ _h , the range shown in the captured image is as shown in FIG. 4 in the view looking down from directly above the moving body. In order to generate a continuous panoramic image, it is necessary that the portions reflected in the slit image extracted from each frame are continuous, so assuming that the distance d to the subject and the slit image are extracted from the left end of the drawing. ,
Angle δ width w _i of the corresponding slit image _i is found to be necessary at time T _i. The number of pixels on the screen corresponding to the angle δ _i can be calculated from the horizontal angle of view θ _h and the number of pixels W _x of the screen in the horizontal direction by the following formula. By extracting the slit image with the slit image width w _i calculated in this way, it is possible to perform correction for speed fluctuations.

[0026]

[Equation 1] Further, from the above equation, when R _i is sufficiently smaller than d / (tan (θ _h / 2)), that is, when slit images are extracted at fine intervals, R _i << d / tan (θ _h / 2 ))

[0027]

[Equation 2] Therefore, the slit width proportional to the velocity can be approximately used.

Next, the camera operation correcting means will be described. FIG. 5 shows the relationship between the shooting conditions as the reference of the camera and the image when the pan angle and the zoom (focal length) are changed. The basic slit image extraction position is assumed to be an arbitrary position on the left half of the screen when the image is taken by a camera having a horizontal field angle θ _h with the front facing the traveling direction. The direction reflected in this basic slit image is defined as the basic direction of the slit image, and the angle with respect to the traveling direction is represented by φ. Here, the angle φ is 0 in the center of the screen, that is, in the traveling direction, and is positive in the clockwise direction. If you take φ on the left side of the screen,
φ will have a negative value. If the pan angle change amount ψ and the current horizontal angle of view θ _hc can be obtained when the pan angle and zoom (focal length) are changed, which part on the screen corresponds to the basic direction of the slit image? It can be easily calculated by the following equation. Here, the position px in the x direction on the screen is 0 at the center of the screen.
And the right direction is positive. The current focal length is f _c
And Then, the basic direction of the slit image is projected onto the position px in the x direction obtained by the following expression on the current screen.

[0029]

[Equation 3] Furthermore, the effect of zoom variation on the slit width can be corrected by the following equation. If the corrected slit width is w _i ',

[0030]

[Equation 4] By correcting the vertically elongated rectangular portion having the slit width w _i 'corrected with the position px in the x direction calculated as the center as a slit image, correction for the influence in the x direction from the fluctuation of the pan angle and the fluctuation of the zoom is performed. Can be carried out.

Next, tilt angle and zoom (focal length)
FIG. 6 shows the relationship of the images when is changed. Assuming that the reference tilt angle is the horizontal direction as an example, a change in the tilt angle η causes a change in the shooting range in the vertical direction as shown in FIG. Here, the vertical angle of view under the standard shooting conditions is θ _v , and the current vertical angle of view is θ _vc . At this time, in order to generate an image as if taken by a camera whose tilt angle has not been changed from an image taken by a camera whose tilt angle has been changed, a fixed distance r around the camera is used.
Set a virtual spherical surface on the screen, display the image obtained from the camera on the virtual screen in the figure, project the image on the virtual spherical surface, and then project the image projected on the spherical surface to the reference tilt angle. Project again on the virtual screen when there was a camera in. Here, the value of the following equation is used as r.

[0032]

[Equation 5] Further, the virtual screen referred to here is a rectangular shape formed by connecting four points where the four ends of the visible range, which are defined by the current tilt angle and zoom (focal length), intersect with the virtual spherical surface of radius r. It is a plane.

By this procedure, it is possible to obtain an image in which the influence in the y direction due to the fluctuation of the tilt angle η and the zoom is canceled. To realize this projection specifically,
Texture mapping of computer graphics (a technique to add (paste) a pattern (pattern etc.) defined separately to a smooth surface (curved surface etc.),
It is sufficient to use a method called "when commonly used when attaching an image pattern taken by a digital camera to a surface on computer graphics". An image obtained as a result of this processing is obtained as a slit image having a width w as a result of being properly processed even when the focal length at the time of shooting is different from the reference value and the image is cut out with the corrected slit width w ′. .

From the above description, it is possible to correct the pan / tilt / zoom operation and obtain a slit image as if it were taken by a camera in the reference state in which each operation is not performed. A correct panoramic image can be generated.

As a literature on texture mapping, "Practical computer graphics basic procedures and applications" (Davbid F. Rogers, edited by Fujio Yamaguchi, Seiko Electronic Industry Co., Ltd. Electronic Equipment Division, published by Nikkan Kogyo Shimbun, page 438- 450).

Further, if a slit image is extracted from the vicinity of the right end of each frame and an panoramic image is generated for an image photographed by a camera which is oriented substantially in the traveling direction, a panoramic image on the right side of the moving body is generated. it can. Similarly, if a slit image is extracted from the vicinity of the left end and a panoramic image is generated, a panoramic image for the left side can be generated. Therefore, a single video camera can be used to obtain a normal image and both left and right panoramic images. Obviously, a video camera dedicated to panorama generation may be installed on each of the left and right sides to acquire a dedicated image, and for example, a slit image may be extracted from around the center of each dedicated image to generate a panoramic image.

In the above description, the length of the slit image in the vertical direction is the number of pixels in the vertical direction of the image acquired by the image acquisition unit 101, but using a general method in computer graphics. You may enlarge / reduce. At this time, if the image is enlarged or reduced only in the vertical direction, the aspect ratio on the panoramic image changes, but this can be avoided by changing the width of the slit image to be extracted by the same ratio.

In the above, the method of generating a panoramic image based on an image from a perspective projection camera such as a normal video camera has been described. However, a panoramic image is captured using a line sensor camera, It is possible to realize the correction in the same manner as described above, and the panoramic image may be created by either method. For line sensor cameras,
Since the scan speed is usually much higher than the frame rate of a general video camera (30 frames / second), instead of adjusting the slit width according to the speed, the rate of thinning out the licensed data is set to the speed. It can be realized by adjusting according to the above, or by sending a trigger signal to the line sensor camera at an interval proportional to the speed, and the line sensor camera creating a slit image only when the trigger signal is received.

As is clear from the method of creating the panoramic image thus created, the faster the moving speed, the faster the scrolling. However, the scrolling speed is such that the slit image is created per unit time. It becomes the product of the frequency and the width of the slit image, and when the slit image is created 10 times per second, and the average slit image width during that time is 5 pixels, it becomes a scroll of 50 pixels per second, which is the width of the normal video screen. Considering that the number of pixels in the direction is 640, it takes about 13 seconds to scroll one screen. Further, considering that a display device which is usually used for a personal computer is one having a higher resolution and 1280 pixels in the horizontal direction, it is widely used until a certain object appears on the panoramic screen and disappears by scrolling. It is displayed for 20 seconds or more, and the scroll is slow enough for visual recognition.

The guidance information storage unit 108 stores the position (latitude, longitude, altitude, etc.) on the earth of the object to which the guidance information should be output and the contents of the guidance information, and can search / extract based on the position. is there. This can be realized by a known method used in a widely used car navigation system or the like. For guidance information, characters, images, H
Guide information and URLs written in languages such as TML
Addresses for referencing information from outside such as are stored.

The image composition processing unit 109 receives the position information obtained by the moving body position detecting unit 102, the traveling direction information obtained by the moving body traveling direction detecting unit 103, and the velocity information obtained by the moving body velocity detecting unit 104. , Camera direction detector 1
By using the camera direction information obtained by 05 and the focal length information obtained by the camera focal length detection unit 106, the region (visible region) displayed in the image obtained from the image acquisition unit 101 is perspective-projected. Of the objects for which the guidance information stored in the guidance information storage unit 108 is to be output, those in the visible region are extracted, and perspective projection is performed from the position on the earth of the object for which the extracted guidance information is to be output. By converting, the projected position on the image is calculated, and the guide position is combined with the calculated position on the image by, for example, characters,
The result is displayed on the display device 110. The above can be realized by a known method described in, for example, Japanese Patent Laid-Open No. 9-33271. At the same time, regarding the panoramic image input from the panoramic image generation unit 107, the positional information of the moving body at the time when the slit image forming the panoramic image is captured, the traveling direction information, and the relative relative to the moving body from which the slit image is extracted. Direction,
A visible area on the slit image is calculated, a target in which guidance information is to be output is extracted from the guidance information storage unit 108, guidance information is combined with the corresponding slit image portion by, for example, characters, and the result is displayed. It is displayed on the device 110. Furthermore, when further related information or more detailed information is requested for the displayed guidance information by a user operation, the related information or more detailed information stored in the guidance information storage unit 108 is combined. Displayed on the display device 110, or the communication control unit 11
The guide information is externally referred to via the Internet via the Internet 3, and the combined guide information is displayed on the display device 110.

FIG. 7 is a schematic diagram showing an example of the guide information displayed on the display device 110. FIG. 7 shows only an example of a portion where guidance information is superimposed and displayed on a photographed image, which is important in the present invention. However, the current position used in a normal car navigation system is displayed on a map. It goes without saying that it may be possible to use them by combining them with functions such as doing. The display device 110 is realized by a display device such as a liquid crystal display or a CRT. One display device may be used, or a plurality of display devices may be used in combination to improve usability and display area. The user operation input unit 111 uses, for example, a commonly used personal computer to input a camera operation (pan / tilt / zoom, etc.) by a general graphical user interface, and displays it on the display device 110. It is realized that the user gives an instruction to obtain more detailed information or related information of the guide information. Specifically, the designation may be realized by moving the mouse cursor with the mouse and pressing the mouse button, or may be realized by using another method such as a joystick or a touch panel.

The camera operation unit 112, for example, when the camera is installed on the electric camera platform, when the user gives an instruction to operate the camera (pan, tilt, zoom, etc.) via the user operation input unit 111. First, a desired operation is performed on the camera.

The communication control unit 113 connects to the Internet via an information center or an access point installed in a remote place by using a communication means such as a mobile phone, and is connected to the outside of the device (such as the information center or the Internet). Acquire the accumulated guidance information.

The image information storage unit 114 uses, for example, a data recorder and a video recorder, the image data and the panoramic image sent to the image composition processing unit 109, and the position / moving direction / moving speed / camera of the moving body. The various parameters of are stored in association with each other and can be used later. It can be fully realized by using the equipment widely used for industry and consumer.

In the configuration diagram of FIG. 1, when the camera direction (pan / tilt) is fixedly set to be the same as the moving body traveling direction or fixed so that the two always have a constant difference, the camera direction detecting unit 105 It can be omitted. Similarly, when the camera focal length is a fixed focal length, the camera focal length detection unit 106 can be omitted. Further, when both the camera direction and the camera focal length are fixedly installed, the camera operation unit 112 can be omitted. Further, the communication control unit 113 is required when using the guide information other than that stored in the device, but in other cases, the communication control unit can be omitted. Furthermore, when the service that uses image data or the like is not performed offline, the image information storage unit 114 can be omitted.

Next, the general operation of the information providing apparatus of this embodiment will be described with reference to the flowchart of FIG. The image acquisition unit 101, the moving body position detection unit 102, the moving body traveling direction detection unit 103, the moving body speed detection unit 104, the camera direction detection unit 105, and the camera focal length detection unit 106 respectively capture the real image data, the moving body position, and the movement. The body traveling direction, the moving body speed, the camera direction, and the camera focal length are acquired or acquired (steps 201 to 206). Further, the user inputs a camera operation (pan / tilt / zoom, etc.) and desired guidance information from the user operation input unit 111 (step 207). The photographed image data acquired by the image acquisition unit 101 is sent to the image composition processing unit 109 and the panoramic image generation unit 107. Position information obtained by the moving body position detecting unit 102, traveling direction information obtained by the moving body traveling direction detecting unit 103, speed information obtained by the moving body velocity detecting unit 104, and camera direction detecting unit 105. The camera direction information and the focal length information obtained by the camera focal length detection unit 106 are sent to the image synthesis processing unit 109, respectively. Further, the speed information obtained by the moving body speed detection unit 104, the camera direction information obtained by the camera direction detection unit 105, and the focal length information obtained by the camera focal length detection unit 109 are sent to the panoramic image generation unit 107. . Panorama image generation unit 10
7, a panoramic image is generated in real time based on the image data, speed information, camera direction information, and focal length information sent, and sent to the image synthesis processing unit 109 (step 20).
8) Further, the camera operation and guidance information input by the user operation input unit 111 is respectively the camera operation unit 11
2 and the image composition processing unit 109 to be processed.
The image composition processing unit 109 processes information obtained from each unit, reads necessary guide information from the guide information storage unit 108, and obtains guide information from the outside via the communication control unit 113 as necessary, An image obtained by superimposing guide information on the real image and the panoramic image is combined (step 209), and the resulting image is displayed on the display device 110 (step 211). The image information storage unit 114 receives and stores the acquired image and the generated panoramic image together with various detected information (step 21).
2).

Next, the procedure for creating a panoramic image will be described with reference to FIGS.

First, as initialization processing, various variables and internal buffers are initialized (step 301). next,
A slit image is generated from the currently captured video frame using each of the detected numerical values (position / speed of the moving body, etc.) (step 302). Next, it is determined which of the left and right sides the created slit image is with respect to the front of the video camera (step 303). The panoramic image stored in the internal buffer is scrolled rightward or leftward by the width w of the slit image created in the previous stage (steps 304 and 306), and the internal buffer obtained by the scrolling process in the previous stage is stored. In space, step 302
The slit image obtained in step 3 is copied to obtain the current panoramic image (step 307). Finally, it is determined whether the system is terminated (step 308).

Next, the contents of the slit image forming routine (step 302) will be described with reference to FIG.

First, the current speed is obtained from the moving speed detecting unit 104 (step 401). Next, the slit width w is calculated by the calculation formula (1) or (2) (step 40).
2). The camera focal length detector 106 obtains the current focal length of the camera (step 403). Camera detector 1
05 to get the current pan angle of the camera (step 40
4). The position where the slit image is cut out is calculated from the current pan angle and focal length by the calculation formula (4) using the coordinate system with the moving image frame center set to 0 (step 40).
5). The current focal length is compared with the reference focal length, and if there is a deviation, the corrected slit width w ′ is calculated by the calculation formula (5) (steps 406 and 407). A rectangular portion having a width w or the slit width w ′ corrected in step 407 centering on the cutout position obtained in step 405 is cut out as a slit image from the moving image frame (step 408). The camera direction detection unit 105 obtains the current tilt angle of the camera (step 409). If the current tilt angle is compared with a predetermined reference tilt angle (step 410) and the current focal length is compared with a predetermined reference focal length (step 411), either of them is different. For this, correction is performed by texture mapping (step 412). As a result, a slit image with a width w is obtained from an image captured with the tilt angle and zoom angle changed, as if the camera was captured with the reference tilt angle and zoom angle set.

Next, "scroll panoramic image" and "slit image" in FIG. 9 are set at the left end (right end) of the panoramic image.
The process of "paste" will be described with reference to FIG. In addition, () is a case where the created slit image is on the left side.

First, a work buffer for storing image data whose vertical length is equal to the vertical size of the panoramic image and whose horizontal length is shorter than the horizontal size of the panoramic image by the slit width w is secured (step 501). Next, an area having a slit width w shorter than the horizontal width of the panoramic image and having a vertical width equal to that of the panoramic image is read from the internal buffer for the panoramic image, left-justified (right-justified), and block-copied to the work buffer (step 502). The image in the working buffer is block-aligned to the internal buffer for storing the panoramic image by right-justifying (left-justifying) (step 503). The slit image (horizontal width w, vertical width is equal to that of the panoramic image) is block copied to the internal buffer for storing the panoramic image by left justification (right justification) (step 504).
Finally, the working buffer is released (step 50).
5).

Next, an example of creating a panoramic image will be described.

First, an example of creating a panoramic image on the right side will be described. The conditions in this example are set as follows.

Reference horizontal angle of view θ _h = 60 degrees Reference focal length = 554.2563 Number of horizontal pixels on video screen W _x = 640 Slit image creation interval = 1/30 seconds Distance to subject d = 6 m Slit image Basic angle φ = 20 degrees Tilt angle of reference = 0 degrees Mobile vehicle speed detection unit 104 present speed = 20 km /
h is obtained (step 401). The slit width w = 5.6024 (about 6 pixels) is obtained by the calculation formula (1) or (2) (step 402). The camera focal length detection unit 106 obtains the current focal length = 686.242 and the current horizontal angle of view = 50 degrees (step 403).
The camera direction detection unit 105 obtains the current pan angle ψ = −3 degrees (step 404). According to the calculation formula (4),
Position where the slit image is cut out px = 291.292≈2
91 is obtained (step 405). Since the current focal length is different from the reference focal length, the correction slit width w '=
6.9365 (about 7 pixels) is calculated by the calculation formula (5) (steps 406 and 407). Since the horizontal size of the video image is 640, px = 291, that is, a slit image having a horizontal width of 7 pixels is cut out with the X coordinate in the right direction of 291 pixels from the center as the center (step 408). The camera direction detection unit 105 obtains the current tilt angle of the camera = 0 degree (step 409). The current tilt angle 0 degrees is the same as the reference tilt angle (0 degrees), and the current focal length 66
Since 8.242 is different from the reference focal length (554.2563), correction by texture mapping is performed (steps 410 to 412). Then, since the angle φ> 0 from the advancing direction of the basic direction of the slit image and the right side, the panoramic image stored in the internal buffer is scrolled to the right by the width w = 6 pixels of the slit image (step 304), the slit image obtained previously is copied to the space (left end) of the internal buffer obtained by the scroll processing, and the current panoramic image (FIG. 7C) is obtained.

Next, an example of creating a panoramic image on the left side will be described. The condition in this example is −20 degrees, which is different from the example of the panoramic image creation on the right side, which is different from the previous example, only in the angle φ that is the basis of the slit image.

In creating the slit image, the position for cutting out the slit image is px = 209.8202.apprxeq.-210, and the point of cutting out the slit image having a width of 7 pixels centering on the X coordinate 210 pixels leftward from the center is the previous point. This is different from the case of creating a panoramic image on the right side. Also, since the angle φ <0 from the advancing direction of the basic direction of the slit image and the left side, the width w of the slit image created by the slit image creation routine: 6 pixels is stored in the internal buffer. The existing panoramic image is scrolled in the left direction, and the previously obtained image is copied to the space (right end) of the obtained internal buffer to obtain the current panoramic image (FIG. 7 (2)).

The processes shown in FIGS. 9 to 11 are executed as a panoramic image generation program using a floppy disk,
It is also possible to record in a recording medium such as a CD-ROM or a magneto-optical disk, and read the panoramic image generation program into the hard disk by the data processing device (CPU) to execute it.

The information providing apparatus shown in FIG. 1 also uses the processes of the panorama image generating section 107 and the image synthesizing processing section 109 as information providing programs, such as a floppy disk and a CD.
It is also possible to record it in a recording medium such as a ROM or a magneto-optical disk, read it into a hard disk by a data processing device (CPU) and execute it.

In the above embodiment, the case where the present invention is applied to the automobile has been described as an example, but the present invention is not limited to the automobile and may be applied to a ship, an aircraft, a helicopter and the like. It may also be applied to a portable electronic map or the like.

[0062]

As described above, the present invention has the following effects.

1) According to the inventions of claims 1 to 6, the left and right images can be generated by one camera directed in the traveling direction of the moving body with a small amount of image calculation, and the moving speed of the moving body changes. Even with this, a panoramic image with little distortion can be obtained.

2) The inventions of claims 7, 8, 13, and 19 are
Because a panoramic image of the side of the moving body (left or right or one side) is created in real time using a video camera, etc., and the guidance information is superimposed and displayed on the panoramic image to scroll at a relatively low speed. Since the characteristics of the panoramic image have good separability even for subjects that are easy to see and are dense, it is possible to provide guidance information to the driver or passengers on a panoramic screen that is easy to see, dramatically improving the visibility of the guidance information. You can

3) According to the invention of claim 9, the parameters (pan, tilt, zoom, etc.) of the camera taking the image are detected, and the guide information is superimposed and displayed at the correct position using the detected parameters. Together with this, a natural panoramic image of the side of the moving body is created as an image based on the detected parameters, and the guidance information is superimposed and displayed on the panoramic image, which enables the user to operate the camera.
It is possible to provide more flexible navigation and information provision.

4) According to the invention of claim 10, the user indicates the guide information displayed on the screen, and the related information or more detailed information registered in advance is displayed. And, it becomes possible to access various kinds of related information, and the information available to the user is significantly increased.

5) According to the eleventh and seventeenth aspects of the present invention, the related information or more detailed information is immediately obtained from a remote information center or the Internet and displayed by using a communication means such as a mobile phone. As a result, it becomes possible to instantly access the information accumulated in the information center and the information published on the Internet, and the amount and quality of the information that the user can obtain are both dramatically improved. .

6) According to the twelfth and eighteenth aspects of the present invention, the photographed video and the generated panoramic image are sequentially recorded in association with the detected position, traveling direction, and speed of the moving body, so that the moving image is recorded along the moving route. It is possible to accumulate landscape information and provide new information using it.

[Brief description of drawings]

FIG. 1 is a configuration diagram of an information providing device according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram of a method of installing a line sensor camera.

FIG. 3 is an explanatory diagram of creating a panoramic image.

FIG. 4 is a diagram showing a relationship between images taken by a camera oriented in a traveling direction.

FIG. 5 is a diagram showing a relationship between a pan angle and an image.

FIG. 6 is a diagram showing a relationship between a tilt angle and an image.

FIG. 7 is a diagram showing a guide information display example.

8 is a flowchart of the overall processing of the embodiment of FIG.

FIG. 9 is a flowchart showing a procedure for creating a panoramic image.

FIG. 10 is a flowchart of a slit image creation routine.

FIG. 11 is a flowchart showing details of the processes of “scrolling panoramic image” and “pasting slit image to left (right) end of panoramic image” in FIG. 9.

[Explanation of symbols]

101 image acquisition unit 102 moving body position detection unit 103 moving body traveling direction detection unit 104 moving body speed detection unit 105 camera direction detection unit 106 camera focal length detection unit 107 panoramic image generation unit 108 guide information storage unit 109 image composition processing unit 110 Display device 111 User operation input unit 112 Camera operation unit 113 Communication control unit 114 Image information storage units 201 to 212, 301 to 308, 401 to 412, 5
01-505 steps

─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Takeshi Ikeda 3-19-2 Nishishinjuku, Shinjuku-ku, Tokyo Nihon Telegraph and Telephone Corporation (56) Reference JP 2000-155839 (JP, A) JP 10-267671 (JP, A) JP-A-9-33271 (JP, A) JP-A-10-132598 (JP, A) JP-A-8-273000 (JP, A) (58) Fields investigated (Int .Cl. ⁷ , DB name) H04N 5/262-5/28 G06T 1/00 G03B 37/00

Claims (20)

(57) [Claims] 1. A panoramic image generation method for generating a panoramic image from an image picked up by a video camera installed on a moving body, the vertical rectangle having a minute width at an arbitrary position and having the same length as the vertical size of a frame. The width of the slit image, which is an image of a portion, is defined as the image signal frame from which the slit image is extracted by the video.
It changes according to the moving speed of the moving body at the time of shooting with a camera, and even when the pan / tilt / zoom of the video camera changes, a certain direction is taken with respect to the moving direction of the moving body. Changing the extraction position so as to extract the image information obtained by creating the slit image, and determining whether the created slit image is on the left or right side with respect to the front of the video camera. , A strip-shaped image formed by pasting the slit images along with the passage of time, and a panoramic image created most recently,
If the slit image is on the right side, the step of scrolling to the right if the slit image is on the right side, and to the left if the slit image is on the left side, and A panoramic image generation method including a step of pasting to the left end of the panoramic image if the image is on the right side, and to the right end of the panoramic image if the slit image is on the left side, and repeating these steps until an end instruction is input. . Wherein the step of creating the slit image, and obtaining a current moving velocity of the moving body, calculating a slit width of the slit image on the basis of said current moving speed, current pan Calculating the position where the slit image is cut out from the angle and the focal length by a coordinate system in which the center of the moving image frame is 0; and correcting the slit width when the focal length fluctuates from a reference value. And a step of cutting out a rectangular portion having the slit width centering on the calculated cutting position as a slit image from the current frame of the moving image, and the current tilt angle is different from a predetermined reference tilt angle. And whether the current focal length is different from the predetermined reference focal length, and the current tilt angle in the previous stage, When at least one of the focal length of the standing are different, comprising performing the correction by texture mapping with respect to the slit image, the panoramic image generation method of claim 1, wherein. 3. A panoramic image generating apparatus for generating a panoramic image from an image taken by a video camera installed on a moving body, said means for obtaining the current moving speed of said moving body, and a minute width at an arbitrary position. the width of the slit image, said images signal frame in which the slit image is extracted video is in length image of vertical size and same elongated rectangular portion of the frame
Varied according to the moving speed of the moving body when taken with the camera and said in the case of pan, tilt and zoom of the video camera is changed, by photographing a certain direction with respect to the moving direction of the movable body A process of creating the slit image by changing the extraction position so as to extract the obtained image information, and a process of determining which of the left and right sides the created slit image is with respect to the front of the video camera,
A strip-shaped image created by pasting the slit images along with the passage of time, the panoramic image created in the vicinity,
The width of the created slit image is scrolled to the right if the slit image is on the right side, and to the left if the slit image is on the left side, and the created slit image is processed to the slit. A panoramic image generation device having means for generating a panoramic image by repeating a process of pasting to the left end of the panoramic image if the image is on the right side, and to the right end of the panoramic image if the slit image is on the left side. 4. The process of creating the slit image, the process of calculating the slit width of the slit image based on the current moving speed, and the position of cutting out the slit image from the current pan angle and focal length. , A process of calculating with a coordinate system in which the center of the moving image frame is 0, a process of correcting the slit width when the focal length varies from a reference value, and a center of the calculated cutting position As a process of cutting out the rectangular portion of the slit width as a slit image from the current frame of the moving image, whether the current tilt angle is different from a predetermined reference tilt angle, and the current focal length is predetermined. In the previous step, at least one of the current tilt angle and current focal length is different from the process of determining whether it is different from the reference focal length. 4. The panoramic image generation device according to claim 3, further comprising the step of performing correction by texture mapping on the slit image. 5. A panoramic image generation program for generating a panoramic image from an image picked up by a video camera installed on a moving body, wherein the vertical rectangle has a minute width at an arbitrary position and the same length as the vertical size of the frame. The width of the slit image, which is an image of a portion, is defined as the image signal frame from which the slit image is extracted by the video.
According to the moving speed of the moving body when taken with the camera is varied, also in the case of pan, tilt and zoom of the video camera is changed even taking a certain direction with respect to the moving direction of the movable body And a process of creating the slit image by changing the extraction position so as to extract the image information obtained as a result, and a process of determining which of the left and right sides the created slit image is with respect to the front of the video camera. , A strip-shaped image formed by pasting the slit images along with the passage of time, and a panoramic image created most recently,
The width of the created slit image, if the slit image is on the right side, to the right direction, if the slit image is on the left side, the process of scrolling to the left, and the created slit image, the slit If the image is on the right side, the process includes pasting to the left end of the panoramic image, and if the slit image is on the left side to the right end of the panoramic image, the computer repeats these processes until an end instruction is input. A recording medium recording a panoramic image generation program to be executed. 6. A process of creating the slit image, the process of obtaining the current moving velocity of the moving body, a process of calculating the slit width of the slit image on the basis of said current moving speed, current pan A process of calculating the position where the slit image is cut out from the angle and the focal length by a coordinate system in which the center of the moving image frame is 0, and when the focal length varies from a reference value, the slit width is corrected. And a process of cutting out the rectangular portion of the slit width centered on the calculated cutting position as a slit image from the current frame of the moving image, and the current tilt angle is different from the predetermined reference tilt angle. And whether the current focal length is different from the predetermined reference focal length, and the current tilt angle in the previous step, When at least one of the focal length of the standing are different, including the process of performing correction by texture mapping with respect to the slit image, claim 5 recording medium according. 7. A step of acquiring actually shot image data by a video camera mounted on a moving body, and a step of detecting a camera direction and a focal length of the video camera.
When the steps of detecting the current position and the traveling direction and the moving speed of the moving body, the length in the minute width of the arbitrary position of the width of the slit image of a vertical size and same elongated rectangular portion of the frame the video image signal frame in which the slit image is extracted
It changes according to the moving speed of the moving body when shooting with the camera, and even when the pan / tilt / zoom of the video camera changes, a certain direction is taken with respect to the moving direction of the moving body. Create the slit image by changing the extraction position to extract the image information obtained,
The panorama image on the side of the moving object is scrolled by scrolling the most recently created panorama image by the pixels of the width of the slit image created this time, and pasting the slit image created this time to the width of the panorama image created by scrolling. In real time, determining guide information of the panoramic image from the detected position, traveling direction , and moving speed , and superimposing the guide information on the panoramic image, An information providing method having a step of displaying. 8. The step of creating the slit image includes the step of calculating the slit width of the slit image based on the current moving speed, and the position of cutting out the slit image from the current pan angle and focal length. , A step of calculating with a coordinate system with the center of the moving image frame set to 0, a step of correcting the slit width when the focal length fluctuates from a reference value, and a step of centering the calculated cutting position As a step of cutting out the rectangular portion of the slit width as a slit image from the current frame of the moving image, whether the current tilt angle is different from a predetermined reference tilt angle, and the current focal length is predetermined. At least one of the current tilt angle and current focal length is different in the step of judging whether it is different from the reference focal length and in the previous step. The method according to claim 7, further comprising the step of performing a texture mapping correction on the slit image. 9. A natural panoramic image of a side of a moving body is generated as an image based on the detected camera direction and focal length, and guide information is displayed at a correct position based on the camera direction and focal length . The method according to claim 7 or 8. 10. The method according to claim 7, wherein related information or detailed information registered in advance is superimposed and displayed on the panoramic image according to a user's instruction. 11. The method according to claim 7, wherein the guide information is obtained by using a communication means, and displayed on the panoramic image.
The method according to any one of 0. 12. The image data and the panoramic image obtained by the video camera are sequentially stored in association with the detected position, traveling direction and moving speed of the moving body.
A method according to any one of claims 7 to 11. 13. A video camera mounted on a moving body, and a camera direction detector for detecting the camera direction of the video camera.
Output means and camera focus detection for detecting the focal length of the video camera
Means, a moving body position detecting means for detecting the position of the movable body, a movable body traveling direction detecting means for detecting the moving direction of the moving body, the moving body speed detecting means for detecting a moving speed of the moving body the width of the slit image of a vertical size and same elongated rectangular portion of the frame length at minute width at an arbitrary position, the image signal frames the video which the slit image is extracted
According to the moving speed of the moving body when taken with the camera is varied, also in the case of pan, tilt and zoom of the video camera is changed even taking a certain direction with respect to the moving direction of the movable body The slit image is created by changing the extraction position so as to extract the image information obtained by scrolling the recently created panoramic image by the pixels of the width of the slit image created this time. A panoramic image creating unit that creates a panoramic image on the side of the moving object in real time by setting the pace of the slit image to the width of the panoramic image created by scrolling, and the panorama from the detected position, traveling direction, and moving speed. Image combining means for determining guide information of the image and superimposing the guide information on the panoramic image; and a display for displaying the superimposed pixels. Information providing device having the b unit. 14. The panoramic image creating means creates a natural panoramic image of a lateral side of a moving object as an image based on the detected camera direction and focal length, and the image synthesizing means creates the camera direction and focal length. Guide information is displayed at the correct position on the display means based on
The described device. 15. The apparatus according to claim 13, further comprising guide information and a guide information storage unit that stores the position of the moving body when displaying the guide information. 16. A user operation input means for inputting an operation of the video camera, or for a user to instruct a more detailed information of the guidance information displayed on the display means and / or a portion where the user wants to see related information. 16. The device according to any one of claims 13 to 15, further comprising: 17. The communication device according to claim 13, further comprising a communication unit for acquiring the guide information stored outside the device.
6. The device according to any one of 6 above. 18. The apparatus according to claim 13, further comprising image information storage means for storing the panoramic image, the position / moving direction / moving speed of the moving body, and various parameters of the camera. 19. A video camera mounted on a moving body.
Processing for acquiring the actual image data, and processing for detecting the camera direction and focal length of the video camera.
And the current position, traveling direction and moving speed of the moving body.
That process and, the width of the slit image of a vertical size and same elongated rectangular portion of the frame length at minute width at an arbitrary position, the image signal frames the video the slit image is extracted
According to the moving speed of the moving body when taken with the camera is varied, also in the case of pan, tilt and zoom of the installed in the mobile the video camera is changed, with respect to the moving direction of the movable body The slit image is created by changing the extraction position so as to extract the image information obtained by shooting a certain direction, and the panorama image created in the vicinity is scrolled by the pixel width of the slit image created this time. Then, by setting the slit image created this time as the width of the panoramic image created by scrolling, a panoramic image creation process for creating a panoramic image on the side of the moving object in real time, and the detected position and traveling direction. and wherein the movement speed determines guidance information of the panoramic image, and the image synthesis processing for superimposing the guidance information on the panorama image, superimposed image The recording medium recording the information provision program for executing processing for displaying on the display unit to the computer. 20. A process of creating the slit image, a process of calculating a slit width of the slit image based on the current moving speed, and a position of cutting out the slit image from a current pan angle and a focal length. , A process of calculating with a coordinate system in which the center of the moving image frame is 0, a process of correcting the slit width when the focal length varies from a reference value, and a center of the calculated cutting position As a process of cutting out the rectangular portion of the slit width as a slit image from the current frame of the moving image, whether the current tilt angle is different from a predetermined reference tilt angle, and the current focal length is predetermined. If the process determines whether the focal length is different from the reference focal length, and if at least one of the current tilt angle and current focal length differs in the previous step, 20. The recording medium according to claim 19, further comprising a process of performing a correction by texture mapping on the slit image when the recording medium has been detected.