JP4046973B2 - Information processing method and image mixing apparatus - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method for mixing an image generated from panoramic image data held in advance with a photographed image photographed by a photographing unit.
[0002]
[Prior art]
In a place with a good view, such as an observatory, you may see the scenery spreading out in front of you. At this time, as a device for more clearly observing the scenery, there is a telescope that optically enlarges and presents the scenery in front of you.
[0003]
There is also an apparatus for photographing the real world in real time with a camera or the like, and observing the real world through the captured image, and an apparatus that applies this to a telescope is also disclosed in, for example, Japanese Patent Laid-Open No. 2001-94971 “Optical apparatus”. Has been.
[0004]
[Problems to be solved by the invention]
A problem when viewing outdoor scenery is that the real world cannot be clearly observed depending on the weather and time zone when using the device. For example, when looking at the surroundings using a viewing mirror on the rooftop of a high-rise building, it may happen that the building in the landscape is hidden by clouds or haze when it is cloudy. Also, at night, the area will be dark and you will not be able to observe the scenery. For example, in the apparatus disclosed in Japanese Patent Application Laid-Open No. 2001-94971 “Optical apparatus”, a plurality of captured images obtained when time, season, or the like changes are stored. An image is selected and displayed instead of the image being shot. This is a clear observation of the scenery seen from the place.
[0005]
However, what can be observed with this method is a landscape at a completely different time, and it has not been possible to clearly observe the landscape in front of you.
[0006]
Conventionally, there is an apparatus that synthesizes an actual landscape and a computer image by mounting an HMD (head mounted display) incorporating a camera on the head and following the movement of the head. With this device, landscape simulation based on the scenery in front of you is realized.
[0007]
In this case, a method of drawing a computer image on a real landscape is generally used. Then, when using a computer image that occupies a relatively large area on the observation screen, there are fewer portions where the actual landscape can be seen, and there is a problem that it is difficult to grasp the positional relationship between the actual landscape and the computer image.
[0008]
The present invention has been made in view of the above points, and an object thereof is to clarify a display image by mixing a generated image with a real-world photographed image photographed by a photographing unit.
[0009]
[Means for Solving the Problems]
In order to achieve the above object, the present invention has the following configuration.
[0010]
The invention of claim 1 of the present application is an information processing method for mixing an image generated from panoramic image data held in advance with a captured image captured by a capturing unit, the captured image captured by the capturing unit, and The posture information of the photographing unit is input, panoramic image data based on a photographed image taken at another time is held, and an image corresponding to the photographed image is generated from the panoramic image data based on the posture information of the photographing unit. Then, the photographed image is analyzed to obtain a mixing condition, and the generated image is mixed with the photographed image using the mixing condition.
[0011]
The invention of claim 8 of the present application is an information processing method for mixing an image generated from data held in advance with a captured image captured by the capturing unit, the captured image captured by the capturing unit and the captured image Input posture information, generate an image corresponding to the photographed image from pre-stored data based on the posture information of the photographing unit, analyze the brightness of the photographed image, obtain a mixing condition, The generated image is mixed with the captured image using the mixing condition.
[0012]
The invention of claim 12 of the present application is based on the photographing means for photographing the real world, the posture measuring means for measuring the posture of the photographing unit, and the posture information of the photographing unit, from the data held in advance to the photographed image. Generating means for generating a corresponding image, means for analyzing the brightness of the captured image and obtaining a mixing condition, and mixing means for mixing the generated image with the captured image using the mixing condition It is characterized by having.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.
[0014]
<First embodiment>
The outline of the first embodiment is as follows. FIG. 1 is a block diagram showing a schematic configuration of an image mixing apparatus according to a first embodiment of the present invention.
[0015]
Reference numeral 1 denotes an image pickup unit, which includes, for example, a camera. The imaging unit 1 captures the real world such as an outdoor landscape and outputs the captured image to the captured image capturing unit 2 as an image signal. Reference numeral 2 denotes a captured image capturing unit that converts an image signal output from the imaging unit 1 into a format suitable for the image mixing unit 3 and sends the image signal to the image mixing unit 3. An image mixing unit 3 mixes the captured image from the captured image capturing unit 2 and the generated image from the mixed image generation unit 7 and sends them to the display unit 9.
[0016]
A support unit 4 is installed on the floor or the like and supports the imaging unit 1. The support unit 4 has a rotationally movable part, and allows the user to manually rotate the imaging unit 1 freely in the left-right direction and the up-down direction. The support unit 4 may be fixed to a floor or the like, but may be moved on a rail or a carriage, for example.
[0017]
Reference numeral 5 denotes an attitude measurement unit, which receives attitude information of the imaging unit 1, that is, rotation information of the imaging unit 1 in the vertical and horizontal directions in response to a request from the mixed image generation unit 7 or a request from the mixed image generation unit 7. Without being sent to the mixed image generating unit 7. The attitude measurement unit 5 is composed of, for example, a mechanical encoder. In addition, for example, it is conceivable to use a gyro sensor or an optical sensor. Further, a combination of a plurality of sensor devices may be used.
[0018]
Reference numeral 6 denotes a lens state control unit which supplies lens state information such as a zoom value and a focus value of a lens used in the imaging unit 1 in response to a request from the mixed image generation unit 7 or without a request from the mixed image generation unit 7. The image is sent to the mixed image generation unit 7. Here, the zoom value / focus value are values output by the encoder or the like in the lens state control unit 6. For example, when the lens state control unit 6 includes an 8-bit encoder, the output value is The value is in the range of 1 to 256. The lens state control unit 6 may include, for example, a microcomputer, calculate the focal length of the lens using a value output by an encoder or the like, and output the calculated value as lens state information. The lens state control unit 6 operates the lens state by a control unit and a control signal (not shown). The lens state control unit 6 includes, for example, a mechanical encoder or a stepping motor.
[0019]
Reference numeral 7 denotes a mixed image generation unit which extracts posture information of the image pickup unit 1 from the posture measurement unit 5 to estimate the horizontal angle and the vertical angle that the image pickup unit 1 is facing, and from the lens state control unit 6 the lens state. Information is extracted and the angle of view of the imaging unit 1 is estimated. Since the visual field of the imaging unit 1 can be obtained if the horizontal direction angle and the vertical angle and the angle of view to which the imaging unit 1 is directed can be estimated, the mixed image generation unit 7 extracts data corresponding to the visual field of the imaging unit 1 from the data unit 8. . The extracted data is sent to the image mixing unit 3.
[0020]
Reference numeral 8 denotes a data portion, which is composed of, for example, a hard disk and holds a generated image to be transferred to the mixed image generating portion 7. Data stored in the data unit 8 is, for example, text data, three-dimensional CG (computer graphics) data, a two-dimensional CG image, and a real panorama image.
[0021]
The data unit 8 sends appropriate data to the mixed image generation unit 3 in response to a request from the mixed image generation unit 7. For example, when the mixed image generating unit 3 requests 3D CG data to be mixed with the field of view of the imaging unit 1, the data unit 8 includes the 3D included in the field of view of the imaging unit 1 from the stored 3D CG data. CG data is extracted and transmitted. Also, for example, when the mixed image generation unit 3 requests a real panorama to be mixed with the field of view of the imaging unit 1, an image close to an image that will be captured in the same posture as the imaging unit 1 at the moment when the request is generated is displayed. Take out the live-action panorama image including it and send it out. The data unit 8 is not limited to a hard disk, and any medium that can hold data may be used. For example, the data unit 8 may be composed of a tape or a memory.
[0022]
Reference numeral 9 denotes a display unit which displays the image signal sent from the image mixing unit 3. At this time, if the display unit 9 moves in conjunction with the imaging unit 1, it is intuitive and easy for the user to look into the telescope and the like, but this is not always necessary. 1 may be movable and the display unit 9 may be fixed. In addition, a plurality of display units 9 may be provided. In this case, a large number of people can see the mixed image.
[0023]
The control of the present embodiment having the above configuration will be described below. FIG. 2 is a flowchart for explaining a processing procedure in the image mixing apparatus of the present invention.
[0024]
In step S0, data to be stored in the data portion 8 is prepared. This data is, for example, a real panorama image obtained by stitching together images taken at time t (= t1, t2,...) Using the imaging unit 1 before using this apparatus. In this case, as shown in FIG. 10, an image captured by the imaging unit 1 is captured by the captured image capturing unit 2, and the captured image is sent to the data unit 8. A panoramic image can be captured by the imaging unit 1 and stored in the data unit 8 as described above, but any method may be used as long as the panoramic image can be stored in the data unit 8. In order to create a real panorama image, an image taken at time t using the imaging unit 1 is hereinafter referred to as an original image.
[0025]
Among the pixels constituting the real panorama image, the original image is a pixel in a predetermined column, for example, the first column from the left, and the horizontal direction of the imaging unit 1 when the pixel is photographed An angle is associated. Similarly, among the pixels constituting the real panorama image, the original image is a pixel in a predetermined row, for example, the pixel in the first row from the top, and the imaging unit 1 when the pixel is photographed Are associated with each other. Here, the horizontal angle and the vertical angle associated with the actual panorama image are estimated by the image mixing unit 7 from the posture information of the imaging unit 1 measured by the posture measurement unit 5 at the time t when the original image was captured. is there.
[0026]
From the actual panorama image as described above, when a certain horizontal angle and vertical angle that the imaging unit 1 can take are specified, an image close to an image that the imaging unit 1 would have captured in its posture at a time close to time t. Can be cut out from a live-action panoramic image. That is, the image P photographed at time t1 with the horizontal angle of the imaging unit 1 being 0 degrees and the vertical angle of 0 degrees, and the attitude of the imaging unit 1 with the horizontal angle of 30 degrees and the vertical angle of 0 degrees When a real panorama image is created using the image Q taken at time t2 as an original image and the pixels located in the first column in the image P and the image Q can be identified from the pixels in the real panorama image, A pixel column corresponding to the first column of an image close to an image that the imaging unit 1 would have taken at a time close to time t1 and time t2 in a posture with a horizontal angle of 15 degrees and a vertical angle of 0 degrees is the first column in the image P. It can be calculated as a column that internally divides the pixel column that was the first column and the pixel column that was the first column in the image Q into 1: 1 (= 15-0: 30-15). Take a picture from a live-action panoramic image so that the calculated row is the first row. If the same range as the angle of view of the section 1 is cut out, the cut out image is an image close to an image that would have been taken with a posture with a horizontal angle of 15 degrees and a vertical angle of 0 degrees at times close to time t1 and time t2. Become. If there is no column that exactly divides the pixel column that was the first column in the image P and the pixel column that was the first column in the image Q into 1: 1, that is, there is exactly In the case of an internal dividing line, the same range as the angle of view of the imaging unit 1 may be cut out from the actual panorama image so that one of the two image sequences in contact with the internal dividing line becomes the first column. In this example, the original image and the cut-out image have the same vertical angle of 0 degrees, but the same method can be applied even if they differ.
[0027]
In addition, it has been stated that if a certain horizontal angle and vertical angle that the imaging unit 1 can take are specified, an image close to an image that will be captured in that posture can be cut out. It is premised that the horizontal direction angle and the vertical direction angle that are included in the live-action panoramic image prepared in step 0 are specified. For example, from a real panorama image created only from the images P and Q, the image is taken at a time close to the time t in a posture in which the horizontal angle of the imaging unit 1 is 0 degrees to 30 degrees and the vertical angle is 0 degrees. Only images that are likely to be close to the image can be cut out.
[0028]
Further, the actual panorama image is obtained by associating the angle of view of the imaging unit 1 at the time of shooting estimated by the image mixing unit 7 from the lens control information extracted from the lens state control unit 6 at the time t when the original image was shot. is there. That is, when a certain angle of view is designated, an area captured when zooming to the designated angle of view can be cut out from this image with the associated angle of view as a reference.
[0029]
Further, the actual panorama image may be obtained by performing image processing such as color tone correction and contour enhancement on the panoramic image photographed using the imaging unit 1.
[0030]
For example, text data, three-dimensional CG data, two-dimensional CG image, and the like may be stored in the data unit 8 in addition to the actual panorama image.
[0031]
Data indicating these virtual worlds needs to correspond to the three-dimensional coordinates in the real world in advance. For example, the three-dimensional CG must specify which three-dimensional position in the real world corresponds to, and if it is a two-dimensional CG image, from which position and in what direction it is viewed.
[0032]
After data preparation, the system is started in step S1. In step S2, an image is acquired from the imaging unit 1, and the acquired captured image is sent to the captured image capturing unit 2 as, for example, an NTSC image signal. At this time, the captured image sent from the imaging unit 1 to the captured image capturing unit 2 is not limited to the NTSC image signal, and any image can be used as long as the image can be expressed.
[0033]
The image captured by the imaging unit 1 is converted into an appropriate format by the captured image capturing unit 2 and sent to the image mixing unit 3. The captured image capturing unit 2 converts, for example, an NTSC image signal sent from the imaging unit 1 into digital data and sends the digital data to the image mixing unit 3. However, the image sent by the imaging unit 1 is not limited to the NTSC image signal, but an image. Any image can be used as long as it can be expressed, and the captured image sent from the captured image capturing unit 2 to the image mixing unit 3 is not limited to digital data, and any image can be processed as long as the image mixing unit 3 can process the image. Good.
[0034]
In step S <b> 3, the posture measurement unit 5 detects the posture of the imaging unit 1, and the detected posture information is sent to the mixed image generation unit 7. In step S <b> 4, the lens state control unit 6 detects lens state information such as a zoom value and a focus value from the lens attached to the imaging unit 1, and the detected lens state information is sent to the mixed image generation unit 7.
[0035]
In step S <b> 5, the mixed image generation unit 7 determines the horizontal angle, the vertical angle, and the angle of view of the imaging unit 1 from the posture information sent from the posture measurement unit 5 and the lens state information sent from the lens state control unit 6. Is calculated. Then, based on the horizontal direction angle, the vertical direction angle, and the angle of view, the data stored in the data unit 8 is read out, and mixed data located in the field of view of the imaging unit 1 is generated.
[0036]
There are various methods for obtaining the field of view of the imaging unit 1 from the posture information sent from the posture measurement unit 5 and the lens state information sent from the lens state control unit 6, but here one of them is used. explain.
[0037]
Generally, since the field of view can be obtained by obtaining the position of the lens center of the lens used in the imaging unit 1 and the viewing angle of the imaging unit 1, first, from the posture information sent from the posture measuring unit 5, A method for obtaining the position of the lens center of the lens used in the imaging unit 1 will be described, and then a method for obtaining the viewing angle of the imaging unit 1 from the lens state information sent from the lens state control unit 6 will be described.
[0038]
A method of obtaining the lens center position of the lens used in the imaging unit 1 from the posture information sent from the posture measurement unit 5 will be described with reference to FIGS. 11 and 12. FIG. 11 is an external view of the image mixing apparatus of this embodiment. The imaging unit 1 and the display unit 9 are integrally supported by the support unit 4 and can be rotated in the vertical direction and the horizontal direction around the rotation center O. Since the support portion 4 is fixed to the floor, the positional relationship between the floor and the rotation center O is unchanged. Therefore, if the relative positional relationship between the rotation center O and the lens center LO is known, the position of the lens center LO with respect to the real world is uniquely determined.
[0039]
Here, the XYZ coordinate system with the rotation center O as the origin is defined as shown in FIG. 11, and the left and right rotation angles φ and the upper and lower rotation angles θ are defined as shown in FIG. When the distance from the lens center LO to the rotation center O of the lens used in the imaging unit 1 is expressed as d, the coordinates of the lens center LO are
x = −d sin θ sin φ
y = dsin θ cos φ
z = d cos θ
Represented as: Here, the XYZ coordinate system is defined as shown in FIG. 11. However, regardless of how the coordinate system is defined and how the vertical and horizontal rotation angles are defined, the horizontal rotation angle and the vertical direction are defined. It goes without saying that the position of the lens center of the lens used in the imaging unit 1 can be obtained from the rotation angle.
[0040]
A method for obtaining the viewing angle of the imaging unit 1 from the lens state information sent from the lens state control unit 6 will be described with reference to FIGS. 13 and 14. First, the focal length of the lens used in the imaging unit 1 is calculated from the zoom value sent from the lens state control unit 6. Here, it is assumed that the zoom value transmitted from the lens state control unit 6 is a value output from an 8-bit encoder provided in the imaging unit 1, and the value and the focal length of the lens are as shown in the table of FIG. It is assumed that Then, by referring to the table of FIG. 13 and interpolating values not stored in the table, the focal length of the lens used in the imaging unit 1 is used by using the zoom value sent from the lens state control unit 6. Can be sought. However, when the lens state control unit 6 can directly output the focal length, it is not necessary to calculate the focal length using a table as shown in FIG.
[0041]
Assuming that the focal length of the lens used in the imaging unit 1 obtained as described above is f and the horizontal width of the imaging surface provided in the imaging unit 1 is x, the horizontal viewing angle ψ of the imaging unit 1 is as follows from FIG. It can be calculated by the following formula.
[0042]
ψ = 2 arctan (x / 2f)
Similarly, assuming that the height of the imaging surface provided in the imaging unit 1 is y, the viewing angle μ in the vertical direction of the imaging unit 1 can be obtained from the following equation from FIG.
[0043]
μ = 2 arctan (y / 2f)
As described above, the position of the lens center of the lens used in the image pickup unit 1 and the viewing angle of the image pickup unit 1 are obtained from the posture information sent from the posture measurement unit 5 and the lens state information sent from the lens state control unit 6. Therefore, the field of view of the imaging unit 1 can be obtained.
[0044]
The method for obtaining the field of view of the imaging unit 1 is not limited to the method described above as long as the field of view of the imaging unit 1 is obtained.
[0045]
In step S <b> 6, the mixed image generation unit 7 generates a generated image using the data acquired from the data unit 8. For example, this generated image has the same angle of view in the same or almost the same posture as that of the imaging unit 1 at the time of step S5 from the actual panorama image generated from the image previously captured at the time t in step S0 by the imaging unit 1. This is an image obtained by cutting out an image close to an image that would be captured when zoomed. The method of cutting out has been described in the explanation part of Step 0.
[0046]
For example, the generated image is a computer image in which text data associated with the real world included in the field of view of the imaging unit 1 is drawn on a transparent background, or a three-dimensional CG associated with the real world. The three-dimensional virtual space composed of data may be a two-dimensional computer image taken by a virtual camera at a position and posture corresponding to the imaging unit 1. Furthermore, the generated image may be an image in which an image cut out from the actual panorama image and the two-dimensional computer image are mixed. The generated image is sent to the image mixing unit 3.
[0047]
In step S <b> 7, the captured image transmitted from the captured image capturing unit 2 and the generated image transmitted from the mixed image generating unit 7 are mixed in the image mixing unit 3. When the generated image is an image cut out from the real panorama image captured by the imaging unit 1, the captured image is an image captured by the imaging unit 1 at the time of processing in step S5, and the generated image is captured by the imaging unit 1 at a time close to time t. However, the horizontal angle and the vertical angle at the time of shooting are exactly the same or substantially the same, and the shooting angle of view is exactly the same. That is, the stationary objects appearing in both images are completely or substantially the same, but have different brightness. When the generated image is a computer image, the computer image includes a space corresponding to the real world included in the captured image.
[0048]
One method of mixing the captured image and the generated image in step S7 includes an alpha blending process. If the luminance of a pixel located at coordinates (x, y) in an image A is represented by A (x, y) [R, G, B], two images A and B having the same size In alpha blending (α blending) processing that generates image C by mixing
αA (x, y) [R, G, B] + (1-α) B (x, y) [R, G, B] = C (x, y) [R, G, B]
Perform the process. α indicates transparency, and is an arbitrary value between 0 and 1 and is called an alpha value. If the value of α is close to 0.5, the mixed image includes both shooting conditions equally, and if it is close to 0 or 1, the mixed image is almost the same as one of the images. In particular, α = 0 corresponds to not performing the mixing process. In this process, a mixed image with smooth image quality can be generated.
[0049]
The above alpha value setting may be performed manually or automatically. For example, when performing manually, although not shown in the block diagram of FIG. 1, the alpha value is set via a mixing rate input unit such as a keyboard connected to the image mixing unit 3. When performing automatically, the image mixing unit 3 analyzes the image obtained from the captured image capturing unit 2, and determines the alpha value according to the analysis result.
[0050]
Hereinafter, an example of an alpha value setting method in the case of performing automatically will be described.
[0051]
Automatic setting method 1: The alpha value (α) is determined according to the brightness of the captured image obtained from the captured image image capturing unit 2.
[0052]
The automatic setting method 1 is effective when the image is bright but details are not clearly visible. For example, by synthesizing a panoramic image showing a clear landscape stored in advance in the data unit 8 with a landscape with unclear details such as a hazy landscape or a foggy landscape, It can make it easy to see details in the landscape. Specifically, if the change in brightness of the photographed image is small, the ratio of the panoramic image showing a clear landscape in the composite image is increased by increasing α.
[0053]
First, the photographed image data obtained from the photographed image capturing unit 2 is converted into a grayscale image using the following conversion formula. A grayscale image is an image whose density is determined by lightness, and is image data composed of only gray of which saturation is 0 and the RGB values are equal for each pixel, and the RGB values are equal to lightness.
[0054]
Lightness of each pixel = each pixel value of RGB = (a × R component + b × G component + c × B component) a, b, and c may be arbitrary values. However, as an example, they are used in the NTSC standard here. The values based on the YIQ model (a = 0.299, b = 0.487, c = 0.144) are given. This value was determined taking into account the difference in sensitivity to human visual color.
[0055]
Next, α is determined from the obtained gray scale image.
[0056]
Two-dimensional Fourier transform (for example, discrete cosine transform) is performed on the obtained gray scale image. Then, from the conversion result, the ratio of high frequency components over a certain frequency to the whole is calculated, and α is determined according to the ratio. For example, α is determined using the following conversion formula.
[0057]
α = proportional constant × high-frequency component ratio + constant
However, 0 if α <0, 1 if α> 1.
Assuming that the proportionality constant is -0.8 and the constant is 0.74, if "mixed image = (1-α) x captured image + α x generated image", the ratio of high-frequency components is 0.3 (30%) If α = 0.5 and the high-frequency component ratio is 0.8 (80%), then α = 0.1.
[0058]
According to this method, if the high-frequency component of the grayscale image is small, that is, if the change in brightness is small, α is large.
[0059]
Further, the brightness variation of the obtained gray scale image may be obtained as a standard deviation, and α may be determined according to the standard deviation.
[0060]
For example, α is determined according to the following conversion formula.
[0061]
α = proportional constant × lightness standard deviation + constant
However, 0 if α <0, 1 if α> 1.
When the proportionality constant is 0.004 and the constant is 0, in the case of “mixed image = (1−α) × captured image + α × generated image”, if the standard deviation of brightness is 150, if α = 0.5, 50 Α = 0.1.
[0062]
According to this method, if the standard deviation of the grayscale image is small, that is, if the change in brightness is small, α is large.
[0063]
Automatic setting method 2: The alpha value (α) is determined by comparing the brightness of the captured image obtained from the captured image capturing unit 2 and the brightness of the generated image transmitted from the mixed image generating unit 7.
[0064]
The automatic setting method 2 is effective when the captured image is dark and cannot be seen as a whole.
[0065]
First, using a method similar to the automatic setting method 1, a grayscale image is generated from the captured image obtained from the captured image capturing unit 2. Further, a gray rail image is generated from the generated image generated by the mixed image generating unit 7. Then, the average brightness of each of the two obtained grayscale images is calculated.
[0066]
Α is determined according to the difference in brightness average obtained. For example, the following conversion formula is used.
[0067]
α = proportional constant × (average brightness of mixed image−average brightness of captured image) + constant
However, 0 if α <0, 1 if α> 1.
When the proportionality constant is 0.004 and the constant is 0, in the case of “mixed image = (1−α) × captured image + α × generated image”, if the difference in brightness is 150, if α = 0.5, 50 α = 0.1.
[0068]
Automatic setting method 3: The brightness of the captured image obtained from the captured image capturing unit 2 and the brightness of the generated image sent from the mixed image generating unit 7 are compared for each pixel to determine the alpha value (α). .
[0069]
The automatic setting method 3 is also effective when the captured image is dark and cannot be seen as a whole.
[0070]
First, two grayscale images are generated from the image obtained from the captured image capturing unit 2 and the generated image generated by the mixed image generating unit 7.
[0071]
Next, the brightness difference between the captured image and the generated image is obtained for each pixel, and the average brightness difference is calculated. Then, α is determined according to the average brightness difference. For example, the following conversion formula is used.
[0072]
α = proportional constant × (average brightness difference between mixed image and captured image) + constant
However, 0 is set when α <0, and 1 when α> 1.
[0073]
When the proportionality constant is 0.004 and the constant is 0, in the case of “mixed image = (1−α) × captured image + α × generated image”, if the average brightness difference is 150, then α = 0.5, 50 Α = 0.1.
[0074]
According to the automatic setting methods 2 and 3, since α is determined according to the brightness difference between the captured image and the generated image, the generated image is generated from the panoramic image generated from the captured image at the time of clearing. Then, when the captured image is dark and cannot be clearly seen, the composition ratio of the generated image is increased, and the configuration of details can be emphasized.
[0075]
Thus, an appropriate automatic setting method varies depending on the purpose. Therefore, a plurality of automatic setting methods may be provided, and an appropriate automatic setting method may be selected according to conditions specified by the user on the operation unit. For example, when the automatic setting method 1 and the automatic setting method 2 or 3 are provided, a sunny button and a night view button may be provided in the operation unit as the condition specifying unit.
[0076]
The above-described alpha value is obtained for each image. However, when the alpha value is embedded for each pixel of the generated image and mixed with the photographed image, the alpha value may be set using the embedded value.
[0077]
For example, assuming that a value of α = 1 is embedded in a part of a landscape in which a building is reflected and a value of α = 0 is embedded in other parts, the image is mixed by this image mixing device. As for the image of the building, an image close to the image that would have been captured by the image capturing unit 1 at a time close to time t is displayed, and the image captured by the image capturing unit 1 during the processing of step S7 is displayed in the other part. Is done.
[0078]
Another method of mixing the captured image and the generated image in step S7 includes a process of mixing both in a mosaic pattern. In this process, a mixed image is created by displaying one of two images that are mixed for each pixel. In other words, this is a process in which the alpha blending process is performed by changing the alpha value for each pixel. For example, when two images D and E are mixed to create an image F, the coordinates of each pixel of the image are represented by (X, Y), and m and n are integers of 1 or more. The pixels that can be represented by coordinates (2m-1, 2n-1) and (2m, 2n) are subjected to alpha blending processing with α = 0 and the corresponding pixels of image D are displayed, and coordinates (2m, 2n-1) and An image corresponding to the image E is displayed on the pixels represented by (2m−1, 2n) by performing alpha blending processing with α = 1. FIG. 7 is a diagram for explaining this mixing process. In this process, a mixed image can be generated with a small amount of calculation.
[0079]
In the above example, adjacent pixels are mixed at different ratios, but the mixing ratio may be changed for each area of the screen. Further, different mixing processes may be performed depending on the region. FIG. 8 is a diagram for explaining the case where mixing is performed by one method over the entire screen. FIG. 9 is a diagram illustrating a case where the mixing process is performed only in a certain area of the screen and the process is not performed in the remaining area.
[0080]
When the method of mixing processing and the mixing ratio are changed in accordance with the area of the screen, the area may be designated automatically or manually. For example, when performing manually, although not shown in the block diagram of FIG. 1, when a certain pixel in the image is designated using the mouse connected to the image mixing unit 3, the distance is within a certain distance from the pixel. It becomes the specified area. When performing automatically, for example, contour line emphasis processing is performed on the captured image, and the image is divided into regions based on the emphasized contour line. Further, for example, although not shown in the block diagram of FIG. 1, when a keyword is input using a keyboard connected to the image mixing unit 3, a region that matches the keyword becomes a designated region.
[0081]
In the above-described mixing, the case where one generated image is mixed with one captured image has been described. However, two or more generated images are mixed with one captured image. May be. In this case, for example, the mixing may be performed by a process of calculating one pixel value from three pixel values, or the generated image to be mixed with the captured image may be switched according to the area of the screen. For example, an image cut out from a panoramic image created in fine weather can be mixed in a part to be emphasized in a captured image, and a black color image can be generated in the other part.
[0082]
The mixed image obtained by mixing the captured image and the generated image is sent to the display unit 9. In step S8, the display unit 9 displays the mixed image information sent from the image mixing unit 3. Thereafter, in step S9, it is checked whether or not to end the system. If the system is to be ended, the system is ended in step S10. If not, the process returns to step S2, and the above-described processing is repeated.
[0083]
As described above, according to the first embodiment, for example, when viewing an outdoor landscape on an observation deck, even if a building or the like in the landscape cannot be seen with the naked eye due to cloudy weather, the landscape in front of you and the clear sky The scenery of the time is mixed and you can see the clear outdoor scenery where the buildings etc. are easier to see. A live-action panoramic image created from an image taken in fine weather is stored in the data unit 8 in advance, and the horizontal direction angle, the vertical direction angle, and the angle of view when the imaging unit 1 captures the real world using posture information and lens state information. The region corresponding to the field of view is cut out from the real panorama image stored in the data unit 8 and mixed with the captured image sent from the captured image capturing unit 2 capturing the captured image of the scenery in front of the eyes. Then, the above items are achieved by displaying on the display unit 9.
[0084]
In addition, according to the first embodiment, it is possible to view a landscape as if the outdoors are illuminated with illumination, for example, in a time zone when the sun goes down and darkens. That is, a live-action panoramic image created from an image captured in a bright time zone is stored in the data unit 8 in advance, and the horizontal direction angle and vertical direction when the imaging unit 1 captures the real world using posture information and lens state information. A captured image capturing unit that captures a captured image obtained by capturing an image of a scene in front of the eye, with the angle and the angle of view determined, the portion of the field of view cut out from the real panorama image stored in the data unit 8 2 is mixed with the corresponding part of the captured image sent from 2 and displayed on the display unit 9, and the remaining part is displayed on the display unit 9 as it is the captured image sent from the captured image capturing unit 2. The above items are achieved.
[0085]
Furthermore, according to the first embodiment, for example, a past or future landscape virtually reproduced can be superimposed on a landscape spreading in front of the present. Three-dimensional CG data that reproduces the past or future landscape of the place is stored in the data unit 8 in advance, and a three-dimensional virtual space configured by the three-dimensional CG data is virtually displayed at a position and orientation corresponding to the imaging unit 1. The above-mentioned matter is achieved by generating a two-dimensional CG image captured by the camera and mixing it with the captured image sent from the captured image capturing unit 2 that captures the image actually captured by the imaging unit 1. .
[0086]
<Second Embodiment>
FIG. 3 is a block diagram showing a schematic configuration of the image mixing apparatus according to the second embodiment. The same components as those of the image mixing apparatus (FIG. 1) according to the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted. The second embodiment is different in that the support portion 4 is not provided. Moreover, the point by which the attitude | position measurement part 5 in 1st embodiment becomes the position / orientation measurement part 10 differs.
[0087]
Reference numeral 10 denotes a position / orientation measurement unit that measures the position and orientation of the imaging unit 1, and requests the position information and orientation information of the imaging unit 1 in response to a request from the mixed image generation unit 7 or a request from the mixed image generation unit 7. Without being sent to the mixed image generating unit 7. The position / orientation measurement unit 10 may use, for example, a gyro sensor or an optical sensor. In the present specification, the portion described as a position / orientation measurement unit includes a part composed of a position measurement unit and an attitude measurement unit. The position / orientation measurement unit 10 may be a combination of a plurality of sensor devices.
[0088]
The mixed image generation unit 7 extracts the position / orientation information of the imaging unit 1 from the position / orientation measurement unit 10 to estimate the position and orientation of the imaging unit 1, and extracts the lens state information from the lens state control unit 6 to acquire the imaging unit 1. Estimate the angle of view. If the position, orientation, and angle of view of the image capturing unit 1 can be estimated, the field of view of the image capturing unit 1 can be obtained. Therefore, the mixed image generation unit 7 extracts data corresponding to the field of view of the image capturing unit 1 from the data unit 8. The extracted data is sent to the image mixing unit 3.
[0089]
The display unit 9 is preferably moved in conjunction with the imaging unit 1 and the position / orientation measurement unit 10, but this is not always necessary. For example, the configuration may be such that the imaging unit 1 is movable and the display unit 9 is fixed. .
[0090]
The mixed image storage unit 13 includes, for example, a hard disk, and receives and records the mixed image signal from the image mixing unit 3. As the mixed image storage unit 13, for example, a video recorder or a hard disk may be used, and any medium can be used as long as it can store images. The In addition, information other than the mixed image is recorded in the mixed image storage unit 13, for example, text information and image information such as the date and place of the composition are recorded.
[0091]
The stored image output unit 14 takes out part or all of the mixed image from the mixed image storage unit 13 and outputs it as an image or printed matter. The stored image output unit 14 may be a printer or the like, but may be anything that can output a stored image. The
[0092]
The input unit 15 includes a keyboard, for example, and inputs information other than the mixed image to the mixed image storage unit 13. The input unit 15 is not limited to a keyboard, and may be anything that can be input by the user of the image mixing apparatus of the present invention.
[0093]
The control of the present embodiment having the above configuration will be described below. FIG. 4 is a flowchart for explaining the processing procedure in the image mixing apparatus of the present invention. In addition, the same reference number is attached | subjected about the step which performs the process similar to the flowchart (FIG. 2) shown in 1st embodiment, and detailed description is abbreviate | omitted. The steps that are not described are exactly the same. Below, a different part is demonstrated by 1st embodiment and 2nd embodiment.
[0094]
In step S0, data to be stored in the data portion 8 is prepared. This data is a combination of images taken at time t (= t1, t2,...) Prior to using this apparatus, for example, using the imaging unit 1 described in step S0 in the first embodiment. This is a live-action panoramic image. An image taken at time t using the imaging unit 1 to create a real panorama image will be referred to as an original image hereinafter.
[0095]
Among the pixels constituting the real panorama image, the original image is a pixel in a predetermined column, for example, the first column from the left, and the position of the imaging unit 1 when the pixel is captured And a horizontal angle are associated with each other. Similarly, among the pixels constituting the real panorama image, the original image includes pixels in a predetermined row, for example, the pixel in the first row from the top, and the imaging unit when the pixel is photographed The position 1 and the vertical angle are associated with each other. Here, the position, the horizontal direction angle, and the vertical direction angle associated with the actual panorama image are estimated from the position / orientation information of the image capturing unit 1 measured by the orientation measuring unit 5 at the time t when the original image was captured.
[0096]
If a certain position, horizontal angle, and vertical angle that can be taken by the imaging unit 1 are designated from the actual panorama image as described above, the imaging unit 1 would have taken a picture at that position and posture at a time close to time t. An image close to the image can be cut out from the real panorama image. However, it is needless to say that this is based on cutting out an image at a position where the entire image to be cut out is included in the real panorama image, and at a horizontal angle and an up-down angle.
[0097]
Further, the actual panorama image is obtained by associating the angle of view of the imaging unit 1 at the time of shooting estimated by the image mixing unit. That is, when a certain angle of view is designated, an area captured when zooming to the designated angle of view can be cut out from this image with the associated angle of view as a reference.
[0098]
Further, the actual panorama image may be obtained by performing image processing such as color tone correction and contour line enhancement on the image panoramicized by using the imaging unit 1.
[0099]
For example, the data stored in the data unit 8 is text data, three-dimensional CG data, a two-dimensional CG image, or the like. This data needs to have correspondence with the three-dimensional coordinates in the real world in advance. For example, if it is a three-dimensional CG, it corresponds to which three-dimensional position in the real world, and if it is a two-dimensional CG image, You have to specify the image from which position and from what direction.
[0100]
After data preparation, the system is started in step S1. In step S 2, an image is acquired from the imaging unit 1, and the acquired captured image is converted into an appropriate format by the captured image capturing unit 2 and sent to the image mixing unit 3. In step S <b> 11, the position and orientation measurement unit 10 detects the position and orientation of the imaging unit 1, and the detected position and orientation information is sent to the mixed image generation unit 7. In step S <b> 4, the lens state control unit 6 detects lens state information such as a zoom value and a focus value of the lens used in the imaging unit 1, and the detected lens state information is sent to the mixed image generation unit 7.
[0101]
In step S <b> 5, the mixed image generation unit 7 determines the position, posture, and angle of view at the time of shooting of the image pickup unit 1 from the position / posture information sent from the position / posture measurement unit 10 and the lens state information sent from the lens state control unit 6. The data corresponding to the visual field of the imaging unit 1 at the time of the process of step S5 is calculated and acquired from the data unit 8. The acquired data is sent to the image mixing unit 3.
[0102]
In step S <b> 6, the mixed image generation unit 7 generates a generated image using the data acquired from the data unit 8. For example, this generated image is obtained by cutting out a region shot when zooming to the same angle of view at the same or substantially the same position and orientation as the imaging unit 1 at the time of processing in step S5 from the real panorama image previously captured by the imaging unit 1. It is an image. For example, the generated image is a computer image in which text data associated with the real world included in the field of view of the imaging unit 1 is drawn on a transparent background, or a three-dimensional CG associated with the real world. The three-dimensional virtual space composed of data may be a two-dimensional computer image taken by a virtual camera at a position and posture corresponding to the imaging unit 1. The generated image is sent to the image mixing unit 3.
[0103]
In the second embodiment, the mixed image is stored in step S14 following step S8. In step S <b> 14, the mixed image storage unit 13 records the mixed image sent from the image mixing unit 3.
[0104]
In the second embodiment, when the user of the image mixing apparatus finishes using the image data in step S9, the image is output from the stored image output unit 14 in step S15, and the output result is passed to the user. The stored image output unit 14 is composed of, for example, a printer, and the user can take home a still image of the mixed image when he / she uses the image mixing apparatus of the present invention.
[0105]
When outputting the stored image in step S15, if information other than the mixed image recorded in the mixed image storage unit 13 is used, for example, date information of the day can be written as text in addition to the stored image. It is. It is also possible for the user to input his / her name from the input unit 15 and add the name as text to the mixed image. If a tablet is used for the input unit 15, for example, the user of the image mixing apparatus of the present invention can freely add a picture or a character to the mixed image. If, for example, a frame image is input to the mixed image storage unit 13 through the input unit 15 in advance, it is possible to output a commemorative photographed storage image by inserting the mixed image into the frame image. It is.
[0106]
As described above, according to the second embodiment, for example, when using a head-mounted see-through display to look at the real correct answer, a landscape in which an image close to the landscape at another time is mixed with the landscape in front of you Or, you can see a landscape in which a virtually created landscape is mixed with the landscape in front of you.
[0107]
<Third embodiment>
The outline of the third embodiment is as follows. FIG. 5 is a block diagram showing a schematic configuration of an image mixing apparatus according to the third embodiment. The same components as those of the image mixing apparatus (FIG. 1) according to the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted. The third embodiment is different in that the support unit 4 and the lens state control unit 6 are not provided.
[0108]
Reference numeral 5 denotes an attitude measurement unit, which is a mixed image generation unit that changes the attitude information of the imaging unit 1, that is, the attitude in the real world, in response to a request from the mixed image generation unit 7 or without a request from the mixed image generation unit 7. 7 to send. The posture measuring unit 5 may be a gyro sensor or an optical sensor, for example.
[0109]
Reference numeral 7 denotes a mixed image generation unit that extracts posture information of the imaging unit 1 from the posture measurement unit 5 and estimates a direction in which the imaging unit 1 is facing. If the direction in which the imaging unit 1 is facing can be estimated, the field of view of the imaging unit 1 can be obtained when the use position of the apparatus and the angle of view of the imaging unit 1 are known. The mixed image generation unit 7 extracts data corresponding to the visual field of the imaging unit 1 from the data unit 8. The extracted data is sent to the image mixing unit 3.
[0110]
The control of the present embodiment having the above configuration will be described below. FIG. 6 is a flowchart for explaining a processing procedure in the image mixing apparatus. In addition, the same reference number is attached | subjected about the step which performs the process similar to the flowchart (FIG. 2) shown in 1st embodiment, and detailed description is abbreviate | omitted. The steps that are not described are exactly the same. Below, a different part is demonstrated by 1st embodiment and 3rd embodiment.
[0111]
In step S1, the system is activated. At this time, the apparatus must be used at a predetermined position.
[0112]
In step S5, the mixed image generation unit 7 estimates the field of view of the imaging unit 1 from the posture information sent from the posture measurement unit 5 and the coordinates of the use position of the apparatus determined in advance, and the imaging unit at the time of processing in step S5 Data corresponding to one field of view is acquired from the data unit 8. The acquired data is sent to the image mixing unit 3.
[0113]
As described above, according to the third embodiment, for example, when a user stands at a predetermined position and views a landscape using a head-mounted type see-through display, the time in front of the landscape is different. It is possible to view a landscape in which images close to the current landscape are mixed, or a landscape in which virtually created landscapes are mixed in front of you.
[0114]
In addition, the image mixing apparatus according to the present embodiment is not limited to use for outdoor scenery, and can be used in all situations where the image mixing apparatus according to the present invention can be effectively used. is there.
[0115]
The object of the present embodiment is to supply a storage medium (or recording medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU or MPU of the system or apparatus). ) Is also achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.
[0116]
Further, after the program code read from the storage medium is written to a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the card or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.
[0117]
【The invention's effect】
As described above, according to the present invention, it is possible to clarify the field of view of a real-world image captured when the field of view is poor.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a schematic configuration of an image mixing apparatus according to a first embodiment.
FIG. 2 is a flowchart illustrating a processing procedure of the image mixing device according to the first embodiment.
FIG. 3 is a block diagram showing a schematic configuration of an image mixing apparatus according to a second embodiment.
FIG. 4 is a flowchart illustrating a processing procedure of the image mixing device in the second embodiment.
FIG. 5 is a block diagram showing a schematic configuration of an image mixing device according to a third embodiment of the present invention.
FIG. 6 is a flowchart illustrating a processing procedure of the image mixing device according to the third embodiment.
FIG. 7 is a diagram illustrating a process of mixing two images in a mosaic shape.
FIG. 8 is a diagram illustrating a case where two images are mixed over the entire screen.
FIG. 9 is a diagram for explaining a case where two images are mixed only in a partial area of the screen, and one of the two images is displayed in the remaining area.
FIG. 10 is a diagram illustrating a case where a panoramic image is captured by an imaging unit and stored in a data unit.
FIG. 11 is a diagram illustrating a method for obtaining the position of the lens center of a lens used in the imaging unit from posture information sent from the posture control unit.
FIG. 12 is a diagram illustrating a method for obtaining the position of the lens center of a lens used in the imaging unit from posture information sent from the posture control unit.
FIG. 13 is a diagram illustrating a method for obtaining a focal length of a lens used in an imaging unit from lens state information sent from a lens state control unit.
FIG. 14 is a diagram illustrating a method for obtaining a viewing angle of a lens from a focal length of the lens used in an imaging unit.

Claims (12)

An information processing method for mixing an image generated from panoramic image data stored in advance with a captured image captured by an imaging unit,
Input the captured image captured by the imaging unit and the posture information of the imaging unit,
Hold panoramic image data based on images taken at other times,
Based on the posture information of the photographing unit, an image corresponding to the photographed image is generated from the panoramic image data,
Analyzing the captured image to determine the mixing conditions,
An information processing method, wherein the generated image is mixed with the captured image using the mixing condition. The information processing method according to claim 1, wherein the mixing condition is obtained using a mixing condition calculation method selected from a plurality of mixing condition calculation methods. The information processing method according to claim 1, wherein the mixing condition is obtained based on brightness information of the captured image. Create a grayscale image from the captured image,
The information processing method according to claim 1, wherein the mixing condition is obtained by analyzing a spatial frequency of the gray scale image. The mixing condition creates a grayscale image of each of the captured image and the generated image,
The information processing method according to claim 1, wherein the mixing condition is obtained from a difference between average values of the gray scale images. The information processing method according to claim 1, wherein the photographed image photographed at another time is a photographed image photographed when the weather is fine. The program for implement | achieving the information processing method described in any one of Claims 1 thru | or 7. An information processing method for mixing an image generated from data held in advance with a captured image captured by an imaging unit,
Input the captured image captured by the imaging unit and the posture information of the imaging unit,
Based on the posture information of the photographing unit, an image corresponding to the photographed image is generated from data stored in advance,
Analyzing the brightness of the captured image to determine the mixing conditions,
An information processing method, wherein the generated image is mixed with the captured image using the mixing condition. The information processing method according to claim 8, wherein the mixing condition is embedded in the generated image. 9. The information processing method according to claim 8, wherein a mixing condition is changed for each pixel of the photographed image. The program for implement | achieving the information processing method in any one of Claims 8 thru | or 10. Photography means for photographing the real world,
Posture measuring means for measuring the posture of the photographing unit;
Generating means for generating an image corresponding to the photographed image from pre-stored data based on posture information of the photographing unit;
Means for analyzing the brightness of the captured image and obtaining a mixing condition;
An image mixing apparatus comprising: a mixing unit that mixes the generated image with the captured image using the mixing condition.

Images