JP4124995B2 - Video composition apparatus and information processing method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an apparatus for switching between display of an image based on a real image captured by an imaging unit and display of an image based on computer graphics.
[0002]
[Prior art]
A telescope placed on an observatory, etc. captures the scenery from the observatory and presents it to the user by magnifying it using an optical device. However, conventional telescopes simply optically magnify and display the scenery. Since it is only a thing, there was a problem that only what actually exists in the real world could be presented. For example, a telescope cannot present information attached to a building in the real world (such as an address or the name of a building), and it is impossible to view a virtual character placed in the real world. is there. Also, when the zoom magnification of the telephoto reaches the limit, switch the target object to a computer graphics image to display it in more detail, or walk around the target object as a virtual world I could not. There are the following conventional examples for solving these problems.
[0003]
Japanese Patent No. 2833206 “Display Device for Prospecting” discloses a technique for superimposing virtual characters and images on a real scene viewed from a telescope. The apparatus disclosed here superimposes external scene light from the front of the apparatus and image display light from a surface on which virtual characters and images are displayed using a combiner composed of a half mirror or the like. Present to the user.
[0004]
[Problems to be solved by the invention]
A device (hereinafter referred to as an enlarged display device) for enlarging and seeing what is actually seen, such as a normal telescope or Japanese Patent Application Laid-Open No. 2000-125171 “Optical device”, has a limitation on the enlargement ratio. Therefore, when the zoom magnification reaches the limit, it cannot be magnified further. In other words, when the zoom value has reached the maximum, even if the user performs an operation to increase the zoom value because he wants to see more details, it is not possible to display more detailed information about the object.
[0005]
In this embodiment, when the control of the optical system of the photographing unit reaches a set value or more, the target object of interest is switched to a computer graphics image or the like, and the user's target object is viewed in more detail. The purpose is to be able to satisfy the demand.
[0006]
[Means for Solving the Problems]
In order to achieve the above object, the present invention has the following structure.
[0007]
The video composition device according to claim 1 of the present application is a video image composition device having a spatial mode for displaying an image based on a photographed image taken by an imaging unit and a spatial mode for displaying an image based on computer graphics. An imaging unit that captures an image of the real world through a lens and generates a photographed image; and a support unit that supports the imaging unit and includes a movable unit that allows a user to manually move the imaging unit. A posture measurement unit that measures a relative posture of the imaging unit and the support unit, a lens state control unit that controls the state of the lens, and a data unit that holds data for generating computer graphics, A space in which a spatial mode is set based on posture information from the posture measurement unit, lens state information from the lens state control unit, and data held in the data unit. And a real image generated by the imaging unit, a computer video generation unit that generates computer video using data held in the data unit based on the spatial mode set by the spatial management unit And a video generation unit that generates a video to be displayed from the computer video generated by the computer video generation unit, wherein the space management unit sets a preset value of a zoom value of a lens used in the imaging unit As described above, when computer graphics displayed in a specific area of the display unit are held in the data unit, a spatial mode for displaying an image based on the computer graphics is set. To do.
[0008]
The information processing method according to the claims 7, an information processing method that allows for display and display of the image which is based on computer graphics image was captured photographed image imaging section and Ace, enter the attitude information and zoom information of the imaging section, and determines whether or not the zoom value of the imaging section based on the zoom information is larger than the set value, it is displayed in a predetermined area of the captured image based on the posture information computer graphics determines whether Luke held in the data unit that, not the zoom value is greater than the established value, and computer graphics displayed in a predetermined area of the captured image is stored in the data section If you can generate computer images corresponding to the computer graphics, displays an image of the computer image said generated as the base And characterized in that.
[0009]
DETAILED DESCRIPTION OF THE INVENTION
<First Embodiment>
The outline of the first embodiment is as follows.
[0010]
The video composition device is used as a telescope at an observation deck or the like, and as an information display terminal at a museum or the like. FIG. 1 is a block diagram showing a schematic configuration of a video composition apparatus according to the first embodiment.
[0011]
Reference numeral 1 denotes an image pickup unit, which includes, for example, a camera. The imaging unit 1 captures a real world such as an outdoor landscape or an indoor exhibit, and outputs the captured video as a video signal to the captured video capturing unit 2.
[0012]
Reference numeral 2 denotes a captured video capturing unit that converts a video signal output from the imaging unit 1 into a format suitable for the video synthesis unit 3 and sends the video signal to the video synthesis unit 3.
[0013]
Reference numeral 3 denotes a video composition unit, which is a composite image obtained by synthesizing the captured video from the captured video capture unit 2 and the computer video from the computer video generation unit 7 according to the space mode information extracted from the space management unit 10 or a computer. Only the video is sent to the display unit 9.
[0014]
A support unit 4 is installed on the floor or the like and supports the imaging unit 1. The support unit 4 has a movable part, and allows the user to manually rotate the imaging unit 1 in the left-right direction and the up-down direction manually. The support unit 4 may be fixed to a floor or the like, but may be moved on a rail or a carriage, for example.
[0015]
Reference numeral 5 denotes an attitude measurement unit that measures the relative attitude of the support unit 4 and the imaging unit 1. The attitude information of the imaging unit 1, that is, the vertical and horizontal rotation information of the imaging unit 1 is obtained from the space management unit 10. The data is sent to the space management unit 10 in response to a request or without a request from the space management unit 10. The attitude measurement unit 5 is composed of, for example, a mechanical encoder. The posture measurement unit 5 may have any configuration as long as it can measure the relative posture of the support unit and the imaging unit, and a gyroscope or an optical sensor, for example, can also be used.
[0016]
Reference numeral 6 denotes a lens state control unit which operates the lens state by an operation unit (not shown) or a control signal. The lens state control unit 6 includes, for example, a mechanical encoder or a stepping motor. In addition, the lens control unit 6 receives lens state information such as a zoom value and a focus value of a lens used in the imaging unit 1 in response to a request from the space management unit 10 or without a request from the space management unit 10. Send to 10. The zoom value / focus value are values output from the encoder or the like by the lens state control unit 6.
[0017]
Reference numeral 10 denotes a space management unit that extracts orientation information of the imaging unit 1 from the orientation measurement unit 5 to estimate a direction in which the imaging unit 1 is facing, and extracts lens state information from the lens state control unit 6 to extract the imaging unit 1. Is estimated and stored as camera state data. The space management unit 10 refers to the camera state data and the data related to the space of the data unit 8 to determine the space mode. The spatial mode is a live-action mode, a VR spatial mode, or the like.
[0018]
The live-action mode presents an actual image taken by the imaging unit 1 or an image obtained by synthesizing an image generated by computer graphics based on the actual image.
[0019]
The VR space mode is a mode that shifts from the actual shooting mode according to the zoom value, and presents an image based on a computer graphics image.
[0020]
According to the VR space mode, for example, the object of interest can be switched to a computer graphics image to display in more detail, information about the object can be displayed, or the object can be walked through by entering into the object. Can be.
[0021]
An example of switching the information level itself presented as an example of the VR space mode will be described.
[0022]
For example, if the object is a cloth, the zooming value can be used to show how the cloth is made of yarn, how the thread is made of fibers, and how the fibers are made of molecules. To do.
[0023]
This can be realized by providing a plurality of models having different resolutions (LOD: Level of Details) for one object and switching and displaying the models according to the viewpoint position or the like. That is, it can be realized by providing a plurality of models for one object and switching and displaying the models according to the zoom value. When switching models, the images can be switched smoothly by blending the images. When the information level itself is switched, the data unit 8 holds a plurality of models corresponding to the spatial position data and holds each model corresponding to the zoom value.
[0024]
As another example of the VR space mode, an example will be described in which an image as if walking through the object as a VR space is presented.
[0025]
For example, the camera viewpoint position of the virtual camera is set in the vicinity of the object, the zoom-in operation corresponds to the forward movement, and the zoom-out operation corresponds to the backward movement so that the camera can walk through the object. When the VR space mode is canceled, for example, an exit from the VR space is prepared in advance, and a condition is set in advance such as when exiting from the VR space. When presenting an image that is walking through, the data unit 8 holds a data group for presenting an image that is walking through corresponding to the spatial position data.
[0026]
The space management unit 10 determines a space mode from the lens state information. Then, the determined spatial mode and camera state data are sent to the computer video generation unit 7 in response to a request from the computer video generation unit 7 or without a request from the display mode determination unit. Further, the determined spatial mode is sent to the video composition unit 3 in response to a request from the video composition unit 3 or without a request from the video composition unit 3.
[0027]
Reference numeral 7 denotes a computer video generation unit which extracts necessary data from the data unit 8 based on the camera state data and the space mode from the space management unit 10 and generates a computer video. The generated computer image is sent to the image composition unit 3.
[0028]
Reference numeral 8 denotes a data portion, which is composed of, for example, a hard disk and possesses data to be transferred to the computer video generation portion 7.
[0029]
Data stored in the data section 8 includes text information, panoramic video, three-dimensional CG (computer graphics) data for generating a computer graphics image to be combined with a live-action image in the live-action mode, and presentation in the VR space mode. The above-described data for generating a computer graphics image is stored. Then, the data unit 8 sends appropriate data to the computer video generation unit 7 in response to a request from the computer video generation unit 7. For example, when the computer image generation unit 7 requests 3D CG data to be combined with the visual field of the imaging unit 1, the data unit 8 selects the 3D data included in the visual field of the imaging unit 1 from the stored 3D CG data. CG data is extracted and transmitted. The data unit 8 is not limited to a hard disk, and any medium that can hold data may be used. For example, the data unit 8 may be composed of a tape or a memory.
[0030]
Reference numeral 9 denotes a display unit which displays the composite video signal sent out from the video composition unit 3. At this time, if the display unit 9 moves integrally with the imaging unit 1, it is intuitive and easy for the user to look into the telescope and the like, but this is not always necessary. 1 may be movable and the display unit 9 may be fixed. In addition, a plurality of display units 9 may be provided, and in this case, a large number of people can see the synthesized video.
[0031]
The control of the present embodiment having the above configuration will be described with reference to FIG. FIG. 2 is a flowchart for explaining a processing procedure in the video composition apparatus of this embodiment.
[0032]
In step S0, data preparation is performed. In step S0, the above-described data to be stored in the data portion 8 is prepared. Each data needs to correspond in advance with the position in the real world. For example, in the case of a three-dimensional CG, it is necessary to specify which position in the real world corresponds to the position. In the case of a panoramic video, it is necessary to specify which part of the panoramic video corresponds to which part of the real world.
[0033]
When a panoramic image is stored in the data unit 8, for example, a panoramic image taken in a different time zone or place can be considered as the stored panoramic image. By storing panoramic images taken in different time zones in advance, it is possible to reproduce a clear view from the observatory even when the observatory is overcast. Further, by storing panoramic images taken at different places, it is possible to reproduce the panoramic pictures taken at completely different places as if they were there.
[0034]
The panoramic video stored in the data unit 8 is captured by the imaging unit 1, for example. In this case, as shown in FIG. 14, the video captured by the imaging unit 10 is captured by the captured video capture unit 11 and the captured video is sent to the data unit 8. The panoramic video can be captured by the imaging unit 1 and stored in the data unit 8 as described above, but any method may be used as long as the panoramic video can be stored in the data unit 8.
[0035]
When a panoramic video is stored in the data portion 8, the panoramic video may include a blank portion. As a method of expressing a blank in a panoramic video, for example, there is a method in which an α value representing transparency is given to each pixel of the panoramic video and an α value of a blank pixel is set to zero. In the captured panoramic video, if the α value of the portion that does not require the synthesis of the panoramic video is set to 0, when the video synthesis unit 3 synthesizes the captured video and the panoramic video, the α value becomes 0. By synthesizing the photographic video and synthesizing the panoramic video to the portion where the α value is not 0, it is possible to display the panoramic video and the photographic video in one synthesized video. At this time, if all the α values of each pixel of the panoramic video are set to 1, a synthesized video including only the panoramic video without including the photographed video is synthesized at the time of video synthesis.
[0036]
A method for giving an α value to each pixel of the panoramic video will be described with reference to FIG. In step S0, a panoramic video I is shot by shooting a surrounding scene from the observation room, and this panoramic video I includes an indoor video R and an outdoor video P. When the α value of the indoor video R is set to 0 and the α value of the outdoor video P is set to 1, when the video synthesis unit 3 performs the video synthesis, the shot video is synthesized for the indoor portion, and the outdoor portion is Can synthesize outdoor video P. This makes it possible to see a real-world captured image when looking at the room and a pre-captured panoramic image when looking outside.
[0037]
Here, a method using the α value is given as a method for expressing a blank portion in a panoramic video, but this method may be any method as long as it can express a blank portion in a panoramic video.
[0038]
Further, the panoramic video data stored here may be, for example, data obtained by rendering a 3D CG as a panoramic video in advance. In general, when generating 3D CG in real time, the accuracy of 3D CG is limited due to limitations of drawing ability, etc., but using this method, 3D CG models can be refined without limitation. In addition, it is possible to further improve the quality of 3D CG images. At this time, if the α value indicating the transparency is given to each pixel of the rendered panoramic image, and the α value of the portion where the CG is not drawn is set to 0 in the panoramic image, the image composition unit 3 causes the captured image and the computer image to be displayed. When synthesizing the three-dimensional CG rendered in advance in the photographed video, it is possible to synthesize the photographed video for the portion where the α value is 0.
[0039]
When storing a three-dimensional CG in the data unit 8, for example, a virtual character can be considered as the stored three-dimensional CG. Here, the 3D CG data stored in the data unit 8 includes not only geometric information such as position, size, and shape, but also data such as constraint relations and motion. For example, by storing information on the shape and movement of the virtual character in the data unit 8, it becomes possible to make the virtual character appear in the real world.
[0040]
Further, the depth information of the real world may be input to the data unit 8 in advance and used. By having and using the depth information of the real world in advance, it is possible to synthesize video that correctly reflects the context of the real world video and computer video. For example, when you want to move a 3D CG character between a building and a building that does not have real-world depth information, the entire body of the 3D CG character will be displayed in front of the building. As shown in FIG. 3, the relationship between the building and the character is not correctly reflected. On the other hand, if you have real-world depth information, compare the depth to the character and the depth to the building, and do not generate computer images of the part where the depth to the character is greater than the depth of the building. Thus, as shown in FIG. 4, it is possible to generate a composite image that correctly reflects the context of the building and the character.
[0041]
Providing depth information in the real world can be realized, for example, by providing a three-dimensional model in the real world. When the depth information of the real world is given as a three-dimensional model, it is possible to express a contact relationship such as placing a three-dimensional CG character on a building as shown in FIG. A three-dimensional model of the real world can be generated from map data, for example.
[0042]
Here, a method for generating a three-dimensional model of the real world from map data will be described with reference to FIGS. Assuming that there is a map as shown in FIG. 12, buildings E, F, G, and H are drawn in the map, and height information of the buildings E, F, G, and H is obtained in some way. Shall. Here, if an XYZ coordinate system as shown in FIG. 13 is defined with the lower left corner of the map as the origin and 1 m as one unit, the building H can be expressed as a rectangular parallelepiped with height 10, width 15, and depth 15. Other buildings E, F, and G can be expressed in the same manner.
[0043]
As described above, a three-dimensional model of the real world can be generated from the map data. Although the coordinate system as shown in FIG. 13 is defined here, it is needless to say that a three-dimensional model can be generated from map data regardless of the coordinate system. Further, the method of providing a 3D model in the real world is not limited to the above method, and any method may be used as long as it can construct a 3D model of the real world.
[0044]
Further, the depth information in the real world is not necessarily a three-dimensional model, and may be a two-dimensional model as shown in FIG. 6, for example. When observing from one viewpoint, it is not always necessary to have a three-dimensional model of the real world, and the real world depth data obtained by a range scanner or the like may be used. The method of providing depth information in the real world is not limited to the above method, and for example, a method using a distance image may be used, and any method may be used as long as the depth to the real world can be understood.
[0045]
After data preparation, the system is started in step S1. In step S2, a video is acquired from the imaging unit 1, and the acquired captured video is sent to the captured video capturing unit 2 as an NTSC video signal, for example. At this time, the captured video sent from the imaging unit 1 to the captured video capturing unit 2 is not limited to the NTSC video signal, and any video can be used as long as the video can be expressed.
[0046]
The captured video captured by the imaging unit 1 is converted into an appropriate format by the captured video capturing unit 2 and sent to the video synthesis unit 3. The captured video capturing unit 2 converts, for example, an NTSC video signal sent from the imaging unit 1 into digital data and sends it to the video synthesis unit 3. The video sent out by the imaging unit 1 is not limited to the NTSC video signal and represents a video. The captured video that the captured video capture unit 2 sends to the video synthesis unit 3 is not limited to digital data, and any video can be processed as long as the video synthesis unit 3 can process the video.
[0047]
In step S <b> 3, the posture measurement unit 5 detects the posture of the imaging unit 1, and the detected posture information is sent to the space management unit 10. In step S <b> 4, the lens state control unit 6 detects lens state information such as a zoom value and a focus value of the lens used in the imaging unit 1, and the detected lens state information is sent to the space management unit 10.
[0048]
In step S <b> 5, the space management unit 10 estimates the field of view of the imaging unit 1 from the posture information sent from the posture measurement unit 5 and the lens state information sent from the lens state control unit 6. Get the data of the range included in the field of view.
[0049]
Here, there are various methods for obtaining the field of view of the imaging unit 1 from the posture information sent from the posture measuring unit 5 and the lens state information sent from the lens state control unit 6, but here one of them is used. The method is shown. Generally, since the field of view can be obtained by obtaining the position of the lens center of the lens used in the imaging unit 1 and the viewing angle of the imaging unit 1, first, from the posture information sent from the posture measuring unit 5, A method for obtaining the position of the lens center of the lens used in the imaging unit 1 will be described, and then a method for obtaining the viewing angle of the imaging unit 1 from the lens state information sent from the lens state control unit 6 will be described.
[0050]
A method for obtaining the position of the lens center of the lens used in the imaging unit 1 from the posture information sent from the posture measurement unit 5 will be described with reference to FIG. 10 and FIG. FIG. 8 is an external view of the video composition apparatus of the present embodiment. The imaging unit 1 and the display unit 9 are integrally supported by the support unit 4 and can be rotated in the vertical direction and the horizontal direction around the rotation center O. Since the support portion 4 is fixed to the floor, the positional relationship between the floor and the rotation center O is unchanged. Therefore, if the relative positional relationship between the rotation center O and the lens center LO is known, the position of the lens center LO with respect to the real world is uniquely determined.
[0051]
Here, the XYZ coordinate system with the rotation center O as the origin is defined as shown in the figure, and the left and right rotation angles φ and the upper and lower rotation angles θ are defined as shown in FIG. When the distance from the lens center LO to the rotation center O of the lens used in the imaging unit 1 is expressed as d, the coordinate of the lens center LO is x = −d sin θ sin φ.
y = dsin θ cos φ
z = d cos θ
Represented as: In this example, the XYZ coordinate system is defined as shown in FIG. 9. However, regardless of how the coordinate system is defined and how the vertical and horizontal rotation angles are defined, the horizontal rotation angle and the vertical direction are defined. It goes without saying that the position of the lens center of the lens used in the imaging unit 1 can be obtained from the rotation angle.
[0052]
A method for obtaining the viewing angle of the imaging unit 1 from the lens state information sent from the lens state control unit 6 will be described with reference to FIGS. First, the focal length of the lens used in the imaging unit 1 is calculated from the zoom value sent from the lens state control unit 6. Here, it is assumed that the zoom value sent from the lens state control unit 6 is a value outputted from an 8-bit encoder provided in the imaging unit 1, and the value and the focal length of the lens are as shown in the table of FIG. It is assumed that Then, by using the table of FIG. 10 and interpolating values not on the table, the focal length of the lens used in the imaging unit 1 is obtained using the zoom value sent from the lens state control unit 6. Can do. However, when the lens state control unit 6 can directly output the focal length, it is not necessary to calculate the focal length using a table as shown in FIG.
[0053]
Assuming that the focal length of the lens used in the imaging unit 1 obtained as described above is f and the horizontal width of the imaging surface provided in the imaging unit 1 is x, the viewing angle ψ in the horizontal direction of the imaging unit 1 is It can be calculated by the following formula.
[0054]
ψ = 2 arctan (x / 2f)
Similarly, assuming that the height of the imaging surface provided in the imaging unit 1 is y, the viewing angle μ in the vertical direction of the imaging unit 1 can be obtained from the following equation from FIG.
[0055]
μ = 2 arctan (y / 2f)
As described above, the position of the lens center of the lens used in the image pickup unit 1 and the viewing angle of the image pickup unit 1 are obtained from the posture information sent from the posture measurement unit 5 and the lens state information sent from the lens state control unit 6. Therefore, the field of view of the imaging unit 1 can be obtained. At this time, the method as described above is used as a method for obtaining the field of view of the imaging unit 1, but any method may be used as long as the method allows the field of view of the imaging unit 1 to be obtained.
[0056]
In step S <b> 6, the space management unit 10 determines the space mode based on the posture information sent from the posture measurement unit 5, the lens state information sent from the lens state control unit 6, and the spatial data in the data unit 8.
[0057]
As a determination method of the spatial mode, when the current spatial mode is the real-shooting mode, the zoom value of the lens is referred to, and when the zoom value of the lens is less than a preset value, the mode is not changed.
[0058]
When the zoom value of the lens is equal to or larger than a preset setting value (hereinafter, for example, the case where the setting value is set as the maximum zoom value), the space area of the central field of view (the central area of the display unit) Search corresponding to Lud over data from the data section 8. Then, when Lud over data to correspond to the data portion 8 is retrieved, to change the spatial mode to VR spatial mode.
[0059]
Note that data in the space area of the central region of the field of view if not, does not change the mode, photographed mode is maintained. If the current space mode is the VR space mode, the mode is not changed until the mode is canceled.
[0060]
In step S7, the computer video generation unit 7 acquires data corresponding to the spatial mode determined in step S6 from the data unit 8, and generates a computer video. The generated video is sent to the video composition unit 3.
[0061]
In step S8, in accordance with the spatial mode determined in step S6, the video image sent out from the shot video capturing unit 2 and the computer video sent out from the computer video generation unit 7 are synthesized in the video synthesis unit 3. Alternatively, the video is generated only from the computer video. The generated video is sent to the display unit 9.
[0062]
In step S <b> 9, the display unit 9 displays the video information sent from the video composition unit 3. Thereafter, in step S10, it is checked whether or not to end the system. If the system is to be ended, the system is ended in step S11. If not, the process returns to step S2, and the above-described processing is repeated.
[0063]
As described above, according to the present embodiment, when the zoom value of the lens reaches a preset value or more in the magnifying display device, the target object is noticed by increasing the zoom value. By switching to a computer graphics image or the like, it is possible to display in more detail, display information about the object, or walk into the object.
[0064]
In other words, when zooming beyond a possible range of the enlargement device to a certain object, the VR space is such that the information that is not possible for the zoom object is switched and displayed, or the object enters the object. Can be presented. Accordingly, it is possible to display an image that satisfies the user's request that appears in the zooming action of a telescope or the like, that the user wants to see the object in more detail, wants to see it closer, or wants to look into it.
[0065]
<Modification>
An operation unit is provided and receives input from a user of the synthesis apparatus of the present embodiment. It also has a sound generator and presents the sound to the user. In addition, a composite video storage unit that stores the composited video and a storage video output unit are provided. In the first embodiment, the VR space in the VR space mode may be a VR space that is not related to the zoom target. A home page of WWW (World Wide Web) may be displayed.
[0066]
In addition, the video composition device according to the present embodiment can be used as a telescope at an observation deck or the like, but is also installed in a museum or the like, for example, to display a virtual video superimposed on a real world exhibit. It can also be used as a display terminal.
[0067]
In addition, the video composition device of the present embodiment is not limited to use in an observation deck or a museum, and can be used in all situations where the video composition device of the present embodiment can be used effectively. It is.
[0068]
The object of the present embodiment is to supply a storage medium (or recording medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and the computer (or CPU or MPU of the system or apparatus). ) Is also achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present embodiment. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.
[0069]
Further, after the program code read from the storage medium is written to a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the card or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.
[0070]
【The invention's effect】
According to the present invention, when the control of the optical system has reached a set value or more, it is desired to switch the target object to a computer graphics image or the like so as to see the user's target object in more detail. Can meet the demand.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a schematic configuration of a video composition device according to a first embodiment of the present embodiment.
FIG. 2 is a flowchart illustrating a processing procedure of the video composition device in the first embodiment.
FIG. 3 is a diagram in a case where the context of a real-world building and a virtual character is not expressed.
FIG. 4 is a diagram in the case where the context of a real-world building and a virtual character is expressed.
FIG. 5 is a diagram in a case where a contact relationship between a real-world building and a virtual character is expressed.
FIG. 6 is a diagram when a real-world model is expressed as a two-dimensional model.
FIG. 7 is a diagram illustrating a case where an α value is given to a part of a panoramic video.
FIG. 8 is a diagram illustrating a method for obtaining the position of the lens center of a lens used in the imaging unit from posture information sent from the posture control unit.
FIG. 9 is a diagram illustrating a method for obtaining the position of the lens center of a lens used in the imaging unit from posture information sent from the posture control unit.
FIG. 10 is a diagram illustrating a method for obtaining a focal length of a lens used in an imaging unit from lens state information sent from a lens state control unit.
FIG. 11 is a diagram illustrating a method for obtaining a viewing angle of a lens from a focal length of the lens used in an imaging unit.
FIG. 12 is a diagram illustrating a method of generating a real-world three-dimensional model from map data.
FIG. 13 is a diagram for explaining a method of generating a real-world three-dimensional model from map data.
FIG. 14 is a diagram for describing a case where a panoramic video is captured by an imaging unit and stored in a data unit.

Claims (8)

A video image synthesizing device having a spatial mode for displaying an image based on a photographed image taken by an imaging unit and a spatial mode for displaying an image based on computer graphics,
An imaging unit that shoots the real world through a lens and generates a real image;
A support unit that supports the imaging unit, and has a movable unit that allows a user to manually move the imaging unit;
A posture measuring unit for measuring a relative posture of the imaging unit and the support unit;
A lens state control unit for controlling the state of the lens;
A data portion for holding data for generating computer graphics;
A space management unit that sets a spatial mode based on posture information from the posture measurement unit, lens state information from the lens state control unit, and data held in the data unit;
A computer video generation unit that generates a computer video using data held in the data unit based on a spatial mode set in the space management unit;
A display unit for displaying the video generated by the video generation unit;
A video generation unit that generates a video to be displayed from a real image generated by the imaging unit and a computer video generated by the computer video generation unit;
The space management unit is configured such that when the zoom value of a lens used in the imaging unit is equal to or larger than a preset setting value and computer graphics displayed in a specific area of the display unit are held in the data unit. A video composition apparatus which sets a spatial mode for displaying an image based on the computer graphics.   The video synthesizing apparatus according to claim 1, wherein the support unit further supports the display unit that displays the video generated by the video generation unit and the imaging unit integrally.   The video composition apparatus according to claim 1, wherein the set value is a maximum value of a zoom value.   The video composition apparatus according to claim 2, wherein the specific area is a center of the display unit.   The video composition device according to claim 1, wherein the support portion is fixed.   The video composition apparatus according to claim 1, wherein the support portion is movable. An information processing method for enabling display and display of the image which is based on computer graphics images and photographed base over scan the photographed image by the imaging unit,
Input the posture information and zoom information of the imaging unit,
It is determined whether the zoom value of the imaging unit is larger than a set value based on the zoom information,
It is determined whether or not computer graphics displayed in a predetermined area of the captured image based on the posture information is held in the data portion,
If the zoom value is larger than the installation value and computer graphics displayed in a predetermined area of the photographed image are held in the data portion, a computer image corresponding to the computer graphics is generated,
An information processing method for displaying an image based on the generated computer video.   A computer program for realizing the information processing method according to claim 7 on a computer.

Images