JP2001177850A - Image signal recorder and method, image signal reproducing method and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a diagram viewing an object directly above and obliquely from a position in a detected three-dimensional space by detecting the position of the object in the three-dimensional space by using a plurality of photographed still pictures. SOLUTION: An image shift amount is detected from an image signal obtained by a lens group 21 and a CCD group 22, and data obtained by an angular velocity sensor 33 are recorded in a recording medium 14 together with the image signal. In the case of reproduction, a position of the object in a three-dimensional space is detected from the recorded data and the image signal. A diagram viewing the object directly above the position and a diagram viewed obliquely from the position are generated based on the position. In the case of photographing, a locus L of a moved digital camera is indicated on the diagrams viewed just above and viewed obliquely. Selecting an optional position in the diagrams viewed just above and viewed obliquely can display an image that is viewed from the selected position.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image signal recording apparatus capable of recording an image signal and its recorded state as sub-data together with the image signal, and reproducing the recorded image signal and sub-data. And a method, an image signal reproducing method, and a recording medium.

[0002]

2. Description of the Related Art Conventionally, a VTR-integrated digital camera (hereinafter simply referred to as a digital camera) can record and reproduce moving images and still images. A method of synthesizing a plurality of images recorded in the digital camera to generate one high-definition still image or high-definition panoramic still image is already known.

That is, after a plurality of shot images are subjected to image distortion correction, rotation correction, enlargement / reduction, etc.,
A synthesized high-definition still image or high-definition panoramic still image is generated.

[0004]

However, it has not been possible to detect the position of a subject in a three-dimensional space from images obtained by photographing a plurality of the same subject. Further, it has not been possible to generate a planar image, for example, an image viewed from directly above and an image viewed from obliquely from images obtained by photographing a plurality of the same subject.

Accordingly, an object of the present invention is to detect the position of a subject in a three-dimensional space from an image obtained by photographing a plurality of the same subject, and to obtain an image viewed from directly above the detected position in the three-dimensional space. It is another object of the present invention to provide an image signal recording device and method, an image signal reproducing method, and a recording medium that can generate an image viewed obliquely.

[0006]

According to a first aspect of the present invention, there is provided an image signal recording apparatus which captures a plurality of images and records the captured image signals.
Posture detecting means for detecting a position (x, y, z) in a dimensional space, a pan angle p, a tilt angle q, and a roll angle r; an image sensor for converting an image of a subject incident from the subject into an image signal; Compression means for compressing the image signal; recording of the compressed image signal; position (x, y, z), pan angle p, tilt angle q in the three-dimensional space detected by the attitude detection means
And a recording means for recording the roll angle r as sub-data.

According to an eighth aspect of the present invention, there is provided an image signal recording method for capturing a plurality of images and recording the captured image signals, wherein the position (x, y, z), the pan angle p, the tilt angle q, and the roll angle r are detected, the image of the subject incident on the image sensor from the subject is converted into an image signal, the image signal is compressed, and the compressed image signal is recorded. An image signal recording method characterized by recording the detected position (x, y, z) in the three-dimensional space, the pan angle p, the tilt angle q, and the roll angle r as sub-data. .

According to a ninth aspect of the present invention, a plurality of image signals obtained by imaging the same subject from a plurality of positions are recorded, and the position (x, y, z), the pan angle p, the tilt angle q, and the roll angle r are used as sub-data in an image signal reproducing method for reproducing a recorded recording medium, and the image signal and the sub-data recorded on the recording medium are reproduced. , Detecting a position of a subject in a three-dimensional space based on sub-data and a plurality of image signals, generating a plane image from the detected position in the three-dimensional space, and displaying the plane image. This is a signal reproduction method.

According to a twenty-second aspect of the present invention, in a recording medium on which a still image and / or a moving image signal is recorded, a position (x, y, z) of the imaging means in a three-dimensional space, a pan angle p, A recording medium characterized in that a tilt angle q and a roll angle r are recorded as sub-data together with an image signal.

[0010] The image signal obtained from the image sensor is compressed,
Along with the compressed image signal, posture information of a digital camera including data of a position (x, y, z) and a pan angle p, a tilt angle q, and a roll angle r in a three-dimensional space is recorded as sub data. Will be recorded. The position of the subject in the three-dimensional space is detected from the recorded sub data. An image viewed from directly above the detected position in the three-dimensional space, for example, a floor plan when inside a building, and a map when outdoors. Similarly, an image viewed obliquely from the detected position in the three-dimensional space, for example, in the case of the inside of a building, is a perspective view penetrating a certain surface, and in the case of outdoors, a bird's-eye view is generated.

[0011]

DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic external view of an embodiment to which the present invention is applied. In this embodiment, as an example of a digital camera, a digital camera in which a main body 1 is provided with a lens group 2 including five lenses as shown in FIG. 1A is used. Further, as another example of the digital camera, as shown in FIG. 1B, a camera in which a lens group 2 including seven lenses is provided in a main body 1 may be used. As in the digital camera shown in FIGS. 1A and 1B, a lens group 2 including a plurality of lenses whose positional relationship is fixed includes a plurality of directions such as the front of the main body 1, its left and right side surfaces, and its oblique direction. You can shoot at the same time. Thus, a wide-angle, high-resolution digital camera is used.

When only one lens is provided on the main body 1, only the image of the front can be taken, so that the operator of the digital camera intentionally changes the direction of the optical axis, and Shoot in all directions.

A schematic configuration of the first embodiment will be described with reference to FIG. For blocks with the same function,
The same reference numerals are given and the description is omitted. The lens group 2 captures images of a plurality of subjects. The captured images of the plurality of subjects are supplied from the lens group 2 to the recording medium 14 and the display unit 15 as image signals. This lens group 2 is controlled by a servo system 12. Servo system 12
Is provided with a control unit such as a zoom, a focus, an iris, and an electronic shutter corresponding to each of the lens groups 2. Data obtained from the servo system 12 is stored in a recording medium 14.
Supplied to The servo system 12 is controlled by a system controller (system computer) 11.

The sensor group 13 includes a plurality of sensors for the main body 1. As an example, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, a GPS, and the like are provided. Data obtained from these sensor groups 13 is supplied to a recording medium 14. As will be described later, the attitude of the digital camera can be known from the sub data obtained from the sensor group 13. The sensor group 13 is controlled by the system controller 11. In the recording medium 14, the supplied plurality of image signals are respectively compressed and recorded as compressed image signals. The data supplied from the servo system 12 and the data supplied from the sensor group 13 are recorded on the recording medium 14 as sub-data together with the compressed image signal. The recording medium 14 is controlled by the system controller 11. Display 1
At 5, the supplied plurality of image signals are displayed.

When taking a picture with this digital camera, the picture taken while moving is compressed and recorded on the recording medium 14 as a compressed image signal, and at the same time data from the servo system 12 of the digital camera and the sensor group 13
Is also recorded as sub-data.

As described above, in this embodiment, a wide-angle and high-resolution digital camera is used. Further, it is better that the photographing speed of the digital camera is faster, but it is not always necessary to be able to photograph 30 images per second. In some cases,
The shooting range may be divided by a plurality of digital cameras.

At this time, the following sub data is simultaneously recorded in each frame. Date and time: Year, Month, Day, Hour, Min, Sec, Frame Zoom angle: Zoom Focal length: Focus Iris value: Iris Shutter speed: Shut Acceleration sensor: Ax, Ay, Az Angular velocity sensor: Vx, Vy, Vz Geomagnetic sensor: M1, M2 Position by GPS: Gx, Gy, Gz.

The parameters representing the attitude of the digital camera are, for example, six positions x, y, z and a pan angle p, a tilt angle q, and a roll angle r in the three-dimensional space of the digital camera. Among them, the tilt angle q and the roll angle r are determined by using the acceleration sensor as a tilt sensor.
Can be obtained statically.

Further, one sensor can detect ± 90 degrees in one direction. However, since the accuracy around 90 degrees is inferior, a sensor for +90 degrees to +45 degrees, and +45 degrees to -45 degrees
By providing a sensor for degrees and a sensor for -45 degrees to -90 degrees, a range of ± 90 degrees can be detected with high accuracy.

The pan angle p can be known to some extent by an angular velocity sensor and an integrator. However, at this time, the azimuth (the east-west north-south) which becomes the absolute value needs to use the azimuth sensor or designate in advance. Furthermore, since the integrator has a disadvantage of accumulating errors, it cannot be determined whether the integrator is stationary for a long time or rotating very slowly.
Therefore, the movement of the digital camera is accurately determined by monitoring the change of the captured image every frame.

The position x, 3 in the three-dimensional space of the digital camera
Similarly, y and z can be detected by an acceleration sensor and an integrator in three directions. At this time, the movement of the digital camera is accurately determined by monitoring the change of the image every frame.

Here, a detailed block diagram of the first embodiment is shown in FIG. The image of the subject incident through the lens group 21 is supplied to the CCD image sensor 22. Lens group 2
The zoom control 1 and the focus control 1 are performed by the servo system 12 (not shown) as described above. CC
In the D image pickup device 22, incident light from a subject is accumulated as electric charges. In the CCD image pickup device 22, on / off of an electronic shutter is controlled by the servo system 12. Thus, the electronic shutter of the CCD image sensor 22 is driven, and the supplied image of the subject is captured. The captured image of the subject is digitized by an A / D converter (not shown), and is temporarily stored in the image memory 28 via the compression circuit 23 as a digital image signal (hereinafter, referred to as an image signal). .

The image memory 28 has a capacity for storing images of several frames or several fields. The image signals stored in the image memory 28 are sequentially compressed by the compression circuit 23. As an example, JPEG (Joint Photographic) is applied to an image captured as a still image.
Experts Group) and MPEG (Moving Picture Experts Group)
Is applied. The compressed image signal thus processed is supplied to the recording medium 14. The compressed image signal supplied to the recording medium 14 is recorded under the control of the system controller 11. As an example of the recording medium 14, a recording medium appropriately selected in consideration of an access speed, a storage capacity, and the like from a magnetic tape, a magnetic disk, an optical disk, a semiconductor memory, or the like is used.

A compressed image signal is read from the recording medium 14 under the control of the system controller 11 in accordance with a designation from the various operation keys 37. The read compressed image signal is
Once stored in the image memory 28 via the expansion circuit 25,
The decompression circuit 25 sequentially performs decompression processing. That is, in the expansion circuit 25, JPEG or MPEG
Is decoded. The decompressed image signal is supplied to a decompression circuit 2
5 is displayed on the display unit 15 and is also displayed on an external TV monitor or the like via the video output terminal 26.

The above-mentioned image memory 28 is used when compressing an image signal and when expanding a compressed image signal. At this time, the area to be compressed and the area to be decompressed may be divided according to the address, or the stored signal may have a compression flag and / or
Alternatively, an extension flag may be added. Also, a memory for compression and a memory for decompression may be provided separately. The image memory 28 has
Although not shown, a memory for storing the current image signal and a memory for storing the image signal image of the previous frame are provided.

In the first embodiment, the current image signal is supplied to and stored in the memory 30a of the image movement amount detector 29. The image signal of one frame before is supplied to the memory 30b of the image movement amount detection unit 29 and stored. The motion detection circuit 31 performs motion detection based on the image signals stored in the memories 30a and 30b. On the basis of the result of the motion detection, the arithmetic circuit 32 generates a horizontal translation component Mx and a vertical translation component My of the image signal.
Is calculated. The calculated translation components Mx and My
Is supplied to the system controller 11.

More specifically, in order to perform motion detection with an accuracy of about 1/2 of one pixel, the image signals stored in the memories 30a and 30b are enlarged about twice in both the vertical and horizontal directions. . At this time, since processing is performed for each small block, the memory 30b only needs to have the block size. On the other hand, the other memory 30a
A memory larger than the memory 30b by the search range is required.

When an image in a certain block is a flat image, the position detection cannot be performed by the image movement amount detecting section 29. In such a case, in order not to output an erroneous value, the variance Va of the image of the block is calculated, and when the calculation result is small, the position of the block is not detected. Here, the variance Va is Va = Σ (yi ² ) / K− (Σ (yi / K)) ² (where, yi: luminance value, K: number of pixels in the block). Also, i is a block number, vertical and horizontal translation components are obtained for each block, the obtained horizontal translation component for each block is x [i], and the vertical translation component for each block is y [ i].

When the translation components x [i] and y [i] obtained from many blocks have the same value, it is considered that the entire screen has been translated. If the translation components x [i] and y [i] do not have the same value, there is inclination or expansion / contraction, but the value or average value at the center is used as the translation components Mx and My of the image signal. I do.

As described above, the image movement amount detection unit 29
The position of the current image signal with respect to the image signal before the frame is checked, and the translation components Mx and My are output. The translation components Mx and My are supplied to the system controller 11. The translation components Mx and My supplied to the system controller 11 are supplied to a synthesis circuit 34.

FIG. 3 shows an example of only the angular velocity sensor from the sensor group 13. Lateral angular velocity Sx detected by angular velocity sensors 33X and 33Y of angular velocity sensor 33
The angular velocity Sy in the vertical direction is supplied to the system controller 11. The angular velocities Sx and Sy supplied to the system controller 11
Is supplied to the synthesis circuit 34. The combining circuit 34 combines the translation components Mx and My and the angular velocities Sx and Sy. The synthesized translation component Mx is supplied to the integration circuit 35X, and the synthesized translation component My is supplied to the integration circuit 35Y.

In the integrating circuits 35X and 35Y, the supplied vertical and horizontal translation components are integrated, and a horizontal displacement X and a vertical displacement Y are obtained.
The movement amounts X and Y are determined by the D / A converters 36X and 3
6Y. In the D / A converters 36X and 36Y, the supplied movement amounts X and Y are digitized,
It is supplied to the recording medium 14. On the recording medium 14, the movement amounts X and Y are recorded together with the compressed image signal.

Here, the vertical synchronizing signal extracted from the video signal is used as a reset signal as integration signals 35X and 35X.
5Y. That is, the integration circuits 35X and 35Y are reset for each frame. Further, the resetting of the integration circuits 35X and 35Y may be performed on a field basis. This reset is performed during a period in which the exposure of the CCD image sensor 22 is invalidated. Specifically, the integration circuits 35X and 35Y are activated at the timing of starting the exposure of the CCD image sensor 22, and the integration circuits 35X and 35Y are reset at the timing of ending the exposure.

As described above, if the moving speed is determined from the change in the image signal, correct control can be performed irrespective of the parallel or rotational movement of the digital camera. It is detected how much the current image signal has shifted from the image signal one frame before. From this detection result, the moving speed per frame is obtained. By integrating this, the instantaneous position in one frame can be continuously obtained.

In the first embodiment, the synthesizing circuit 34 synthesizes the translation components Mx and My and the angular velocities Sx and Sy, and outputs the synthesized output to the integrating circuit 3.
5X and 35Y, but only the translation components Mx and My are integrated into the integration circuits 35X and 35Y.
Y may be supplied.

FIG. 4 shows an embodiment of a reproducing system for reproducing the photographed image signal. Computer 4
1. A recording medium 1 on which a compressed image signal is recorded in a reproducing apparatus including a keyboard 42, a mouse 43, and a display unit 44.
4 are connected. The recording medium 14 is controlled by the computer 41 in accordance with the operation of the keyboard 42 and the mouse 43.

For example, a recorded compressed image signal is read from the recording medium 14 and decompressed. The sub data composed of the data of the servo system 12 and the data of the sensor group 13 recorded together with the recorded compressed image signal is stored in the recording medium 14.
Is read from. From the read sub-data, it is determined from which position the digital camera is shooting in which direction. Further, the position of the subject in the three-dimensional space is determined based on the plurality of image signals. As a result, the position in the three-dimensional space of many things, such as the position of the wall, the height of the ceiling, the arrangement of furniture, and the like, which are photographed as the subject, are determined.

Here, in the first embodiment, at the time of reproduction, the position and direction of the subject are determined using the information obtained from the reproduced image signal together with the sub data of the digital camera. For this reason, the sub data to be recorded together with all the images has to be recorded even if it is not necessary. Therefore, an example in which sub-data is recorded according to a change in the posture of the digital camera during recording will be described with reference to FIG.

The data obtained from the sensor group 13 is supplied to the recording medium 14 and also to the attitude detection circuit 51. The posture detection circuit 51 detects the posture of the digital camera from the data supplied from the sensor group 13.
The supplied data is output from the attitude detection circuit 51 to the system controller 1.
1 and the recording medium 14. Attitude detection circuit 51
Is controlled by the system controller 11. Syscon 11
Controls the recording medium 14 according to the supplied data. Specifically, when it is determined from the data that the posture of the digital camera has not changed, the data supplied from the sensor group 13 is not recorded on the recording medium 14, and when it is determined that the posture has changed, Data supplied from the group 13 is recorded on the recording medium 14.

As described above, since the posture detection is performed at the time of recording, the image is recorded on the recording medium 14 for each frame, but the data supplied from the sensor group 13 is recorded only when the posture changes. I do. When sub data is not required, for example, in a corridor, it is also possible to record only an image and not to record sub data.

Here, an example of specifying the position of a subject in a three-dimensional space from a captured two-dimensional image signal will be described with reference to FIG. In this example, for the sake of simplicity, three images are taken based on two image signals of the same subject.
Identify a position in dimensional space. When the subject A is located at an arbitrary position (h1, v1) within the angle of view of the digital camera having the origin P1, the focal length f1, and the posture R1 of the optical axis, the subject A is positioned at a position within the angle of view with the origin P1 h1, v1) on a straight line l1. Direction vector d1 of straight line l1
Is obtained by normalizing the length to 1 to obtain the direction vector d1.

[0042]

(Equation 1)

Is as follows. Using this equation (1), a straight line l1
The coordinates q1 of the above arbitrary position are as follows: q1 = d1 t + P1 (2) Here, t is a parameter.

Similarly, the origin P2 of the optical axis, the focal length f2,
Arbitrary position (h2
, V2) exists on the straight line l2.
When the length of the direction vector d2 of the straight line l2 is normalized to 1 to obtain the direction vector d2,

[0045]

(Equation 2)

Is as follows. Using this equation (3), a straight line l2
The coordinates q2 of the above arbitrary position are as follows: q2 = d2 t + P2 (4)

As described above, regarding the first image, the origin P1, the focal length f1, and the posture R1 of the optical axis of the digital camera are obtained.
From the position (h1, v1) within the angle of view, the direction of the optical axis, that is, a straight line l1 is determined. As for the second image, similarly to the first image, the origin P2 of the optical axis of the digital camera, the focal length f2, the attitude R2, and the position (h2,
v2), the direction of the optical axis, that is, a straight line l2 is determined. The intersection of these two optical axes, that is, the straight lines l1 and l2, is the position of the subject 1 in the three-dimensional space. However, in general, the straight lines l1 and l2 may not intersect accurately due to a measurement error or the like. Therefore, the straight lines l1 and l
The midpoint C between points q1 and q2 connecting 2 at the shortest distance is an approximate solution of the position of the subject A in the three-dimensional space.
If there is no error at the points q1 and q2 connecting the straight lines l1 and l2 at the shortest distance, this approximate solution coincides with the true solution. Hereinafter, the midpoint C which is the position of the subject A in the three-dimensional space
Find an approximate solution of.

When the straight lines l 1 and l 2 are the shortest, they are orthogonal, so that (q 1 -q 2) · d 1 = 0 (5) (q 1 -q 2) · d 2 = 0 (6) Solving the simultaneous equations of equations (5) and (6) gives (d1 t1 + P1 -d2 t2 -P2) .d1 = 0 (d1 t1 + P1 -d2 t2 -P2) .d2 = 0, and d1 .d1 t1-d1 · d2 t2 =-(P1 · d1-P
2 · d2) d1 · d2 t1-d1 · d2 t2 =-(P1 · d1-P
2 · d2)

[0049]

(Equation 3)

[0050]

(Equation 4)

Accordingly, the midpoint C to be obtained is as follows: C = (q1 + q2) / 2 = 1/2 · (d1t1 + P1 + d2t2 + P2) (8) Is required. Here, d1, t1, d2, and t2 are known from equations (1), (3), and (7).

At this time, as the direction of the optical axis of the digital camera,

[0053]

(Equation 5)

Since the above is not enough, a rotation matrix R representing the attitude of the digital camera was used. In this rotation matrix R,

[0055]

(Equation 6)

There is the following relationship. Therefore, the rotation matrix R is

[0057]

(Equation 7)

Is as follows.

Here, an example will be described in which, when the position of the subject A in the three-dimensional space is known, an image viewed from directly above the position in the three-dimensional space is formed. The position (x, y) on the xy plane is specified by simply setting z = 0 from the position (x, y, z) in the three-dimensional space of the subject A.

At this time, generally, there is only an image taken from the horizontal direction, and almost no image taken from the horizontal direction is taken from all sides. Is processed like an illustration, and is displayed as an image viewed from directly above. As an example, in FIG. 7, the photographed subject is inside a building, and a floor plan is displayed as an image viewed from directly above.

Next, an example of forming an image obliquely from the position in the three-dimensional space when the position of the subject A in the three-dimensional space is known will be described with reference to FIG.
First, the subject A is projected onto the image plane H from a position in the three-dimensional space. As shown in FIGS. 8A and 8B, the horizontal angle of view th [deg] from the origin E (0, 0, ZE) of the optical axis on the z-axis.
p-0], and the range photographed by the digital camera with the aspect ratio Ra (= 4/3) is a quadrangular pyramid made of ABCDE. The bottom surface of the quadrangular pyramid, that is, the quadrilateral composed of ABCD on the xy plane is the image plane H photographed by the digital camera. Here, Lh = ZE tan (t
h), Lv = Lh / Ra, 2Lh, 2Lv, z =
0 is the image size on the plane.

As shown in FIG. 8C, an arbitrary position P (xp, yp, zp) in the quadrangular pyramid is projected to a position Q (x, y) on a plane of Z = 0. Triangle EQO and EPO '
Are similar, x = xp.ZE / (ZE-ZP) y = yp.ZE / (ZE-ZP) As a result, an arbitrary position P (x
p, yp, zp) is obtained as an image in the xy plane.

As described above, the origin E (0, 0, Z) of the optical axis
When viewing the direction of the origin O (0, 0, 0) from E), the position P (xp, yp, zp) in the three-dimensional space is projected as the position Q (xq, yq) on the xy plane. You. Here, the rotation of the coordinate axes is performed. For example, when the y-axis and the z-axis are rotated around the x-axis by θ [rad] to the right in the positive direction of x, the position P (xp, yp, zp) becomes P2 (x
2, y2, z2). That is, the following equation (10) is obtained.

[0064]

(Equation 8)

At this time, the position P has not moved. Only the coordinate system has changed. That is, assuming that the origin E of the optical axis and the image plane H rotate integrally with the coordinate system, on the displayed image plane H, it appears as if the position P has rotated to the left around the x-axis. In other words, the subject that has been viewed from directly above appears to be viewed from obliquely above or from the side. Further, at this time, if the subject is rotated around the z-axis before the subject is rotated around the x-axis, the subject can be viewed from any direction. Specifically, Equation (10) is multiplied by the following matrix.

[0066]

(Equation 9)

Although the image may be rotated (panned) around the y axis, it is not suitable for an image viewed obliquely, so that the description is omitted here.

Thus, the direction of the optical axis is always set to the origin O
By changing the origin E of the optical axis while aiming at (0, 0, 0), an image can be obtained as viewed obliquely. As an example of the image viewed obliquely, when the subject is outdoors, a bird's-eye view shown in FIG. 9 is obtained. Also in this case, the image is processed and displayed like an illustration as shown in FIG. 9 instead of an actually photographed image.

The trajectory of the movement of the digital camera at the time of shooting can be displayed in the image viewed from directly above and the image viewed obliquely as generated above. First,
FIG. 10 shows an example of displaying a trajectory on an image viewed from directly above. As shown in FIG. 10, the locus L of the movement of the digital camera is displayed as a dotted line on the floor plan. Actually, the locus L
Are represented by, for example, a red solid line so as to be easily distinguished in the figure. Around the center of the room, the trajectories L overlap one another. From the locus L, it can be seen that the optical axes are directed in various directions from one point in all directions in order to photograph a plurality of areas in the room.

Next, FIG. 11 shows an example of displaying a trajectory on an image viewed obliquely. As shown in FIG. 11, a locus L of the movement of the digital camera is displayed by a dotted line in a room viewed obliquely. As in FIG. 10, the locus L overlaps many times around the center of the room. From the locus L, it can be seen that the optical axes are directed in various directions from one point in all directions in order to photograph a plurality of areas in the room. Also,
The locus L reciprocates several times around the table at the corner of the room. Since it has a characteristic shape, it can be seen from the locus L that the optical axis is directed to one point from various directions.

When images in various directions from substantially one point are taken as shown in FIGS. 10 and 11, the positions may be displayed as specific marks, for example, small balls. Then, when the small ball is selected, a panoramic image obtained by synthesizing an image taken from one point is displayed.
In addition, when an arbitrary position of the displayed panoramic image is left-clicked, for example, an enlarged image is displayed centering on the clicked arbitrary position. For example, when a grid is displayed, an image in a grid direction is displayed as an enlarged image. . When right-clicked, the image is displayed as a reduced image.

Further, when a certain object is photographed from all directions, the image is displayed in an image viewed from directly above and an image viewed obliquely so as to surround the object in an arc shape. This object can be rotated and displayed, for example.

When an arbitrary position on the locus L is designated,
An image that can be viewed from the designated arbitrary position can be displayed. As an example, FIG. 12A is an image of an apartment entrance. FIG. 12B is an image of a living room, and FIG. 12C is an image of a living room different from the living room shown in FIG. 12B. FIG. 13A is an image of a Japanese-style room, and FIG. 13B is an image of a bathroom.

Then, when an arbitrary position is selected from an image viewed from directly above or an image viewed from obliquely, an image viewed from directly above or an image viewed from obliquely is displayed. The image is switched and displayed. Further, the image at the selected arbitrary position may be displayed in an area different from the image viewed from directly above or the image viewed obliquely, that is, in a so-called window.

As described above, the following operations can be performed when reproducing the photographed image signal and the sub data. (1) When the photographed image is a building, the layout is taken as an image viewed from directly above. A map is displayed, and in the case of outdoors, a map is displayed as an image viewed from directly above. (2) When a captured image is a building, a perspective view through a certain surface is displayed as an image viewed obliquely. In the case of outdoors, a bird's-eye view is displayed as an image viewed from directly above, and a three-dimensional image can be displayed by slowly rotating the display at a certain angle width. The locus L of the digital camera at the time of shooting can be displayed in the image viewed obliquely. (4) When an arbitrary position on the locus L of the digital camera is designated, an image seen from the designated arbitrary position is displayed. Can be displayed Wear. At this time, a wide-angle display is provided, and zooming can be performed when the direction of the subject is specified. (5) When an arbitrary position of the displayed image is specified, the specified arbitrary position can be displayed from various directions. An image whose arbitrary position can be seen can be displayed as a continuous still image like a moving image.

In this embodiment, by inputting image signals and sub-data of the entire interior of a plurality of rooms of a certain building to a computer and processing them, a floor plan of the room can be obtained. In addition, if you specify the location and direction,
An image of the place can be displayed.

In this embodiment, since the height direction of the digital camera is also detected, it is possible to display contour lines on the created map. Further, by photographing a subject around one rotation, three-dimensional data of the subject can be created, and the subject can be rotated three-dimensionally and displayed.

In this embodiment, a landscape can be photographed with a digital camera from a car, a bicycle, a train, or the like, so that a map can be created from the photographed image signals and sub data. However, the created map is only the photographed range. Therefore, a map created from image signals and sub-data taken by a digital camera while traveling in a valley can only map in a narrow range. On the other hand, when a distant building or a mountain is photographed for a long time, a map of a wide range can be obtained. Further, since the same subject is photographed for a long time, the subject can be stereoscopically displayed or rotated. At this time, it is sufficient that at least the front and left and right image signals and sub data have been photographed, and the upper and lower and rear portions need not be photographed. In addition, the trajectory of the travel can be displayed on the created map.

A person, an animal or a robot carrying the digital camera is roamed on an unexplored ground, and the obtained image signals and sub-data are input to a computer and processed. Since all the scenes that can be seen from this are recorded, the main ones in the image can be marked on the map because the distance and the direction are known.

In this embodiment, a specified direction from a specified position can be displayed as an image. At this time, the magnification of the displayed image can be changed, for example, from a telephoto angle of 2 degrees in the horizontal angle of view to a wide angle angle of 90 degrees. Further, all directions of ± 180 degrees in the horizontal direction and ± 90 degrees in the vertical direction can be displayed. However, when an omnidirectional image is not obtained, the display is insufficient. When it is desired to see a close-up image of a small area, a plurality of images including the area are repeatedly displayed at the speed of the moving image.

In this embodiment, when an image is being reproduced, controls such as slow reproduction, high-speed reproduction, rewinding, and pause can be performed.

In this embodiment, an image viewed from directly above and an image viewed from obliquely are generated and displayed using images captured by one digital camera.
Images taken by a plurality of digital cameras may be used. At this time, by using the position of the specific subject as a reference in the three-dimensional space, an image viewed from directly above and an image viewed obliquely are generated using images captured by a plurality of digital cameras, It may be displayed.

In this embodiment, a moving image is shot. However, the present invention is not limited to a moving image, and a still image may be shot and sub-data may be recorded on the recording medium together with the still image. At this time, still images may be taken at 1/30 second intervals, or may be taken at any arbitrary intervals.

Further, by applying the variable optical axis element to this embodiment, it is possible to suppress pixel blurring caused when the operator positively changes the direction of the optical axis.

[0085]

According to the present invention, the position of a subject in a three-dimensional space is detected from an image signal taken randomly while moving with a digital camera, and the position in the three-dimensional space is detected from directly above the detected position in the three-dimensional space. An image viewed and an image viewed obliquely are generated and displayed. Then, the locus of the movement of the digital camera at the time of shooting can be displayed on the displayed image viewed from directly above and the image viewed from obliquely. Further, at the time of reproduction, a necessary image can be immediately taken out.

[Brief description of the drawings]

FIG. 1 is an external view of a digital camera to which the present invention can be applied.

FIG. 2 is a schematic view of an example of a digital camera to which the present invention can be applied.

FIG. 3 is a block diagram of an example of a digital camera to which the present invention can be applied.

FIG. 4 is a block diagram of an example of a reproducing system to which the present invention can be applied.

FIG. 5 is a schematic view of another example of a digital camera to which the present invention can be applied.

FIG. 6 is a schematic diagram for explaining the present invention.

FIG. 7 is a plan view for explaining the present invention.

FIG. 8 is a schematic diagram for explaining the present invention.

FIG. 9 is a bird's-eye view for explaining the present invention.

FIG. 10 is a plan view for explaining the present invention.

FIG. 11 is a sketch for explaining the present invention.

FIG. 12 is a schematic diagram for explaining the present invention.

FIG. 13 is a schematic diagram for explaining the present invention.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Camera group, 2 ... Servo system, 3 ... Sensor group, 4 ... Recording medium, 5 ... Display part, 6 ... System computer, 21 ... Lens group 21, 22 ...
CCD group, 23: compression circuit, 25: expansion circuit, 28: image memory, 29: image movement amount detection unit, 30a, 30b: memory, 31: motion detection circuit , 32 ... arithmetic circuit, 33, 33X, 33Y ...
.Angular velocity sensor, 34 ... synthesis circuit, 35X, 35Y
... Integrating circuit, 36X, 36Y ... D / A converter

Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat II (reference) H04N 5/76 H04N 5/76 Z

Claims (22)

[Claims] 1. An image signal recording apparatus which captures a plurality of images and records the captured image signals, wherein: a position (x, y, z) of the imaging means in a three-dimensional space; a pan angle p; Attitude detecting means for detecting a tilt angle q and a roll angle r, an image sensor for converting an image of the subject incident on the subject into an image signal, compression means for compressing the image signal, and the compressed image signal And the position (x, y, position) in the three-dimensional space detected by the posture detecting means.
z) recording means for recording the pan angle p, the tilt angle q, and the roll angle r as sub-data. 2. The apparatus according to claim 1, further comprising: a plurality of lenses having a fixed positional relationship with each other, wherein a plurality of image signals can be obtained from the plurality of lenses. Image signal recording device. 3. The position (x, y, z) in the three-dimensional space according to claim 1, wherein data obtained from the first and second acceleration sensors arranged in directions orthogonal to each other are integrated. An image signal recording apparatus characterized in that the image signal is detected. 4. The image signal recording apparatus according to claim 1, wherein the tilt angle q and the roll angle r are obtained by integrating data obtained from the acceleration sensor. 5. The image signal recording apparatus according to claim 1, wherein the pan angle p is obtained by integrating data obtained from an angular velocity sensor. 6. The image signal recording apparatus according to claim 1, wherein an absolute value in a pan direction is determined by using a geomagnetic sensor. 7. The image signal recording apparatus according to claim 1, wherein an absolute value in the pan direction is determined by designating an azimuth in advance. 8. An image signal recording method for capturing a plurality of images and recording the captured image signals, wherein a position (x, y, z) of the imaging means in a three-dimensional space, a pan angle p, Detecting a tilt angle q and a roll angle r, converting the image of the subject incident on the image sensor from the subject into an image signal, compressing the image signal, recording the compressed image signal, and detecting And recording the position (x, y, z), pan angle p, tilt angle q, and roll angle r in the three-dimensional space as sub-data. 9. A plurality of image signals obtained by imaging the same subject from a plurality of positions are recorded, and a position (x, y, z) of the imaging means in a three-dimensional space, and a pan angle In an image signal reproducing method for reproducing a recording medium on which p, tilt angle q and roll angle r are recorded as sub-data, an image signal and sub-data recorded on the recording medium are reproduced. An image signal reproducing method, comprising: detecting a position of a subject in a three-dimensional space based on an image signal; generating and displaying a planar image from the detected position in the three-dimensional space. 10. The image signal reproducing method according to claim 9, wherein the subject is inside a building, and the two-dimensional image is a floor plan of a room. 11. The image signal reproducing method according to claim 9, wherein the subject is outdoors, and the planar image is a map. 12. An image according to claim 9, wherein when the image signal is being reproduced, if an arbitrary position being displayed is clicked, an enlarged image or a reduced image is displayed centering on the clicked arbitrary position. An image signal reproducing method, characterized in that an enlarged image or a reduced image is displayed around the dragged position by dragging the displayed enlarged image or reduced image. 13. The method according to claim 9, wherein a correct position in the three-dimensional space of the subject is obtained based on the reproduced sub data and information on the subject obtained from the reproduced image signal. An image signal reproducing method, characterized in that: 14. The image signal reproducing method according to claim 9, wherein the subject is inside a building or outdoors, and the planar image is an image of the subject viewed obliquely. 15. The image signal reproducing method according to claim 14, wherein the obliquely viewed image is displayed by changing an object three-dimensionally in accordance with a position of a changing viewpoint. 16. The image signal reproducing method according to claim 9, wherein a trajectory of the image pickup means at the time of photographing is displayed on the plane image. 17. The image signal reproducing method according to claim 16, wherein when an arbitrary position on the trajectory is selected, an image viewed from the arbitrary position is displayed. 18. A moving image according to claim 16, wherein when an arbitrary position of said trajectory is designated, a moving image is reproduced from said arbitrary position, and control such as slow reproduction, high-speed reproduction, rewinding or pause is performed on said moving image. An image signal reproducing method characterized in that it can be performed. 19. The apparatus according to claim 16, wherein, when an image of one object is taken from all directions by the imaging means, the one object is displayed so as to surround the object in an arc shape. Image signal reproducing method. 20. The image processing apparatus according to claim 16, further comprising: when an omnidirectional image is photographed from substantially one position by the imaging means, the one position is displayed as a specific mark, and the specific mark is selected. And displaying a panoramic image obtained by synthesizing the photographed omnidirectional images. 21. The panoramic image according to claim 20, wherein when the user clicks an arbitrary position on the panoramic image, an enlarged image or a reduced image is displayed around the clicked arbitrary position, and the displayed enlarged image or reduced image is dragged. The enlarged image or the reduced image is displayed with the dragged position as a center. 22. A recording medium on which an image signal of a still image and / or a moving image is recorded, wherein a position (x, y, z) of the imaging means in a three-dimensional space, a pan angle p, a tilt angle q, and a roll angle r Is the sub data,
A recording medium recorded with an image signal.